A method and apparatus for generating paralinguistic speech training data

Abstract

The application discloses a method and device for generating paralinguistic speech training data. The method comprises: obtaining original audio data, a text sequence and corresponding text unit timestamps; using a first sound event detection model to preliminarily screen out audio data containing a preset paralinguistic event; using a second sound event detection model to detect the screened-out audio data at a frame level to accurately locate the start and end positions of the paralinguistic event; generating a paralinguistic time interval according to the start and end positions; time-fusing the paralinguistic time interval with the timestamps of the text units, and automatically generating corresponding paralinguistic labels in the text sequence according to the overlapping relationship between the paralinguistic time interval and the timestamps of the text units. The application can realize the automatic, low-cost and high-precision generation of paralinguistic speech training data, replace the manual labeling process, and improve the data production efficiency and label quality.

Description

Technical Field

[0001] This invention relates to the fields of data mining, speech synthesis, and speech recognition, and in particular to a method, apparatus, electronic device, computer program product, and computer-readable storage medium for automatically generating speech training data with sub-language tags. Background Technology

[0002] Text-to-speech (TTS) technology aims to convert input text sequences into natural and fluent speech signals. Traditional TTS systems primarily focus on the clarity and naturalness of speech, that is, accurately reading out the text content. However, everyday human communication goes far beyond this, often accompanied by rich paralinguistic information, such as laughter, sighs, whispers, and cries. These non-verbal sound features can greatly enhance the expressiveness and emotional color of speech, making communication more vivid and human.

[0003] To make synthesized speech more realistic and expressive, modern TTS models, especially emotional TTS and stylized TTS models, have begun to incorporate paralinguistic synthesis capabilities. Achieving this requires massive amounts of high-quality training data. Each ideal training data sample typically consists of two parts: an audio clip containing a specific paralinguistic event, and a transcribed text that precisely corresponds to it, labeled with the paralinguistic tag in the correct location within the text. For example, for the audio phrase "Hello Anna, [laughter], nice to meet you," the corresponding text tag should be "Hello Anna, <laughter>Nice to meet you."

[0004] However, in existing technologies, constructing such paralinguistic speech datasets is extremely cumbersome and costly. Traditional methods heavily rely on manual operation, mainly involving two labor-intensive steps: first, manual screening of audio segments containing target paralinguistic events (such as laughter and sighs) from a massive raw audio library; second, annotators need to carefully listen to these audio clips, accurately determine the start and end times of the paralinguistic events, and manually insert or modify labels in the corresponding text transcripts. This process is not only inefficient and consumes a large amount of manpower and time, but also makes it difficult to guarantee the accuracy and consistency of the annotation results, greatly limiting the construction of large-scale, high-quality paralinguistic TTS training datasets and becoming a major bottleneck restricting the development of paralinguistic speech synthesis technology.

[0005] While it can determine the emotional information corresponding to the spoken content, it does not involve a sound event detection model and cannot achieve precise temporal localization of paralinguistic events. Another approach also fails to disclose any sound event detection model, particularly lacking frame-level event recognition capabilities, making fine-grained audio event analysis impossible. Furthermore, it cannot readily deduce a technical solution for automatically generating structured tags by fusing the time intervals of paralinguistic events with the timestamps of text units.

[0006] Therefore, how to design an automated method to efficiently and cost-effectively generate large amounts of speech training data with accurate sub-language labels from raw audio data is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0007] The main objective of this invention is to provide a method, apparatus, electronic device, and storage medium for generating paralinguistic speech training data, thereby solving the technical problems of existing technologies where obtaining paralinguistic speech training data relies on manual annotation, is costly, and inefficient.

[0008] To achieve the above objectives, the first aspect of the present invention provides a method for generating training data for secondary language speech, comprising:

[0009] Obtain raw audio data, and generate a text sequence and timestamps for multiple text units corresponding to the text sequence based on the raw audio data;

[0010] The original audio data is initially detected using a first sound event detection model to filter out events with a confidence level higher than a first threshold. Audio data containing preset secondary language events;

[0011] The second sound event detection model is used to perform frame-level detection on the selected audio data to determine the start and end positions of the preset sub-language event in the audio frame sequence.

[0012] The preset secondary language event's secondary language time interval is generated based on the start and end positions;

[0013] The time interval of the sub-language is fused with the timestamps of the multiple text units in a time sequence, and based on the overlap between the timestamps of the sub-language time interval and the text units, sub-language labels corresponding to the preset sub-language events are automatically generated in the text sequence to obtain target training data.

[0014] In a preferred embodiment, the timestamp of the text unit is a word-level timestamp or a phoneme-level timestamp.

[0015] In a preferred embodiment, before performing preliminary detection using the first sound event detection model, the method further includes: segmenting the original audio data into multiple audio segments, wherein the preliminary detection is performed on the multiple audio segments.

[0016] In a preferred embodiment, the first sound event detection model is an audio tagging model, and the preliminary detection step specifically involves performing multi-label classification on the original audio data to identify audio data containing one or more of the preset sub-language events.

[0017] In a preferred embodiment, the preset secondary language event includes at least one of: laughter, sighing, whispering, crying, and coughing.

[0018] In a preferred embodiment, the second sound event detection model is a sequence labeling model based on a recurrent neural network or a convolutional recurrent neural network, and the frame-level detection step specifically involves predicting the probability that each audio frame of the selected audio data belongs to each of the preset sub-language events.

[0019] In a preferred embodiment, the step of generating the sub-language time interval based on the start and end positions includes: smoothing the probability of each audio frame; and ensuring that the smoothed probability is higher than a second threshold. A continuous audio frame sequence is identified as a sub-language event segment; the times corresponding to the start and end frames of the sub-language event segment are converted into the start and end timestamps of the sub-language time interval.

[0020] In a preferred embodiment, the sub-language tag includes an insert tag and a persistent tag; the step of generating the sub-language tag based on the overlap relationship includes: determining whether the sub-language time interval overlaps with the timestamp of any of the text units; if no overlap occurs, generating the insert tag; if an overlap occurs, generating the persistent tag.

[0021] In a preferred embodiment, the step of generating the insert tag includes: merging and sorting the start and end timestamps of the sub-language time interval with the start and end timestamps of all text units; locating a text unit that is temporally adjacent to the sub-language time interval in the sorted time sequence; and inserting the insert tag representing the sub-language event after the text corresponding to the text unit that is temporally adjacent to the sub-language time interval in the text sequence.

[0022] In a preferred embodiment, the step of generating the persistence tag includes: identifying one or more consecutive text units that overlap with the sub-language time interval in time, forming a target text unit set; inserting a start tag of the persistence tag before the first text unit in the target text unit set in the text sequence; and inserting an end tag of the persistence tag after the last text unit in the target text unit set in the text sequence.

[0023] A second aspect of the present invention provides an apparatus for generating training data for secondary language speech, comprising:

[0024] The data acquisition module is used to acquire raw audio data and generate a text sequence and timestamps of multiple text units corresponding to the text sequence based on the raw audio data.

[0025] The preliminary detection module is configured with a first sound event detection model. This module performs preliminary detection on the raw audio data to filter out events with a confidence level higher than a first threshold. Audio data containing preset secondary language events;

[0026] The fine detection module is equipped with a second sound event detection model. The fine detection module is used to perform frame-level detection on the selected audio data to determine the start and end positions of the preset sub-language event in the audio frame sequence.

[0027] The interval generation module is used to generate the sub-language time interval of the preset sub-language event based on the start and end positions;

[0028] The tag generation module is used to perform time-series fusion of the sub-language time interval with the timestamps of the multiple text units, and automatically generate sub-language tags corresponding to the preset sub-language events in the text sequence based on the overlap relationship between the sub-language time interval and the timestamps of the text units, thereby obtaining target training data.

[0029] A third aspect of the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method as described in any of the preceding claims.

[0030] A fourth aspect of the present invention provides a computer program product including computer instructions that, when executed by a processor, implement the method as described in any of the preceding claims.

[0031] The fifth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any of the preceding claims.

[0032] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0033] This invention proposes a fully automated data generation process that combines coarse-grained rapid screening with fine-grained precise localization through a two-stage sound event detection strategy. This replaces the traditional time-consuming and labor-intensive manual screening and annotation process, greatly improving the production efficiency of paralinguistic speech training data.

[0034] Since it does not require a large amount of manpower, this invention significantly reduces the economic and time costs of data preparation, making it possible to construct a large-scale paralinguistic TTS training dataset from massive amounts of inexpensive raw audio data, laying the data foundation for training more powerful and expressive speech synthesis models.

[0035] This invention automatically generates two types of structured tags: "insertive" and "continuous," by temporally fusing and logically judging the precise time intervals of paralinguistic events detected by the AED model with the timestamps of text units (such as word-level) obtained by ASR and FA tools. This refined tagging strategy not only accurately locates paralinguistic events but also distinguishes their relationship with speech content (independent or concurrent occurrence), providing high-quality supervisory information for the TTS model to learn the synchronization relationship between paralinguistic events and text content. This helps generate synthesized speech with more natural prosody and emotional expression. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a schematic diagram of the application environment of a method for generating training data for secondary language speech provided in an embodiment of the present invention.

[0038] Figure 2 This is a structural block diagram of a paralinguistic speech training data generation device provided in an embodiment of the present invention.

[0039] Figure 3 This is a flowchart of a method for generating training data for secondary language speech provided in an embodiment of the present invention.

[0040] Figure 4 This is a schematic diagram of a two-stage sound event detection process provided in an embodiment of the present invention.

[0041] Figure 5 This is a schematic diagram of the timing fusion of the secondary language time interval and the text unit timestamp provided in the embodiment of the present invention. Detailed Implementation

[0045] To make the technical problems, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0046] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential," etc., indicating orientation or positional relationships, are based on the orientation or positional relationships shown in the accompanying drawings and are only for the convenience of describing this invention and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention.

[0047] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0048] Example 1

[0049] like Figure 1 As shown, this embodiment provides an application environment for a method of generating sublingual speech training data. This method can be executed by a generation device (100), which can be one or more servers, a cloud computing platform, or a personal computer with considerable computing power. The generation device (100) interacts with a raw data repository (101) and a target data repository (102). The raw data repository (101) stores a massive amount of unprocessed raw audio data. The generation device (100) obtains data from the raw data repository (101), and after a series of automated processing steps, stores the generated target training data (audio-text pairs) containing accurate sublingual labels into the target data repository (102) for use in subsequent TTS model training tasks.

[0050] like Figure 2 As shown, this embodiment provides a device (200) for generating secondary language speech training data. This device is Figure 1 A specific implementation of the audio generation device (100). The device (200) includes: a data acquisition module (201), an audio segmentation module (202), a preliminary detection module (203), a fine detection module (204), a range generation module (205), and a tag generation module (206).

[0051] The data acquisition module (201) is used to acquire raw audio data and generate a text sequence and timestamps for multiple text units corresponding to the text sequence based on the raw audio data. Specifically, this module integrates an Automatic Speech Recognition (ASR) engine and a Forced Alignment (FA) tool. The ASR engine (e.g., the open-source Whisper model or Paraformer model) is invoked to transcribe a long raw audio segment, obtaining a preliminary text sequence. Subsequently, the FA tool is used to force-align the text sequence with the original audio, thereby obtaining the precise start and end timestamps of each text unit in the text sequence. The granularity of the text units can be configured according to the needs of the TTS model. In this embodiment, word-level timestamps are preferred for easier subsequent processing. In other embodiments, finer-grained phoneme-level timestamps may also be used.

[0052] The audio segmentation module (202) is used to segment long segments of raw audio data and corresponding text sequences into multiple audio segments that are easier to process. To ensure the semantic integrity of the segmentation points, this module segments based on punctuation marks and the silence duration between words in the text. Specifically, a preferred segmentation condition is that there are punctuation marks (such as commas, periods, question marks, etc.) in the text sequence, and the time interval between the end time of the word corresponding to that position and the start time of the next word is greater than a preset silence duration threshold (e.g., greater than 1 second). Positions that meet this condition are determined as segmentation points, and the original audio and text sequences are thus segmented into a series of short audio segments and their corresponding transcribed text.

[0053] The preliminary detection module (203) is equipped with a first sound event detection model, which is used to quickly perform preliminary detection on multiple audio segments output by the audio segmentation module (202) to efficiently filter out candidate audio segments that may contain target sub-language events from massive amounts of data. In this embodiment, the first sound event detection model is an Audio Event Tagging (AET) model. The AET model is good at multi-label classification of the entire audio segment, outputting multiple sound event types that may exist in the audio and their respective confidence levels. The preliminary detection module (203) inputs each short audio segment into the AET model to obtain a list containing sound event labels and corresponding confidence levels. The module checks the top five sound event labels with the highest confidence levels in the list to determine whether they contain preset sub-language events. The preset sub-language events are what the TTS model wants to learn and synthesize, and may include at least one of the following: laughter, sigh, whisper, cry, and cough. If the detection result contains any preset sub-language event and its confidence level is higher than a preset first threshold, the system will detect the sub-language events. If the audio segment is selected as a valid candidate segment, it is retained and passed to the next module; otherwise, the segment is discarded. This module can significantly reduce the amount of data that needs to be processed in detail later, improving the efficiency of the overall process. It should be noted that although this embodiment uses the AET model as an example, the first sound event detection model of this application is not limited to this. Any model that can classify audio segments to determine whether they contain a specific sound event is applicable.

[0054] The fine detection module (204) is equipped with a second audio event detection model, which is used to perform more granular frame-level detection on the candidate audio data screened by the preliminary detection module (203) to accurately locate the start and end positions of the sub-language events on the time axis. In this embodiment, the second audio event detection model is an Audio Event Detection (AED) model. Unlike the AET model, the AED model can predict each frame of the audio and output the probability that the frame belongs to various audio events. Preferably, the AED model can be implemented using a sequence labeling architecture based on a recurrent neural network (RNN) or a convolutional recurrent neural network (CRNN). The fine detection module (204) inputs each candidate audio segment into the AED model and predicts the probability sequence of each audio frame of the segment (e.g., each frame is 10 milliseconds long) as belonging to each preset sub-language event.

[0055] The interval generation module (205) is used to decode and generate the start and end time intervals of the sub-language events, i.e., the sub-language time intervals, based on the frame-level probability sequence output by the fine detection module (204). This decoding process specifically includes the following steps: smoothing the frame-level probability sequence (e.g., using median filtering) to eliminate isolated noise points; and ensuring the smoothed probability is higher than a preset second threshold. A continuous sequence of audio frames is identified as a sublingual event segment. To further reduce false identification, sublingual event segments with excessively short durations (i.e., fewer than a certain number of frames) can be filtered out. The indices of the start and end frames of each valid sublingual event segment are converted into actual start and end timestamps in the original audio based on the audio's frame rate and frame shift, thus forming one or more sublingual time intervals.

[0056] The label generation module (206) is the core module for achieving automated labeling. This module is used to perform time-series fusion of the sub-language time intervals generated by the interval generation module (205) and the timestamps of multiple text units generated by the data acquisition module (201), and automatically generate corresponding sub-language labels in the text sequence based on their overlap relationship, ultimately obtaining target training data that can be used for training. The core logic of this module lies in distinguishing whether sub-language events occur independently or accompany speech, and generating two different types of labels accordingly: inserted labels and persistent labels.

[0057] Combination Figures 3 to 5 The complete process of the method for generating training data for secondary language speech provided in this embodiment will be described in detail below.

[0058] like Figure 3 As shown, the method mainly includes the following steps:

[0059] S301: Get audio and text timestamps.

[0060] After the system starts, the data acquisition module (201) performs its function to read a piece of raw audio data from the raw data repository (101). Through the internal ASR engine and FA tool, a text sequence corresponding to the audio content is generated, as well as the start and end timestamps of each word in the sequence. For example, for an audio piece, the text "Hi Anna nice to meet you" may be generated, and the timestamps of each word may be obtained, such as [('Hi', 0.2s, 0.4s), ('Anna', 0.5s, 0.8s),...].

[0061] S302: Segment audio data.

[0062] The audio segmentation module (202) performs its function. Based on punctuation marks and inter-word silences (>1s) in the text sequence generated by S301, the long audio and text are segmented into multiple short segments. This step decomposes the raw data into units that are more suitable for sound event detection.

[0063] S303: Preliminary screening of AET models.

[0064] like Figure 4 The upper part shows the function of the preliminary detection module (203). Each short audio segment obtained in S302 is input into the AET model (401). The AET model (401) outputs the confidence scores of various sound events contained in the segment. The system determines whether the events with the highest confidence scores contain preset sub-language events (such as laughter) and whether the confidence scores are greater than the first threshold. If yes, then keep the audio segment (402); otherwise, discard it.

[0065] S304: Fine-grained detection of AED model.

[0066] like Figure 4 The lower half shows the function of the fine detection module (204). The candidate audio segments (402) selected by S303 are input into the AED model (403). The AED model (403) analyzes each frame of the audio and outputs a frame-level probability matrix (404) representing the probability of a specific paralinguistic event (such as laughter or whispering) occurring in each frame.

[0067] S305: Decoding and generating secondary language time interval.

[0068] Perform the function of the interval generation module (205). Post-process the frame-level probability matrix (404) output by S304. Determine the probability threshold (>). The process involves continuous frame merging and short-time interval filtering, ultimately decoding to obtain one or more precise sub-language time intervals, each containing the event type, start time, and end time. For example, a possible result might be: ('laughter', 2.1s, 2.5s).

[0069] Before proceeding to the next step of time-series fusion, the method may also include an optional merging step. Specifically, it involves detecting whether there are multiple sequences of the same type with a time interval less than a preset duration threshold. The paralinguistic time interval. If so, these adjacent, similar paralinguistic events (e.g., several short laughs in a row in a conversation) are combined into a single, longer paralinguistic time interval, which helps to represent a complete paralinguistic event more macroscopically.

[0070] S306: Temporal Fusion and Tag Generation.

[0071] The core innovation of this invention lies in executing the tag generation module (206). The goal of this step is to integrate the sub-language time interval obtained in S305 with the word timestamps obtained in S301.

[0072] like Figure 5 As shown, the system places all word time intervals (501) and sub-language time intervals (502) on the same time axis. Then, for each sub-language time interval, it determines whether it overlaps with any word time interval.

[0073] - If no overlap occurs (Case A), such as Figure 5 As shown in the `laughter` event, its time interval is `[ _start, The string "_end]" falls entirely between the two words "Anna" and "nice", so the system determines that an insert tag should be generated.

[0074] - If overlap occurs (Case B), such as Figure 5 As shown in the `whisper` event, its time interval overlaps with the time interval of the three words "nice tomeet you", and the system determines that a persistent tag should be generated.

[0075] S307: Generate insert labels.

[0076] When it is determined that an insert tag needs to be generated, the detailed logic includes the following steps: S601: Merge the start and end timestamps of the sub-language time interval with the start and end timestamps of all text units, and sort them in ascending order according to time to construct a time series table containing all event points.

[0078] S602: Traverse the sorted time series list and identify a continuous time period consisting of the start and end times of the sub-language time interval, without containing any text units within it. This indicates that the sub-language event occurs during the silent period between two words.

[0079] S603: In the original text sequence, determine the preceding text unit whose start time point is closest in time to the start time point of the continuous time period.

[0080] S604: Place the insert tag representing the secondary language event after the end of the previous text unit. The insert tag is formatted as a single, independent tag, for example... <laughter>or <sigh>This is used to indicate the occurrence of a momentary or short-lived non-speech event. The final generated text is, for example: "Hi Anna, <laughter>Nice to meet you.

[0081] S308: Generate persistent tags.

[0082] When it is determined that persistent tags are to be generated, the detailed logic includes the following steps: S701: Determine the start timestamp of the time interval with the secondary language. and end timestamp All text units that have a temporal intersection. The criterion for temporal intersection is the time interval of the text unit. With secondary language time interval The intersection of them is not empty.

[0084] S702: Identify all text units that meet the judgment conditions as affected text units, forming a target text unit set.

[0085] S703: In a text sequence, locate the first text unit that appears earliest in time among the affected text units and insert a start label before its text, for example... <whisper>.

[0086] S704: Locate the last text unit in the affected text unit that appears latest in time, and insert an end tag after its text, for example...< / whisper> In this way, all text content continuously affected by paralinguistic events is completely encapsulated to represent a persistent paralinguistic state accompanying speech. For example, the generated text might be: "Hi Anna, <whisper> It's nice to meet you.< / whisper> This tag format ensures that the TTS model can learn the synchronous relationship between paralinguistic features (such as whispers) and specific text content, thereby generating target speech with more natural rhythm and emotional expression.

[0087] S309: Result Filtering and Output.

[0088] Since there may be slight differences between the global judgment of the AET model and the local judgment of the AED model, the system performs a final verification to ensure the quality of the final data. The text sequence with sub-language labels generated in S307 or S308 is compared with the sub-language detection result of the AET model for that segment in S303. If the final generated label type does not match the main sub-language type detected by AET, the data segment is considered to be potentially erroneous and is removed. The verified data segments and their corresponding labeled texts are combined into a complete target training data set and stored in the target data repository (102).

[0089] In summary, this embodiment achieves fully automated generation of high-quality TTS training data with accurate structured labels from raw audio using the above method, effectively solving the problems mentioned in the background art.

[0090] Example 2

[0091] This embodiment provides another audio segmentation scheme. In Embodiment 1, audio segmentation is performed after ASR transcription, based on text punctuation and silence duration. In a variant of this embodiment, the audio segmentation module (202) can employ a different workflow, i.e., segmentation is performed before ASR.

[0092] Specifically, the audio segmentation module (202) can use Voice Activity Detection (VAD) technology to process long segments of raw audio. VAD technology can identify speech segments and silence segments in the audio. This module can set segmentation points at the detected longer silence segments, pre-segmenting the raw audio into multiple shorter audio segments. Then, the data acquisition module (201) performs ASR and FA operations on these pre-segmented short audio segments respectively to obtain their respective text sequences and word timestamps.

[0093] This "segmentation before recognition" approach may be advantageous when processing audio with complex background noise or containing a large amount of non-speech content, as it can first remove large sections of pure silence or noise, reducing the burden on the ASR engine and potentially improving recognition accuracy. Subsequent preliminary detection, fine detection, and tag generation steps are the same as in Example 1.

[0094] It should be noted that the various models used in this application, such as the ASR model, FA tool, AET model, and AED model, can all be any mature model in the field capable of achieving the corresponding functions. For example, the AET and AED models, in addition to being based on CRNN, can also be models based on pure convolutional neural networks (CNN) or Transformer architectures. Similarly, the various thresholds involved in the process, such as the first threshold... Second threshold The threshold for silence duration, the threshold for the shortest duration of secondary speech, etc., can be set empirically or determined through experimentation based on the actual application scenario and data characteristics, and are not intended to limit the present invention.

[0095] Example 3

[0096] This invention also provides an electronic device for performing the above-described method. The electronic device (800) may be a server, a personal computer (PC), a laptop computer, or a mobile terminal, etc.

[0097] The electronic device (800) includes a processor (801), a memory (802), a network interface (803), and an input / output interface (804), which communicate with each other via a system bus (805) or other connection methods.

[0098] The processor (801) is the control center of the electronic device (800), and may be one or more central processing units (CPU), graphics processing units (GPUs), or application-specific integrated circuits (ASICs), etc. It is responsible for executing computer program instructions stored in memory (802) to implement the method for generating secondary language speech training data described in the foregoing embodiments of the present invention.

[0099] The memory (802) can be volatile memory (such as random access memory RAM) or non-volatile memory (such as read-only memory ROM, flash memory, hard disk), or any combination of these two types of memory. The memory (802) is used to store the operating system, device drivers, and various application software programs. In particular, the memory (802) stores a computer program that implements the method of the present invention. When the processor (801) executes the program, it loads and executes the various steps in the aforementioned method, including but not limited to: acquiring raw audio data and generating timestamps, performing preliminary screening using a first sound event detection model, performing fine detection using a second sound event detection model, generating sub-language time intervals, and generating insertive or persistent sub-language tags based on overlap relationships, etc.

[0100] The network interface (803) is used for the electronic device (800) to communicate with other devices (such as the original data repository and the target data repository), and can be a wired interface (such as an Ethernet interface) or a wireless interface (such as a Wi-Fi or Bluetooth interface).

[0101] The input / output interface (804) is used to connect external input devices (such as keyboards and mice) and output devices (such as monitors) to manage human-computer interaction.

[0102] In this embodiment, the processor (801) specifically implements the program by executing the program in the memory (802). Figure 2 All or part of the functions of the data acquisition module (201), audio segmentation module (202), preliminary detection module (203), fine detection module (204), interval generation module (205) and tag generation module (206) described in the document.

[0103] Those skilled in the art will understand that all or part of the steps in the embodiments of the present invention can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the steps of the methods described in the above embodiments. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0105] Furthermore, the embodiments of this application can be further described through the following technical solutions, which in detail define the scope of protection and technical details of this application:

[0106] 1. A method for generating training data for secondary language speech, characterized in that it includes:

[0107] Obtain raw audio data, and generate a text sequence and timestamps for multiple text units corresponding to the text sequence based on the raw audio data;

[0108] The original audio data is initially detected using a first sound event detection model to filter out events with a confidence level higher than a first threshold. Audio data containing preset secondary language events;

[0109] The second sound event detection model is used to perform frame-level detection on the selected audio data to determine the start and end positions of the preset sub-language event in the audio frame sequence.

[0110] The preset secondary language event's secondary language time interval is generated based on the start and end positions;

[0111] The time interval of the sub-language is fused with the timestamps of the multiple text units in a time sequence, and based on the overlap between the timestamps of the sub-language time interval and the text units, sub-language labels corresponding to the preset sub-language events are automatically generated in the text sequence to obtain target training data.

[0112] 2. The method according to Scheme 1, wherein the timestamp of the text unit is a word-level timestamp or a phoneme-level timestamp.

[0113] Optionally, before performing preliminary detection using the first sound event detection model, the method further includes:

[0114] The original audio data is divided into multiple audio segments, and the preliminary detection is performed on the multiple audio segments.

[0115] 3. The method according to Scheme 1, wherein the first sound event detection model is an audio tagging model, and the preliminary detection step specifically involves performing multi-label classification on the original audio data to identify audio data containing one or more of the preset sub-language events.

[0116] Optionally, the preset secondary language events include at least one of: laughter, sighing, whispering, crying, and coughing.

[0117] 4. The method according to Scheme 1, wherein the second sound event detection model is a sequence labeling model based on a recurrent neural network or a convolutional recurrent neural network, and the frame-level detection step specifically involves predicting the probability that each audio frame of the selected audio data belongs to each of the preset sub-language events.

[0118] Optionally, the step of generating the sub-language time interval based on the start and end positions includes:

[0119] The probability of each audio frame is smoothed.

[0120] The probability after smoothing is higher than the second threshold. The continuous audio frame sequence was identified as a fragment of a secondary language event;

[0121] The times corresponding to the start and end frames of the sub-language event fragment are converted into the start and end timestamps of the sub-language time interval.

[0122] 5. The method according to Scheme 1, characterized in that the sub-language tags include insert tags and persistent tags; the step of generating the sub-language tags based on the overlap relationship includes:

[0123] Determine whether the sub-language time interval overlaps with the timestamp of any of the text units;

[0124] If no overlap occurs, the inserted label is generated;

[0125] If overlap occurs, the persistent tag is generated.

[0126] Optionally, the step of generating the insert label includes:

[0127] The start and end timestamps of the sub-language time interval are merged and sorted with the start and end timestamps of all text units;

[0128] In the sorted time series, locate a text unit that is temporally adjacent to the sub-language time interval;

[0129] In the text sequence, the insert tag representing the sub-language event is inserted after the text corresponding to the text unit immediately preceding it in time.

[0130] Optionally, the step of generating the insert label specifically includes:

[0131] Construct a time series table containing all event points, where each event point includes the start and end time points of each text unit, as well as the start and end time points of the sub-language time interval, and attach a type identifier to each event point;

[0132] Sort the time series list in ascending order according to time sequence;

[0133] Traverse the sorted time series table to identify a continuous time period consisting of the start and end times of the sub-language time interval, which does not contain the start or end times of any text unit.

[0134] In the text sequence, the start time of the continuous time period is determined to be the end position of the preceding text unit that is closest in time, and the inserted tag is placed after the end position. The format of the inserted tag is configured as a single independent tag to indicate the occurrence of a momentary or short-term non-voice event.

[0135] Optionally, the step of generating the persistent tag includes:

[0136] Identify one or more consecutive text units that overlap with the sub-language time interval in time, forming a target text unit set;

[0137] In the text sequence, the start tag of the persistent tag is inserted before the first text unit in the target text unit set;

[0138] In the text sequence, the end tag of the persistence tag is inserted after the last text unit in the target text unit set.

[0139] Optionally, the step of generating the persistent tag specifically includes:

[0140] Determine the start timestamp of the sub-language time interval. and end timestamp All text units that have a temporal intersection, wherein the temporal intersection is determined by the start timestamp of the text unit. With end timestamp The interval formed With secondary language time interval The intersection is not empty;

[0141] All text units that meet the aforementioned judgment conditions are identified as affected text units;

[0142] In the text sequence, locate the first text unit that appears earliest in time among the affected text units, and insert a start tag before its text;

[0143] Locate the last text unit that appears latest in time among the affected text units, and insert an end tag after its text, thereby completely wrapping all text content that is continuously affected by the paralinguistic event to represent a persistent paralinguistic state that accompanies speech occurrence.

[0144] 6. The method according to Scheme 1, characterized in that, after generating the secondary language time interval and before performing the temporal fusion, it further includes:

[0145] Detect whether there are multiple items of the same type with a time interval less than a preset duration threshold. The aforementioned secondary language time interval;

[0146] If so, the multiple sub-language time intervals are merged into a single, longer sub-language time interval.

[0147] 7. A device for generating training data for secondary language speech, characterized in that it comprises:

[0148] The data acquisition module is used to acquire raw audio data and generate a text sequence and timestamps of multiple text units corresponding to the text sequence based on the raw audio data.

[0149] The preliminary detection module is configured with a first sound event detection model. This module performs preliminary detection on the raw audio data to filter out events with a confidence level higher than a first threshold. Audio data containing preset secondary language events;

[0150] The fine detection module is equipped with a second sound event detection model. The fine detection module is used to perform frame-level detection on the selected audio data to determine the start and end positions of the preset sub-language event in the audio frame sequence.

[0151] The interval generation module is used to generate the sub-language time interval of the preset sub-language event based on the start and end positions;

[0152] The tag generation module is used to perform time-series fusion of the sub-language time interval with the timestamps of the multiple text units, and automatically generate sub-language tags corresponding to the preset sub-language events in the text sequence based on the overlap relationship between the sub-language time interval and the timestamps of the text units, thereby obtaining target training data.

[0153] Optionally, the first sound event detection model configured in the preliminary detection module is an audio tagging model. The preliminary detection module is specifically used to perform multi-label classification on the original audio data to identify audio data containing one or more of the preset sub-language events, wherein the preset sub-language events include at least one of laughter, sighing, whispering, crying, and coughing.

[0154] Optionally, the second sound event detection model configured in the fine detection module is a sequence labeling model based on a recurrent neural network or a convolutional recurrent neural network. Specifically, the fine detection module is used to predict the probability that each audio frame of the selected audio data belongs to each of the preset sub-language events.

[0155] Optionally, the label generation module is specifically used for:

[0156] Determine whether the sub-language time interval overlaps with the timestamp of any of the text units;

[0157] If no overlap occurs, an insert tag is generated;

[0158] If overlap occurs, a persistent tag is generated.

[0159] Optionally, when generating the insertable tag, the tag generation module is specifically used to locate a text unit in the text sequence that is temporally adjacent to the sub-language time interval, and insert the insertable tag representing the sub-language event after the text corresponding to that text unit.

[0160] Optionally, when generating the persistent tag, the tag generation module is specifically used for:

[0161] Identify one or more consecutive text units that overlap with the sub-language time interval in time, forming a target text unit set;

[0162] In the text sequence, the start tag of the persistent tag is inserted before the first text unit in the target text unit set;

[0163] In the text sequence, the end tag of the persistence tag is inserted after the last text unit in the target text unit set.

[0164] Optionally, when generating the persistent tag, the tag generation module is further configured to:

[0165] By determining the time interval of the text unit With secondary language time interval The text units that have a temporal intersection with the sub-language time interval are determined as affected text units by checking whether the intersection is empty.

[0166] In the text sequence, the earliest text unit in the affected text unit is located, and a start label is placed in front of it. The latest text unit in the affected text unit is located, and an end label is placed behind it. The combination of the start label and the end label is used to define the text range affected by the continuous sub-language event.

[0167] Optionally, the data acquisition module is specifically used to generate word-level timestamps or phoneme-level timestamps as timestamps for the text units.

[0168] Optionally, it also includes an audio segmentation module, which is used to segment the original audio data into multiple audio segments before the preliminary detection module performs the detection.

[0169] 8. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that the processor executes the computer program to implement the method as described in any one of claims 1 to 13.

[0170] 9. A computer program product comprising computer instructions, characterized in that, when the instructions are executed by a processor, they implement the method as described in any one of claims 1 to 13;

[0171] 10. A computer-readable storage medium having a computer program stored thereon, characterized in that the program, when executed by a processor, implements the method as described in any one of claims 1 to 13.< / laughter> < / sigh> < / laughter> < / laughter>

Claims

1. A method for generating training data for secondary language speech, characterized in that, include: Obtain raw audio data, and generate a text sequence and timestamps for multiple text units corresponding to the text sequence based on the raw audio data; The original audio data is initially detected using a first sound event detection model to filter out events with a confidence level higher than a first threshold. Audio data containing preset secondary language events; The second sound event detection model is used to perform frame-level detection on the selected audio data to determine the start and end positions of the preset sub-language event in the audio frame sequence. The preset secondary language event's secondary language time interval is generated based on the start and end positions; The time interval of the sub-language is fused with the timestamps of the multiple text units in a time sequence, and based on the overlap between the timestamps of the sub-language time interval and the text units, sub-language labels corresponding to the preset sub-language events are automatically generated in the text sequence to obtain target training data.

2. The method according to claim 1, characterized in that, Before performing preliminary detection using the first sound event detection model, the method further includes: dividing the original audio data into multiple audio segments, wherein the preliminary detection is performed on the multiple audio segments; optionally, the timestamp of the text unit is a word-level timestamp or a phoneme-level timestamp.

3. The method according to claim 1, characterized in that, The first sound event detection model is an audio tagging model. The preliminary detection steps specifically involve performing multi-label classification on the original audio data to identify audio data containing one or more of the preset sub-language events. Optionally, the preset secondary language events include at least one of: laughter, sighing, whispering, crying, and coughing.

4. The method according to claim 1, characterized in that, The second sound event detection model is a sequence labeling model based on a recurrent neural network or a convolutional recurrent neural network. The specific step of the frame-level detection is to predict the probability that each audio frame of the selected audio data belongs to each of the preset sub-language events.

5. The method according to claim 4, characterized in that, The step of generating the sub-language time interval based on the start and end positions includes: The probability of each audio frame is smoothed. The probability after smoothing is higher than the second threshold. The continuous audio frame sequence was identified as a fragment of a secondary language event; The times corresponding to the start and end frames of the sub-language event fragment are converted into the start and end timestamps of the sub-language time interval.

6. The method according to claim 1, characterized in that, The sub-language tags include insert tags and persistent tags; the step of generating the sub-language tags based on the overlap relationship includes: Determine whether the sub-language time interval overlaps with the timestamp of any of the text units; If no overlap occurs, the inserted label is generated; If overlap occurs, the persistent tag is generated.

7. The method according to claim 6, characterized in that, The step of generating the insert label includes: The start and end timestamps of the sub-language time interval are merged and sorted with the start and end timestamps of all text units; In the sorted time series, locate a text unit that is temporally adjacent to the sub-language time interval; In the text sequence, the insert tag representing the sub-language event is inserted after the text corresponding to the text unit immediately preceding it in time.

8. A device for generating training data for secondary language speech, characterized in that, include: The data acquisition module is used to acquire raw audio data and generate a text sequence and timestamps of multiple text units corresponding to the text sequence based on the raw audio data. The preliminary detection module is configured with a first sound event detection model. The preliminary detection module is used to perform preliminary detection on the original audio data to filter out audio data containing preset sub-language events with a confidence level higher than a first threshold. The fine detection module is equipped with a second sound event detection model. The fine detection module is used to perform frame-level detection on the selected audio data to determine the start and end positions of the preset sub-language event in the audio frame sequence. The interval generation module is used to generate the sub-language time interval of the preset sub-language event based on the start and end positions; The tag generation module is used to perform time-series fusion of the sub-language time interval with the timestamps of the multiple text units, and automatically generate sub-language tags corresponding to the preset sub-language events in the text sequence based on the overlap relationship between the sub-language time interval and the timestamps of the text units, thereby obtaining target training data.

9. The apparatus according to claim 8, characterized in that, The first sound event detection model configured in the preliminary detection module is an audio tagging model. The preliminary detection module is specifically used to perform multi-label classification on the original audio data to identify audio data containing one or more of the preset sub-language events. The preset sub-language events include at least one of laughter, sighs, whispers, crying, and coughing.

10. A computer program product comprising computer instructions, characterized in that, When the instructions are executed by the processor, they implement the method as described in any one of claims 1 to 7.

Images