Data generation device, data generation method, and program

Abstract

This data generation device comprises a data generation unit configured to generate second speech data in which a timing of utterance of a second speaker in first speech data corresponding to a period of alternation of a speaker from a first speaker to the second speaker in a conversation is changed relative to a timing of utterance of the first speaker, thereby enabling generation of a data set including various alternate latent times.

Description

Data generation device, data generation method and program

[0001] The present invention relates to a data generation device, a data generation method, and a program.

[0002] In the classification of events for speaker-turning during a conversation based on speech, an event can be, for example, an "interruption" or "non-interruption." To train a classifier using supervised learning, it is necessary to annotate the correct event data for speaker-turning. Because such annotation is costly, it is necessary to efficiently train a model using a small amount of training data.

[0003] In the field of speech processing, there is a data augmentation technique for efficiently learning a model from a small amount of data, which involves masking the spectrogram of speech data to improve robustness against speaker differences (Non-Patent Document 1).

[0004] DS Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, ED Cubuk, and QV Le, "Specaugment: A simple data augmentation method for automatic speech recognition", arXiv:1904.08779(2019)

[0005] Turn-taking latency is an important feature in turn-taking event classification. For example, turn-taking events with short turn-taking latencies are more likely to be "interruptions" than "non-interruptions." Therefore, it is important that the training data contain a variety of turn-taking latencies.

[0006] Conventional speech data feature conversion techniques could manipulate only voice pitch and speech rate, but could not change the timing of speech collisions or the start of speech. Therefore, when speech timing, such as turn-on latency, is important information, recording actual human speech is an option. However, this method has limitations on the diversity of timing (i.e., turn-on latency), making it difficult to obtain training data that includes a variety of turn-on latency. If a model is trained using training data that does not include a variety of turn-on latency, it is difficult to obtain a model that achieves high performance in event detection.

[0007] The present invention has been made in view of the above points, and has as its object to enable the generation of a data set including a variety of alternation latencies.

[0008] In order to solve the above problem, the data generating device has a data generating unit configured to generate second audio data in which the timing of the speech of the second speaker in first audio data corresponding to the period of speaker change from a first speaker to a second speaker in a conversation is changed relative to the timing of the speech of the first speaker.

[0009] It may be possible to generate datasets containing a variety of alternation latencies.

[0010] FIG. 1 is a diagram illustrating an example of a hardware configuration of the speech interruption estimation device 10 according to an embodiment of the present invention. FIG. 2 is a diagram illustrating an example of a functional configuration of the speech interruption estimation device 10 according to an embodiment of the present invention. FIG. 3 is a diagram for explaining extension data. FIG. 4 is a flowchart for explaining an example of a processing procedure for training data extension processing. FIG. 5 is a first diagram for explaining generation of extended utterance pair data. FIG. 6 is a second diagram for explaining generation of extended utterance pair data.

[0011] An embodiment of the present invention will now be described with reference to the accompanying drawings. Fig. 1 is a diagram showing an example of the hardware configuration of a speech interruption estimation device 10 according to an embodiment of the present invention. The speech interruption estimation device 10 shown in Fig. 1 includes a drive device 100, an auxiliary storage device 102, a memory device 103, a processor 104, and an interface device 105, all of which are interconnected via a bus B.

[0012] A program that realizes the processing in the speech interruption estimation device 10 is provided by a recording medium 101 such as a CD-ROM. When the recording medium 101 storing the program is set in the drive device 100, the program is installed from the recording medium 101 to the auxiliary storage device 102 via the drive device 100. However, the program does not necessarily have to be installed from the recording medium 101, but may be downloaded from another computer via a network. The auxiliary storage device 102 stores the installed program as well as necessary files, data, etc.

[0013] When an instruction to start the program is received, the memory device 103 reads and stores the program from the auxiliary storage device 102. The processor 104 is a CPU or a GPU (Graphics Processing Unit), or a CPU and a GPU, and executes functions related to the speech interruption estimation device 10 in accordance with the program stored in the memory device 103. The interface device 105 is used as an interface for connecting to a network.

[0014] 2 is a diagram showing an example of the functional configuration of the speech interruption estimation device 10 according to an embodiment of the present invention. In FIG. 2, the speech interruption estimation device 10 includes a data generation unit 11, a learning unit 12, an estimation unit 13, and a speech interruption estimation model 14. These units are implemented by a processor 104 executing one or more programs installed in the speech interruption estimation device 10. The speech interruption estimation device 10 also uses a learning data storage unit 111 and an extension data storage unit 112. These storage units can be implemented using, for example, an auxiliary storage device 102, a storage device connectable to the speech interruption estimation device 10 via a network, or the like.

[0015] The training data storage unit 111 stores a training data set, which is a collection of training data for the speech interruption estimation model 14. One training data set includes utterance pair data and a speech interruption label. That is, one utterance pair data is associated with one speech interruption label.

[0016] Utterance pair data is audio data (corresponding to the period of the speaker change) extracted from audio data of a certain conversation (hereinafter referred to as "conversation data") for several seconds before and after the speaker change. For example, when a speaker changes from a first speaker to a second speaker, audio data from a time interval from 1.5 seconds before the second speaker's utterance start time to 1.0 seconds after the second speaker's utterance start time may be used as the utterance pair data. Typically, speaker changes occur multiple times between the start and end of a conversation. Therefore, multiple utterance pair data may be extracted from one conversation data. The training dataset may include training data including utterance pair data extracted from conversation data of different conversations. Note that the audio data from which the utterance pair data is extracted contains utterances in different channels for each speaker. That is, in this embodiment, the number of channels of the audio data is two or more. The same applies to the number of channels of the utterance pair data, etc. extracted from the audio data.

[0017] The speech interruption label is a label (numerical value) that indicates the correct answer of whether a speaker change in the corresponding utterance pair data is "a speech interruption" or "not a speech interruption." For example, 0 indicates "a speech interruption," and 1 indicates "not a speech interruption."

[0018] Note that interruption is an example of a speaker change event. Interruption refers to a phenomenon in which a second speaker begins speaking despite the first speaker not having finished speaking (including a silent state where the first speaker is searching for words), excluding accidental speech collisions, in which the second speaker intentionally interrupts the first speaker's speech and begins speaking. When interruption occurs, the speech of the two speakers basically overlaps, but when interruption occurs while the first speaker is searching for words, such overlap does not occur.

[0019] The learning unit 12 learns the speech interruption estimation model 14 using the learning data set stored in the learning data storage unit 111. Specifically, for each piece of learning data included in the learning data set, the learning unit 12 updates the parameters of the speech interruption estimation model 14 so that when speech pair data included in the learning data is input to the speech interruption estimation model 14, the output from the speech interruption estimation model 14 approaches the speech interruption label included in the learning data.

[0020] The speech interruption estimation model 14 is a machine learning model that inputs speech pair data and outputs information indicating whether or not a speech interruption has occurred (respective posterior probabilities). The model structure of the speech interruption estimation model 14 can be, for example, the same structure as that described in Reference 1 (a pre-trained speech encoder and a feed-forward neural network).

[0021] The estimation unit 13 uses the trained speech interruption estimation model 14 to estimate whether or not arbitrarily input speech pair data is a speech interruption.

[0022] The data generating unit 11 performs a process (hereinafter referred to as a "training data expansion process") to increase (expand) the diversity of the changeover latencies for a training data set (hereinafter referred to as an "initial training data set"). Specifically, for each training data set included in the training data set, the data generating unit 11 generates new utterance pair data (hereinafter referred to as "extended utterance pair data") in which, for a speaker change from a first speaker to a second speaker included in the utterance pair data of the training data, the timing of the utterance of the second speaker is changed relative to the timing of the utterance of the first speaker. In this case, the data generating unit 11 changes the timing of the utterance of the second speaker in the extended utterance pair data according to the utterance interruption label corresponding to the utterance pair data from which the extended utterance pair data was generated. Speech with an utterance interruption tends to have a short changeover latency, while speech without an utterance interruption tends to have a long changeover latency. Therefore, for example, if the speech interruption label indicates a speech interruption (if it is "0"), the data generating unit 11 makes the timing of the utterance of the second speaker earlier than that of the speech pair data, and if the label indicates no speech interruption, makes the timing of the utterance of the second speaker later than that of the utterance pair data. The data generating unit 11 adds training data including the generated extended utterance pair data and the speech interruption label corresponding to the utterance pair data from which the extended utterance pair data was generated to the training data storage unit 111.

[0023] The extension data storage unit 112 stores a set (hereinafter referred to as an "extension data set") of data (hereinafter referred to as "extension data") used in the training data extension process.

[0024] Each piece of augmented data corresponds to a piece of training data in the original training dataset and is used to generate extended training data based on that training data. The augmented data includes extension data, labels for identifying each speaker in the extension data (hereinafter referred to as "speaker labels"), and the start and end times of each speaker's utterance. Each time is the elapsed time from the start of the extension data.

[0025] The extension data is speech data that also includes a time interval of about several seconds after the time interval corresponding to the utterance pair data of the corresponding training data.

[0026] FIG. 3 is a diagram for explaining extension data. FIG. 3 shows the relationship between the time intervals of utterance pair data in the conversation from which the utterance pair data was extracted and the time intervals of the extension data. As shown in FIG. 3, the extension data is audio data obtained by cutting out a time interval from the start time of the utterance pair data to the end time of the utterance pair data + α (seconds) in the audio data from which the utterance pair data of the corresponding training data was extracted. Therefore, the extension data also includes utterances of the second speaker in a time interval not included in the utterance pair data. The reason why α (seconds) of speech audio is required is because when the timing of the second speaker's utterance is shifted forward (advanced), this portion needs to be included in the extended utterance pair data. This point will be described later.

[0027] In this embodiment, for convenience, the extension data is data obtained by extending only the rear side of the corresponding utterance pair data, but the front side may also be extended. In this case, the difference in start time between the extension data and the corresponding utterance pair data (i.e., the extension time of the front side of the extension data) may also be included in the extension data.

[0028] The processing procedure executed by the speech interruption estimation device 10 will be described below.

[0029] FIG. 4 is a flowchart illustrating an example of the processing procedure of the training data expansion process. The processing procedure of FIG. 4 is executed for all training data (utterance pair data) included in the training dataset. The utterance pair data to be processed in FIG. 4 is hereinafter referred to as "target data." Note that, although an example in which there are two speakers will be described here, the same processing procedure can also be executed when there are three or more speakers. When there are three or more speakers, the first speaker can simply be replaced with multiple speakers other than the second speaker.

[0030] In step S101, the data generator 11 generates first speaker data, which is voice data other than that of the second speaker, by removing the voice of the second speaker from the target data of multiple channels. The removal of the voice of the second speaker can be achieved by removing the channel corresponding to the second speaker from the target data.

[0031] FIG. 5 is a first diagram for explaining generation of extended utterance pair data. (1) in FIG. 5 shows utterance pair data d1-1, which includes both an utterance by a first speaker and an utterance by a second speaker, as an example of target data. In the utterance pair data d1-1, a series of "X"s indicates an utterance by the first speaker, and a series of "Y"s indicates an utterance by the second speaker. In the utterance pair data d1-1, the start time of the utterance by the second speaker is t1 seconds earlier than the end time of the utterance by the first speaker, and an utterance interruption label indicating a speech interruption is associated with the utterance pair data d1-1.

[0032] 5(2) shows first speaker data d2-1 generated by removing the speech of the second speaker from utterance pair data d1-1. In this way, the first speaker data d2-1 is speech data having the same time length as the utterance pair data d1-1 (corresponding to the same time period), but in which the period in the utterance pair data d1-1 in which only the second speaker spoke is silent.

[0033] 6 is a second diagram for explaining generation of extended utterance pair data, which shows that the start time of an utterance by a second speaker may be later than the end time of an utterance by a first speaker.

[0034] 6(1) shows utterance pair data d1-2 as an example of target data. In the utterance pair data d1-2, the start time of the second speaker's utterance is t2 seconds later than the end time of the first speaker's utterance, and an utterance interruption label indicating that the utterance pair data d1-2 is not an utterance interruption is associated with the utterance pair data d1-2. Note that FIG. 6(2) shows first speaker data d2-2 generated by removing the audio of the second speaker's utterance from the utterance pair data d1-2.

[0035] Hereinafter, when there is no need to distinguish between the first speaker data d2-1 and the first speaker data d2-2, they will be referred to as "first speaker data d2."

[0036] Next, the data generation unit 11 extracts a predetermined time interval (hereinafter referred to as the "extraction target interval") of the speech of the second speaker from the extension data corresponding to the target data, starting from the start time of the speech (S102). How the extraction target interval is determined will be described later. As described with reference to FIG. 3, the extension data also includes an interval (a interval of α seconds) of the speech of the second speaker that is not included in the utterance pair data. Therefore, in step S102, there is a possibility that the speech of this interval will also be extracted.

[0037] Next, the data generation unit 11 synthesizes and combines the first speaker data d2 with the audio (one channel of audio) of the second speaker extracted in step S102, thereby generating multi-channel extended utterance pair data with an additional channel of audio of the second speaker (S103). At this time, the data generation unit 11 performs synthesis so that the start time (start timing) of the utterance of the second speaker in the extended utterance pair data changes from the start time (start timing) of the utterance of the second speaker in the target data. The start time in the target data is the same as the start time of the utterance of the second speaker in the extension data included in the extension data including extension data corresponding to the target data.

[0038] 5 (3) shows extended utterance pair data d3-1. In the extended utterance pair data d3-1, the start time of the second speaker's utterance is x1 seconds earlier than in the utterance pair data d1-1. In other words, in (3), the start time of the second speaker's utterance is (t1 + x1) seconds earlier than the end time of the first speaker's utterance. As a result, the speech interval of the second speaker's utterance included in the utterance pair data d1-1 is the interval from the start time of the utterance to X1 seconds, whereas the speech interval of the second speaker's utterance included in the extended utterance pair data d3-1 is the interval from the start time of the utterance to (X1 + x1) seconds.

[0039] 6 (3) shows extended utterance pair data d3-2. In the extended utterance pair data d3-2, the start time of the second speaker's utterance is x2 seconds later than that of the utterance pair data d1-2. In other words, in (3), the start time of the second speaker's utterance is (t2 + x2) seconds later than the end time of the first speaker's utterance. As a result, the speech interval of the second speaker's utterance included in the utterance pair data d1-2 is the interval from the start time of the utterance to X2 seconds, whereas the speech interval of the second speaker's utterance included in the extended utterance pair data d3-2 is the interval from the start time of the utterance to (X2 - x2) seconds.

[0040] In this way, the extraction target section in step S102 differs depending on whether the timing of the start of the second speaker's utterance in the extended utterance pair data is advanced or delayed. If the timing is advanced by x seconds, the extraction target section should be made x seconds longer than the speech section of the second speaker in the target data. If the timing is delayed by x seconds, the extraction target section should be made x seconds shorter than the speech section of the second speaker in the target data. The value of x is, for example, around 0.1 seconds.

[0041] Basically, if the speech interruption label associated with the speech pair data corresponding to the target data is "0" indicating "a speech interruption", the data generation unit 11 advances the timing of the second speaker's speech as shown in (3) of Figure 5, and if the speech interruption label is "1" indicating "not a speech interruption", the data generation unit 11 delays the timing of the second speaker's speech as shown in (3) of Figure 6.

[0042] However, the data generating unit 11 may delay the start timing of the utterance of the second speaker in the extended utterance pair data even when the speech interruption label is "0" indicating "a speech interruption," and may advance the start timing of the utterance of the second speaker in the extended utterance pair data even when the speech interruption label is "1" indicating "not a speech interruption." As an example, the data generating unit 11 may calculate a distribution of the difference between the end time of the utterance of the first speaker and the end time of the utterance of the second speaker for each piece of utterance pair data of all training data including the speech interruption label indicating "a speech interruption" in the initial training data set, and delay the start timing of the utterance of the second speaker in the extended utterance pair data so as not to exceed a range based on the distribution. Similarly, the data generating unit 11 may calculate the distribution of the difference between the end time of the utterance of the first speaker and the end time of the utterance of the second speaker for each utterance pair data of all training data in the initial training data set that includes a speech interruption label indicating “not a speech interruption,” and may advance the start timing of the utterance of the second speaker in the extended utterance pair data so as not to exceed the range based on the distribution.

[0043] Here, not exceeding the "range of the distribution" may mean, for example, the range from the minimum value to the maximum value of the distribution, the range of the upper α-percentile of the distribution, or the average value of the distribution.

[0044] Next, the data generating unit 11 associates the generated extended utterance pair data with the speaker labels of the target data, and stores (adds) the extended training data in the training data storage unit 111 (S104).

[0045] It should be noted that a plurality of pieces of extended utterance data, each having a different x1 second or x2 second, may be generated based on one piece of utterance pair data.

[0046] The learning unit 12 learns the speech interruption estimation model 14 using the extended learning data added to the initial learning data set by the above process.

[0047] During inference, the estimation unit 13 uses the trained speech interruption estimation model 14 to estimate whether the input utterance pair data is a speech interruption or not.

[0048] As described above, according to this embodiment, extended training data with expanded turn-taking latency diversity can be added to the initial training dataset. In the extended utterance pair data, the speaker turn duration varies as the utterance start time of the second speaker varies, resulting in different speaker turn-taking timings and overlap times. By varying the transition time of the utterance start time, it is possible to generate a dataset including diverse speaker turn-taking timings and overlap times. Therefore, it is possible to generate a dataset including diverse turn-taking latencies. As a result, it is possible to create a dataset including diverse speaker turn-taking timings and overlap times from a small dataset.

[0049] In the above, an example was given in which audio data in which the first speaker and the second speaker are recorded on different channels was used, but the speech of the two speakers may be included in the same channel as long as it is possible to clearly remove the second speaker from the speech pair data and to synthesize the second speaker's speech with the first speaker data.

[0050] In the present embodiment, the speech interruption estimation device 10 is an example of a data generating device. The utterance pair data is an example of first speech data. The extended utterance pair data is an example of second speech data.

[0051] [Reference 1] Quchen Fu; Szu-Wei Fu; Yaran Fan; Yu Wu; Zhuo Chen; Jayant Gupchup; Ross Cutler, "Real-Time Speech Interruption Analysis: from Cloud to Client Deployment", ICASSP 2023. The above describes in detail the embodiments of the present invention, but the present invention is not limited to such specific embodiments, and various modifications and variations are possible within the scope of the gist of the present invention described in the claims.

[0052] REFERENCE SIGNS LIST 10 Speech interruption estimation device 11 Data generation unit 12 Learning unit 13 Estimation unit 14 Speech interruption estimation model 100 Drive device 101 Recording medium 102 Auxiliary storage device 103 Memory device 104 Processor 105 Interface device 111 Learning data storage unit 112 Extended data storage unit B Bus

Claims

1. A data generating device comprising: a data generating unit configured to generate second voice data in which the timing of the speech of a second speaker in first voice data corresponding to a period of speaker change from a first speaker to a second speaker in a conversation is changed relative to the timing of the speech of the first speaker.

2. The data generating device of claim 1, wherein the first speech data is speech data for training the machine learning model so that the output from the machine learning model to which the first speech data is input approaches a label corresponding to the first speech data indicating whether the alternation is a speech interruption or not, and the data generating unit is configured to change the timing of the speech of the second speaker in the second speech data depending on the label corresponding to the first speech data.

3. A data generation method characterized by being executed by a computer: a data generation procedure for generating second voice data in which the timing of the speech of a second speaker in first voice data corresponding to a period of speaker change from a first speaker to a second speaker in a conversation is changed relatively to the timing of the speech of the first speaker.

4. A program for causing a computer to execute a data generation procedure for generating second voice data in which the timing of the speech of a second speaker in first voice data corresponding to a period of speaker change from a first speaker to a second speaker in a conversation is changed relatively to the timing of the speech of the first speaker.

Images