Speech timing estimation system, dialogue system, and program

Abstract

To provide a speech timing estimation system and dialogue system that can improve accuracy of speech timing estimation and / or efficiency of preparation for the next speech, improving responsiveness of the system and enabling smooth dialogue.SOLUTION: In a dialogue system 10, a sentence generator 31 is provided that uses a recognized token sequence up to the middle of the user utterance obtained by speech recognition process, and generates a completion token sequence by estimating a subsequent token sequence using a language model. And predicted information obtained from the subsequent token sequence or the completion token sequence (at least one of the output of a syntactic completion estimator 33, output or intermediate feature values of a dialog act estimator 40, or an output of language feature value extraction means 32), and acoustic feature values are input to a speech timing estimator 50. In addition, the next speech is prepared using the completion token sequence.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a speech timing estimation system configured by a computer that executes processing to estimate the timing of switching from user utterance to system utterance, a dialogue system configured by a computer that executes processing for voice dialogue with a user, and a program, and can be used in, for example, a news dialogue system that conveys the contents of articles to users using scenario data generated from article data describing various topics such as news, columns, history, etc., a guidance dialogue system that explains how to use devices to users and guides them around facilities, a questionnaire dialogue system that conducts various user trend surveys such as election status and consumer orientation, an information retrieval dialogue system that allows users to search for information on stores, products, travel destinations, songs they want to listen to, etc., an operation dialogue system that allows users to operate various devices and equipment such as home appliances and cars, an education dialogue system that educates users such as children, students, new employees, etc., and an information identification dialogue system that allows a system to identify information such as user attributes, etc. [Background technology]

[0002] In general, a voice dialogue system performs a desired task and achieves its purpose (e.g., conveying the contents of a news article or other article to a user, providing guidance to the user, surveying the user, allowing the user to search for information, operating equipment or the like, educating the user, identifying information by the system, etc.) by exchanging linguistic information between the user (a human user) and the computer system (a voice dialogue system).

[0003] In the conventional voice dialogue system, a series of processes, including voice signal acquisition, speech interval detection, voice recognition, speech content generation, voice synthesis, and voice signal playback, are performed sequentially, and therefore there is a problem that delays accumulate in each process. That is, first, a voice signal of a user's utterance is acquired (voice signal acquisition), and speech interval detection is performed to extract a voice signal of an utterance unit from the continuously obtained voice signal using the interruption of the user's utterance as a clue (utterance interval detection), and then a voice recognition process is performed to convert the obtained voice signal of the speech interval into language information, thereby estimating the meaning of the detected user's utterance (voice recognition), and then the next utterance is determined according to the estimated meaning, that is, the contents of a system utterance suitable for the obtained language information of the user are generated (utterance content generation), and further a voice synthesis process is performed to convert the contents of the utterance into a voice signal (voice synthesis), and then a process is performed to play the generated voice signal of the system utterance in order to convey the contents of the system utterance to the user (voice signal playback). Therefore, the accumulation of delays in each process results in a long delay from when the user finishes speaking until the system responds.

[0004] Therefore, the applicant of the present application has proposed a dialogue system that can improve the responsiveness of the system, while avoiding or suppressing the occurrence of collisions (overlap between a system utterance that has been generated and started to be played back and a continued user utterance) and avoiding or suppressing the occurrence of unnecessarily long system switching latency (the time between the end of a user's utterance and the start of a system utterance) (see Patent Document 1).

[0005] More specifically, in the dialogue system described in Patent Document 1, a pattern recognition process is repeatedly performed to identify whether the user's right to speak, which indicates that the user has a position or status to speak, in a period that is not dependent on the execution timing of the speech recognition process by a speech recognizer (ASR), using acoustic features extracted from the voice signal of the user's utterance and language features extracted from the language information of the user's utterance obtained as a result of the speech recognition process, and the start timing of the system utterance is detected using the result of this pattern recognition process. In addition, a preparatory process is performed to obtain or generate content data for the system's next utterance using material data prepared in advance or material data obtained via a network, in a period that is not dependent on the period of the pattern recognition process for detecting the system utterance timing, and before the start timing of the system utterance is detected by this system utterance timing detection, using at least a part of the dialogue history information between the user and the system and / or the result of the speech recognition process up to the middle of the ongoing user utterance by the speech recognizer. These improve the responsiveness of the system.

[0006] Also, a document on speech timing estimation by the inventors of the present application is known (see Non-Patent Document 1). However, Non-Patent Document 1 does not include any description of a sentence generator (a technology for estimating a subsequent token string from a recognized token string and generating a complementary token string) or a syntactic completion estimator (a technology for calculating a probability distribution or a cumulative probability distribution indicating how many tokens or how long it will take for an utterance to be completed) according to the present invention, which will be described later.

[0007] There are two approaches to speech timing estimation: one is the timing estimation method synchronized with the detection of the end of utterance (EoU) (ES-TD) and the other is the timing estimation method synchronized with the analysis frame (FS-TD).

[0008] The former, ES-TD, is a method for estimating the timing of switching from user speech to system speech in a conventional voice dialogue system that performs the above-mentioned series of processes sequentially, first detecting the end of the user speech, then judging whether the user is holding or releasing the turn (right to speak), and if it is judged that the user has released the turn, the system starts speaking after a certain time. As a turn-taking identification method, the Gated Multimodal Fusion (GMF) model by Jiudong Yang et al., which combines prosodic information, time information, and linguistic information, is known (see Non-Patent Document 2), but this is also ES-TD.

[0009] The latter FS-TD is a method for determining whether or not to start system utterance for each analysis frame. The method for detecting system utterance timing in a dialogue system by the applicant of the present application described in the above-mentioned Patent Document 1 and the method described in Non-Patent Document 1 by the inventors of the present application are FS-TD.

[0010] Furthermore, experiments are being conducted to confirm the effectiveness of the present invention, and the Harper Valley Bank corpus is known as data that can be used for the experiments (see Non-Patent Document 3).

[0011] Furthermore, the present invention can be applied to various dialogue systems, such as a news dialogue system, a guidance dialogue system, a questionnaire dialogue system, an information retrieval dialogue system, an operation dialogue system, and an educational dialogue system. As a dialogue system capable of realizing efficient information transmission to a user, a news dialogue system developed by the present inventors that transmits the contents of news articles, etc. to a user using scenario data consisting of a main plan and a sub-plan is known (see non-patent document 4).

[0012] Furthermore, as will be described later, when the present invention is combined with a low-latency speech recognizer, the accuracy of speech timing estimation can be further improved, or the system's preparation process for the next utterance can be made more efficient. One example of a low-latency speech recognizer is the multi-look-ahead speech recognizer (MLA-ASR) developed by the present inventors (see non-patent document 5).

[0013] In addition, HuBERT (Hidden unit BERT) is known as a technique that can be used to extract acoustic features in the present invention (see Non-Patent Document 6), and BERT is known as a technique that can be used to extract language features in the present invention (see Non-Patent Document 7). [Prior art documents] [Patent documents]

[0014] [Patent Document 1] Patent No. 7274210 Publication [Non-patent literature]

[0015] [Non-Patent Document 1] Shinya Fujie, Hayato Katayama, Jin Sakuma, Tetsunori Kobayashi, “Timing Generating Networks: Neural Network Based Precise Turn-Taking Timing Prediction in Multiparty Conversation,” Interspeech 2021, Sep. 2021. [Non-Patent Document 2] Jiudong Yang, Peiying Wang, Yi Zhu, Mingchao Feng, Meng Chen, and Xiaodong He, "Gated multimodal fusion with contrastive learning for turn-taking prediction in human-robot dialogue," ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7747-7751, 2022. [Non-Patent Document 3] M. Wu, J. Nafziger, A. Scodary, and AL Maas, "Harpervalleybank: A domain-specific spoken dialog corpus," ArXiv, vol. abs / 2010.13929, 2020. [Non-Patent Document 4] Hiroaki Takatsu, Ishin Fukuoka, Shinya Fujie, Yoshihiko Hayashi, Tetsunori Kobayashi, "A Speech Dialogue System Enabling Diverse Information Behavior with Different Intentions," Journal of the Japanese Society for Artificial Intelligence, vol.22, no.1, p.DSH-C_1-24, 2018 [Non-Patent Document 5] Huaibo Zhao, Shinya Fujie, Tetsuji Ogawa, Jin Sakuma, Yusuke Kida, Tetsunori Kobayash, "Conversation-oriented ASR with Multi-Look-Ahead CBS Architecture," [online], Cornell University Library Archive Site (Date and time of upload to preprint server: Wed, 2 Nov 2022 (Wednesday, November 2, 2022) 03:58:56 UTC (Coordinated Universal Time)), [Retrieved May 31, 2023], Internet<URL:https: / / arxiv.org / abs / 2211.00858v1> [Non-Patent Document 6] W.-N. Hsu, B. Bolte, Y.-HH Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, "Hubert: Self-supervised speech representation learning by masked prediction of hidden units," IEEE / ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3451-3460, 2021. [Non-Patent Document 7] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, "Bert: Pretraining of deep bidirectional transformers for language understanding," ArXiv, Vol. abs / 1810.04805, 2019. Summary of the Invention [Problem to be solved by the invention]

[0016] As mentioned above, there are two approaches to utterance timing estimation: ES-TD and FS-TD. The former, ES-TD, has the problem that it takes time to detect the end of utterance (EoU), resulting in poor responsiveness. In addition, it has the problem that it is strongly affected by the detection accuracy of the EoU. Furthermore, it has the problem that it cannot take into account events after the EoU (e.g., changes in gaze, etc.). In addition, it has the problem that it is difficult to control the time from the end of the user's utterance to the start of the system utterance. In contrast, the latter, FS-TD, can solve these problems by determining whether or not to start the system utterance for each analysis frame.

[0017] However, even in the latter FS-TD where a judgment is made for each analysis frame, when acoustic features obtained from the user's voice signal and linguistic features obtained from the voice recognition result (linguistic information) are used as input data, there is a delay in the voice recognition process, so if timing estimation is performed after waiting for the voice recognition result so that both features correspond, the response timing will inevitably be delayed. On the other hand, if timing estimation is performed without waiting for the voice recognition result, judgment will be made for each frame using incomplete voice recognition results obtained up to that point (linguistic information that does not include the recognition result for the end of the user's utterance) obtained at each moment, and the accuracy of the speech timing estimation will not improve accordingly.

[0018] Furthermore, if speech recognition processing with less delay could be realized, the problem of insufficient language information due to the delay as described above would be resolved. However, if the user is in the middle of speaking and intends to continue speaking, it is impossible to perform speech recognition processing on information that has not yet been uttered (information that has not become a speech signal), so in the first place, FS-TD performs timing estimation based on incomplete speech recognition results (language information) that have not yet reached the end of the speech for much of the time. Therefore, taking this into consideration, the delay in speech recognition processing seems to have a small effect, but it is not, and its effect becomes greater as the end of the speech approaches. In other words, since it is clear that the user is still speaking in the middle of the speech that is not close to the end of the speech (a time period much earlier than the end of the speech), timing estimation based on incomplete speech recognition results is sufficient, and a correct estimation result that the user is maintaining the right to speak can be obtained. On the other hand, near the end of the speech, the likelihood (output of the speech timing estimator) indicating the maintenance / end of the user's right to speak approaches a threshold value, so the effect of the delay in speech recognition processing becomes greater.

[0019] Furthermore, in a dialogue system, if the speech timing estimator determines that it is time to start a system utterance, the system will start speaking, so it is necessary to prepare the content data of the system's next utterance in advance. In this case, in the dialogue system by the applicant described in the above-mentioned Patent Document 1, preparation processing for the next utterance is performed using an incomplete voice recognition result (a delay in the voice recognition process and a recognition result up to the middle of the utterance). Therefore, in this respect, the accuracy of the preparation processing for the next utterance cannot be improved, and the narrowing down of the next utterance candidates is delayed.

[0020] An object of the present invention is to provide a speech timing estimation system, a dialogue system, and a program that can improve the accuracy of speech timing estimation and / or the efficiency of preparation for the next utterance, improve the responsiveness of the system, and realize smooth dialogue. [Means for solving the problem]

[0021] The present invention achieves the above-mentioned object by estimating a subsequent token sequence constituting a portion following the recognized token sequence for the user utterance by a language model such as Generative Pre-trained Transformer 2, 3 or 4 (hereinafter abbreviated as "GPT-2, 3 or 4") using a recognized token sequence constituting a portion of the user utterance obtained by speech recognition processing, or by providing a sentence generator that generates a complementary token sequence by adding the recognized token sequence to the subsequent token sequence in addition to this estimation, and by performing an utterance timing estimation process and / or a next utterance preparation process for a voice dialogue using the subsequent token sequence and the complementary token sequence. Specifically, the process is as follows.

[0022] <Invention of speech timing estimation system / Configuration for estimating speech timing using prediction information obtained from subsequent token strings or complementary token strings>

[0023] The present invention provides an utterance timing estimation system configured by a computer that executes a process of estimating a timing of a change from a user utterance to a system utterance, A sentence generator that uses a recognized token sequence constituting a portion of the user utterance obtained by the speech recognition process to estimate a subsequent token sequence constituting a portion following the recognized token sequence for the user utterance using a Generative Pretrained Transformer 2, 3, or 4 (GPT-2, 3, or 4) or other language model, or generates a complementary token sequence by adding the subsequent token sequence to the recognized token sequence in addition to this estimation; an utterance timing estimator that receives as input data information including prediction information obtained from the subsequent token sequence estimated by the sentence generator or the generated complementary token sequence and acoustic features obtained from a voice signal of a user utterance, and executes a pattern recognition process to estimate a timing to start the next system utterance, The prediction information to be input to this speech timing estimator is (a) providing a syntax completion estimator that uses a sequence of multiple subsequent tokens estimated by the sentence generator to calculate a probability distribution of the number of speech completion tokens or the speech completion time length, which indicates how many tokens or how much time is required to complete the utterance, or a cumulative probability distribution that accumulates the probability of the number of speech completion tokens or the speech completion time length, and providing vector data of the probability distribution or cumulative probability distribution of the number of speech completion tokens or the speech completion time length calculated by the syntax completion estimator; (b) providing a dialogue act estimator that receives as input data information including linguistic features obtained from the complement token sequence generated by the sentence generator and acoustic features obtained from the speech signal of the user utterance, and executes pattern recognition processing to estimate a dialogue act of the user utterance, or to estimate a dialogue act of the next system utterance in addition to the dialogue act of the user utterance, and (c) Linguistic features obtained from the complemented token sequence generated by the sentence generator and The present invention is characterized in that the information is at least one of the above.

[0024] Here, the description that the "prediction information" is "at least one piece of information" among (a), (b), and (c) means that it may be any one piece of information among (a), (b), and (c), or a combination of two or more pieces of information.

[0025] In the speech timing estimation system of the present invention, the speech timing is estimated using prediction information obtained from the subsequent token string or the complementary token string, so that the accuracy of the speech timing estimation is improved.

[0026] More specifically, when the information (a), i.e., the output of a syntax completion estimator using multiple subsequent token strings (vector data of the probability distribution or cumulative probability distribution of the number of utterance completion tokens or utterance completion time length obtained by the syntax completion estimator) is input to the speech timing estimator as prediction information, the accuracy of speech timing estimation is improved.

[0027] Furthermore, when the information in (b), i.e., the estimation result of the dialogue act estimator using the language features obtained from the complementary token sequence or its intermediate features (the estimation result of the dialogue act of the user utterance or, in addition, the estimation result of the dialogue act of the next system utterance, or intermediate features leading to these estimation results) is input to the speech timing estimator as prediction information, the accuracy of the speech timing estimation is improved. More precisely, the accuracy of the speech timing estimation is improved simply by using the dialogue act information (the estimation result or intermediate features), but the accuracy of the dialogue act estimation is further improved by using the complementary token sequence, so that the accuracy of the speech timing estimation is further improved.

[0028] Furthermore, when the information (c) is input as prediction information to the speech timing estimator, that is, the linguistic features obtained from the complemented token sequence generated by the sentence generator, the accuracy of speech timing estimation is improved.

[0029] For this reason, regardless of which of (a), (b), and (c) is used as the predictive information to estimate the timing of speech, the responsiveness of the system can be improved and smooth dialogue can be achieved. Furthermore, when two or more of (a), (b), and (c) are used in combination, even greater effects can be obtained, thereby achieving the above-mentioned objective.

[0030] <Invention of speech timing estimation system / Configuration with means for measuring elapsed time after user speech ends>

[0031] In addition, in the case where the speech timing is estimated using prediction information obtained from the following token sequence or the complementary token sequence, a user utterance end elapsed time measuring means for measuring an elapsed time from the end of the user utterance or an alternative value indicating the length of the elapsed time by using a voice signal of the user utterance, It is preferable that the input data to the speech timing estimator includes the elapsed time from the end of the user's utterance measured by the means for measuring elapsed time from the end of the user's utterance or a substitute value thereof.

[0032] In this manner, when a configuration is provided in which a means for measuring the elapsed time since the end of a user's utterance is provided, the accuracy of estimating the timing of the utterance is further improved, thereby further improving the responsiveness of the system and enabling smooth dialogue to be realized.

[0033] <Dialogue system invention / Configuration equipped with the speech timing estimation system of the present invention>

[0034] The present invention provides a dialogue system configured by a computer that executes processing for a voice dialogue with a user, The speech timing estimation system of the present invention described above, A voice signal acquisition means for acquiring a voice signal of a user's speech; a voice recognizer that executes a voice recognition process on the voice signal of the user's speech acquired by the voice signal acquisition means; a next utterance preparation means for executing a preparation process for acquiring or generating content data for the next utterance of the system, using subject data stored in the subject data storage means or subject data stored in an external system connected via a network, at a timing that is not dependent on the period of a pattern recognition process by an utterance timing estimator constituting the utterance timing estimation system and before the start timing of a system utterance is detected by the utterance timing estimator, and using a complementary token sequence generated by a sentence generator constituting the utterance timing estimation system, or using at least a part of dialogue history information between a user and the system in addition to the complementary token sequence; an utterance generating means for executing a system utterance generating process including reproducing a voice signal of the system utterance, using content data of the next utterance obtained by the preparation process by the next utterance preparing means, after the start timing of the system utterance is detected by the utterance timing estimator; The present invention is characterized by comprising:

[0035] In such a dialogue system of the present invention, since the speech timing estimation system of the present invention is used, the accuracy of speech timing estimation is improved, so that the responsiveness of the system can be improved and smooth dialogue can be realized. Also, since the content data of the next utterance of the system is prepared using the complementary token sequence, the efficiency of the next utterance preparation process is improved, so that in this respect, the responsiveness of the system can be improved and smooth dialogue can be realized, thereby achieving the above-mentioned object.

[0036] In the above-mentioned dialogue system, The next utterance preparation means is A preparation process is performed to obtain or generate content data of a plurality of next utterance candidates, A configuration may be adopted which includes a next utterance selection means which performs a process of selecting content data of the next utterance to be used by the utterance generation means from content data of multiple next utterance candidates obtained by the preparation process by the next utterance preparation means using the complementary token sequence generated by the sentence generator (since the user utterance has ended, it may ultimately become a recognized token sequence for the entire user's speech section, i.e., there may be no subsequent token sequence) after the speech timing estimator detects the start timing of the system utterance.

[0037] In addition, in the case where the above-mentioned next utterance selection means is provided, When the information (b) is used as prediction information to be input to the speech timing estimator, i.e., the estimation result of the dialogue act estimator or its intermediate features (including the case where it is used in combination with the information (a) or (c)), The next utterance selection means may be configured to execute a process of selecting content data of the next utterance to be used by the utterance generation means from content data of multiple next utterance candidates obtained by preparation processing by the next utterance preparation means, using a combination of the estimated result of the dialogue act of the user utterance obtained by processing by the dialogue act estimator, or in addition, the estimated result of the dialogue act of the next system utterance, and a complementary token sequence generated by the sentence generator.

[0038] <Invention of dialogue system / Configuration not limited to use of the speech timing estimation system of the present invention>

[0039] The present invention also provides a dialogue system configured by a computer that executes processing for a voice dialogue with a user, comprising: A voice signal acquisition means for acquiring a voice signal of a user's speech; a voice recognizer that executes a voice recognition process on the voice signal of the user's speech acquired by the voice signal acquisition means; a sentence generator that uses a recognized token sequence constituting a portion of the user's utterance obtained by the speech recognition process of the speech recognizer to estimate a subsequent token sequence constituting a portion following the recognized token sequence for the user's utterance using a Generative Pretrained Transformer 2, 3, or 4 (GPT-2, 3, or 4) or other language model, and generates a complementary token sequence by adding the subsequent token sequence to the recognized token sequence; a next utterance preparation means for executing a preparation process for acquiring or generating content data for a next utterance of the system by using the subject data stored in the subject data storage means or the subject data stored in an external system connected via a network, and using a complement token string generated by a sentence generator, or by using the complement token string and at least a part of dialogue history information between the user and the system; an utterance generating means for executing a system utterance generating process including reproducing a voice signal of the system utterance by using the content data of the next utterance obtained by the preparation process by the next utterance preparing means; The present invention is characterized by comprising:

[0040] In such a dialogue system of the present invention, it is not necessary to use the speech timing estimation system of the present invention; that is, the start timing of the system utterance may be determined by conventional methods (including the method in the dialogue system by the applicant of the present application described in the above-mentioned Patent Document 1), and therefore the improvement in accuracy achieved by the speech timing estimation system of the present invention cannot be obtained. However, since the content data for the system's next utterance is prepared using the complementary token sequence, the efficiency of the next utterance preparation process is improved. In this respect, the responsiveness of the system can be improved and smooth dialogue can be realized, thereby achieving the above-mentioned objective.

[0041] In the above-mentioned dialogue system, The next utterance preparation means is A preparation process is performed to obtain or generate content data of a plurality of next utterance candidates, A configuration may be adopted which includes a next utterance selection means which performs a process of selecting content data of the next utterance to be used by the utterance generation means from content data of multiple next utterance candidates obtained by the preparation process by the next utterance preparation means, using the complementary token sequence generated by the sentence generator (since the user's utterance has ended, this may ultimately become a recognized token sequence for the entire user's speech section, i.e., there may be no subsequent token sequence). This allows for appropriate next utterance selection.

[0042] In addition, in the case where the above-mentioned next utterance selection means is provided, a dialogue act estimator that receives as input data information including linguistic features obtained from a complement token sequence generated by the sentence generator and acoustic features obtained from a speech signal of a user utterance, and executes pattern recognition processing to estimate a dialogue act of the user utterance, or to estimate a dialogue act of a next system utterance in addition to the dialogue act of the user utterance; The next utterance selection means may be configured to execute a process of selecting content data of the next utterance to be used by the utterance generation means from content data of multiple next utterance candidates obtained by preparation processing by the next utterance preparation means, using a combination of the estimated result of the dialogue act of the user utterance obtained by processing by the dialogue act estimator, or in addition, the estimated result of the dialogue act of the next system utterance, and a complementary token sequence generated by the sentence generator. This allows for a more appropriate selection of the next utterance.

[0043] <Program invention>

[0044] The program of the present invention is for causing a computer to function as the above-described speech timing estimation system. Also, the program of the present invention is for causing a computer to function as the above-described dialogue system.

[0045] In addition, these programs or parts thereof can be recorded and stored or distributed on recording media such as magneto-optical disks (MO), compact disks (CD), digital versatile disks (DVD), flexible disks (FD), magnetic tapes, read-only memories (ROM), electrically erasable and programmable read-only memories (EEPROM), flash memories, random access memories (RAM), hard disk drives (HDD), solid-state drives (SSD), flash disks, etc., and can be transmitted using transmission media such as wired networks such as local area networks (LAN), metropolitan area networks (MAN), wide area networks (WAN), the Internet, intranets, extranets, etc., wireless communication networks, or combinations of these networks, or can be carried on carrier waves. Furthermore, these programs may be parts of other programs, or may be recorded on recording media together with separate programs. Effect of the Invention

[0046] As described above, according to the present invention, a sentence generator is provided that uses a recognized token sequence to estimate a subsequent token sequence using a language model such as GPT-2, 3 or 4, or in addition to this estimation, generates a complementary token sequence by adding the subsequent token sequence to the recognized token sequence, and performs utterance timing estimation processing and / or next utterance preparation processing for voice dialogue using the subsequent token sequence and the complementary token sequence, thereby achieving the effect of improving the accuracy of utterance timing estimation and / or the efficiency of next utterance preparation, improving system responsiveness, and realizing smooth dialogue. [Brief description of the drawings]

[0047] [Figure 1] 1 is a diagram showing the overall configuration of a dialogue system including an utterance timing estimation system according to an embodiment of the present invention; [Diagram 2] FIG. 4 is a configuration diagram of an utterance timing estimator according to the embodiment. [Diagram 3] FIG. 4 is a configuration diagram of a dialogue act estimator according to the embodiment. [Figure 4] 5A and 5B are explanatory diagrams of a complementary token string and a subsequent token string according to the embodiment. [Diagram 5] FIG. 4 is a diagram showing an image of estimation of a subsequent token string according to the embodiment. [Figure 6] FIG. 4 is an explanatory diagram of a method for creating a subsequent token string according to the embodiment. [Figure 7] FIG. 4 is a diagram illustrating an example of an output value of the syntax completion estimator according to the embodiment. [Figure 8] FIG. 11 is a flowchart showing a process flow when a speaker change is made from a user to the system in the embodiment. [Figure 9] FIG. 4 is an explanatory diagram of the definition of precision, which is an evaluation index according to the embodiment. [Figure 10] FIG. 4 is an explanatory diagram of the definition of recall, which is an evaluation index according to the embodiment. [Figure 11] 11A and 11B are diagrams showing evaluation results of speech timing estimation when the dialogue act estimator of the embodiment is used. [Figure 12]FIG. 4 is a diagram showing an evaluation result (1) of the speech timing estimation according to the embodiment. [Figure 13] FIG. 4 is a diagram showing an evaluation result (1) of the dialogue act estimation according to the embodiment. [Figure 14] FIG. 11 is a diagram showing an evaluation result (2) of the speech timing estimation according to the embodiment. [Figure 15] FIG. 11 is a diagram showing an evaluation result (2) of the dialogue act estimation according to the embodiment. [Figure 16] FIG. 13 is a diagram showing evaluation results of speech timing estimation when a syntactic completion estimator and a low-delay speech recognizer are used. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0048] An embodiment of the present invention will be described below with reference to the drawings. FIG. 1 shows the overall configuration of a dialogue system 10 including an utterance timing estimation system 20 of this embodiment. FIG. 2 shows the configuration of an utterance timing estimator 50, and FIG. 3 shows the configuration of a dialogue act estimator 40. FIG. 4 is an explanatory diagram of a complementary token string and a subsequent token string, FIG. 5 is a diagram showing an image of estimating a subsequent token string, and FIG. 6 is an explanatory diagram of a method of creating a subsequent token string. FIG. 7 is an exemplary diagram of an output value of a syntax completion estimator 33. FIG. 8 shows a flow chart of a process flow when a speaker is changed from a user to a system. FIGS. 9 and 10 are explanatory diagrams of the definitions of precision and recall, which are evaluation indices. FIGS. 11 to 16 show various evaluation results of utterance timing estimation and dialogue act estimation.

[0049] (Overall composition) 1, a dialogue system 10 is configured with one or more computers, and includes an utterance timing estimation system 20. The utterance timing estimation system 20 is configured with one or more computers, and includes a speech signal acquisition means 21, a time elapsed since the end of a user utterance measurement means 22, an acoustic feature extraction means for utterance timing estimation 23, an acoustic feature extraction means for dialogue act estimation 24, a speech recognizer (ASR) 30, a sentence generator 31, a language feature extraction means 32, a syntax completion estimator 33, a dialogue act estimator 40, and an utterance timing estimator 50.

[0050] In addition to the speech timing estimation system 20, the dialogue system 10 also includes a next utterance preparation means 60, a next utterance selection means 61, an utterance generation means 62, a subject data storage means 70, a next utterance candidate storage means 71, and a dialogue history storage means 72.

[0051] Here, each of the processing units (including a processing unit that executes arithmetic processing according to an algorithm by a neural network and a processing unit that executes frequency analysis) of the elements (elements described as "means" or "device", but excluding an element described as "storage means") 21, 22, 23, 24, 30, 31, 32, 33, 40, 50, 60, 61, 62 that constitute the dialogue system 10 are composed of an arithmetic processing device (CPU, GPU, etc.) provided inside the computer, one or more programs that define the operation procedures, and working memories such as main memory and cache memory. Also, each storage unit (including a storage unit for neural network parameters) that stores data necessary for these arithmetic processing is composed of the main memory during calculation, and is composed of non-volatile memory such as HDD and SSD when the computer is not operating.

[0052] Furthermore, the subject data storage means 70 stores data prepared in advance, and is therefore configured with a non-volatile memory such as an HDD or SSD, and during calculation, data required for the calculation process at that time (a portion of data according to the purpose of the voice dialogue) is placed in the main memory. The next utterance candidate storage means 71 and the dialogue history storage means 72 store data used temporarily during calculation, and are therefore configured with a main memory, but may be stored in a non-volatile memory such as an HDD or SSD.

[0053] Since the dialogue system 10 engages in voice dialogue with the user, at least a part of the voice signal acquisition means 21 (the microphone and camera parts) and at least a part of the speech generation means 62 (the speaker and display parts) are installed in the vicinity of the user, but the other components may be installed in remote locations and connected via a network such as the Internet.

[0054] (Configuration of the audio signal acquiring means 21) The voice signal acquisition means 21 acquires a voice signal uttered by the user, and is configured to include a microphone which is a device for converting sound (here, voice) into an analog electric signal, A / D conversion means, voice signal storage means for holding the digital voice signal obtained by the A / D conversion, transmission means for transmitting the voice signal to various places, etc. In addition, as an associated function, it may be equipped with a camera for acquiring a face image of the user, etc.

[0055] (Configuration of the means 22 for measuring time elapsed after user's speech) The elapsed time after end of user utterance measurement means 22 executes a process of determining the end of the user utterance using the audio signal of the user utterance and measuring the elapsed time from the end of the utterance (ETAU) or an alternative value indicating the length of this elapsed time.

[0056] Here, the elapsed time also includes a time feature (a feature representing the elapsed time). The time elapsed after the end of the speech is detected and the length of the previous user speech can be used as the time feature. A model (hereinafter referred to as a VAD model) that estimates whether the user is speaking or not is constructed by pre-learning, and the user's speech state is estimated at each time by this VAD model, and the cumulative sum of the likelihood indicating the probability that the user is not speaking is used as the feature representing the elapsed time from the end of the speech. For this VAD model, an LSTM-based model is used, and the bottleneck feature of the CNN autoencoder (for details, see the configuration of the speech timing estimation acoustic feature extraction means 23 described later) is input. The cumulative sum is reset to 0 when the user starts speaking. In addition, based on the estimation result of the VAD model, the elapsed time from the start of the user's speech and the elapsed time from the start of the IPU of the current user are also used as features. Then, these are combined and a linear layer is applied to obtain the time feature. An IPU (Inter-Pausal Unit) is a speech section separated by a pause of a specific length (for example, 100 milliseconds or more), and is widely used as a unit of speech section in speech analysis and conversation analysis. However, silent sections of around 100 milliseconds frequently occur within the speech section of a single speaker, so a speaker change does not necessarily occur before or after it.

[0057] In addition, when using an alternative value, a numerical value that increases as the elapsed time increases can be adopted, and this numerical value can be a numerical value proportional to the elapsed time. However, it is not necessarily required to be proportional. For example, a count number (one-dimensional data) corresponding to the period of the pattern recognition processing by the speech timing estimator 50 can be adopted, and during the user's speech, ETAU=0,0,0,... continues, and the processing time when it is determined that the speech is over is the final 0, and at the arrival of the next period, ETAU=1, and thereafter, ETAU=2,3,4,5,... may be increased. Then, at the start of the next user's speech, it becomes 0 again. In addition, the count number may be increased every time a frame (for example, a frame used in the processing by the speech timing estimation acoustic feature extraction means 23) is shifted when extracting acoustic features from a speech signal (that is, at a period shorter than the period of the pattern recognition processing by the speech timing estimator 50).

[0058] (Configuration of the speech timing estimation acoustic feature extraction means 23) The speech timing estimation acoustic feature extraction means 23 executes a process of extracting acoustic features for estimating speech timing from the audio signal of the user's speech acquired by the audio signal acquisition means 21. Any acoustic feature may be used as long as it is obtained from the audio signal of the user's speech, and for example, the technology described in the above-mentioned Patent Document 1 by the applicant of the present application may be adopted. However, the technology is not limited to this technology.

[0059] That is, an autoencoder (autoencoder) that encodes and decodes narrowband spectrograms is constructed using a neural network (CNN), and the output of the intermediate layer is used as the acoustic feature (bottleneck feature). Specifically, the input is an arrangement of multiple (e.g., 10) 256-point power spectra obtained, for example, every 10 ms by frequency analysis of the audio signal, and the 256-dimensional output of the intermediate layer is used as the feature. That is, the input of the CNN autoencoder is, for example, a time-series arrangement of 10 spectrograms generated from an audio signal cut out with a frame size of 800 samples (50 ms) and a frame shift of 160 samples (10 ms), and the size is 10 x 256 dimensions. Then, this input size is compressed to 256 dimensions by the CNN autoencoder to use it as the acoustic feature. In addition to this, an 80-dimensional log-mel spectrogram extracted with, for example, a 25 ms window and a 10 ms shift may be used. After adjusting the frame rate of this log-mel spectrogram to 50 ms in the CNN layer, it may be concatenated with the bottleneck features of the above-mentioned CNN autoencoder to obtain acoustic features for speech timing estimation.

[0060] (Configuration of the dialogue act estimation acoustic feature extraction means 24) The dialogue act estimation acoustic feature extraction means 24 executes a process of extracting acoustic features for estimating a dialogue act from the voice signal of the user's speech acquired by the voice signal acquisition means 21 .

[0061] Specifically, for example, it is possible to use HuBERT having a CNN layer and a transformer described in the above-mentioned Non-Patent Document 6. Note that it is also possible to employ a configuration similar to the above-mentioned speech timing estimation acoustic feature extraction means 23 (for example, a configuration in which a 256-dimensional bottleneck feature extracted from the intermediate layer of a CNN autoencoder is used as the acoustic feature). Therefore, the dialogue act estimation acoustic feature extraction means 24 and the speech timing estimation acoustic feature extraction means 23 can be made common.

[0062] (Configuration of the speech recognizer 30) The speech recognizer 30 executes a process of outputting a token sequence corresponding to a speech signal of a user utterance acquired by the speech signal acquisition means 21. That is, the speech recognizer 30 executes a process of extracting acoustic features from the speech signal, encoding the acoustic features by an encoder, sending the encoded code to a decoder, and outputting a token sequence. In this case, the acoustic features extracted by the dialogue act estimation acoustic feature extraction means 24 may be used as the acoustic features (dotted line in FIG. 1).

[0063] Any speech recognizer can be used as the speech recognizer 30 as long as it can sequentially output a token sequence of a user's speech. For example, an end-to-end streaming speech recognizer configured by a neural network can be used. Specifically, a speech recognizer can be used that inputs the obtained acoustic features to a linear layer and outputs a token sequence of a user's speech by a recurrent neural network (RNN)-based CTC (Connectionist temporal classification) by Alex Graves et al. In addition, a multi-look-ahead speech recognizer (MLA-ASR) by the present inventors described in the above-mentioned Non-Patent Document 5 may be used. This MLA-ASR is a CSB transducer (Contextual block streaming transducer) that performs block processing using a context vector. When the MLA-ASR is used, low-delay speech recognition processing can be realized, and the accuracy of speech timing estimation is improved as described later.

[0064] Here, a token is a division unit of linguistic information that is conveniently determined by designers and researchers of speech recognizers for speech recognition, and is coarser than a phoneme. In the case of English, tokens often have a one-to-one relationship with English words (such as this, will, us, etc.), but in order to reduce unknown words and vocabulary, tokens are often associated with subwords (such as ing, er, ly, etc.), and conversely, consecutive words may be associated with tokens, and further, tokens may be associated with subwords indicating the beginning or end of a sentence.<SOS(Start of Sentence) / EOS(End of Sentence)> There are also tokens, which are symbols containing text (character information). In the case of Japanese, tokens correspond to characters (hiragana characters, such as "a", "i", "u", "ba", "pa", etc.) or character strings (such as "dewa", "shikashi"), and are often one or two characters long, but tokens may also contain kanji characters. If kanji characters are not included, conversion from kanji characters to kanji characters can be performed separately. In this way, tokens are conveniently divided for speech recognition, and the present invention can be applied to tokens divided in any way.

[0065] (Configuration of the sentence generator 31: Figures 4, 5, and 6) The sentence generator 31 uses the recognized token sequence constituting the intermediate portion of the user utterance obtained by the speech recognition process by the speech recognizer 30 to estimate a subsequent token sequence constituting the portion following the recognized token sequence for the user utterance by a language model, and executes a process of generating a complementary token sequence by adding the subsequent token sequence to the recognized token sequence. Thus, the sentence generator 31 has a role of supplementing the token sequence partially obtained by speech recognition up to the end of the sentence.

[0066] As shown in FIG. 4, when there is a delay in speech recognition by the speech recognizer 30, speech recognition results are not obtained for the entire portion of the speech that the user has finished speaking (speech signal obtained up to the current time t), and a recognized token sequence and a speech recognition delay portion indicated by diagonal lines in the figure are generated. If a low-delay speech recognizer 30 is used, the speech recognition delay portion is shortened, but in the strict sense, the delay is not zero and a delay always occurs. Also, for the portion to be spoken (speech signal obtained by utterance after the current time t), the speech recognition process is not performed as a matter of course. Therefore, the sentence generator 31 estimates a token sequence that combines the token sequence corresponding to the speech recognition delay portion and the token sequence corresponding to the portion to be spoken as the subsequent token sequence.

[0067] Any language model can be adopted as long as it can estimate a subsequent token sequence from a recognized token sequence. Many language models are composed of neural networks, but language models other than neural networks may also be used. Specifically, in this embodiment, two models are prepared as the sentence generator 31: a Long short-term memory (LSTM) and a Generative Pre-trained Transformer 2 (GPT-2). GPT-2 is a model composed of multiple overlapping decoder parts of a transformer, and it estimates the immediately preceding token sequence (S1, ..., S n-1 ) followed immediately by token S n That is, we estimate the token sequence (S1,…,S n-1 ) to get the token S n The task is to find the distribution of the conditional probability that the output is a GPT. Note that other versions of GPT, such as GPT-3 or 4, or versions developed subsequently, may be used.

[0068] The sentence generator 31 is trained by adding an end-of-sentence (EoS) token to the end of a transcribed sentence in a training dataset. In the present application, the EoS token is also called an end-of-utterance (EoU) token. During operation (inference), the sentence generator 31 generates sentences until an end-of-sentence token appears, that is, estimates a subsequent token string. This makes it possible to generate sentences up to the end of the sentence (end of utterance), that is, generate a complementary token string. When an end-of-sentence token (end-of-utterance token) does not appear, the upper limit of token generation is set to 10 tokens, for example, in this embodiment.

[0069] As shown in Fig. 5 and Fig. 6, the sentence generator 31 estimates a plurality of subsequent token sequences. The example in Fig. 5 shows the timing at which the utterance "How about XX?" ends at the current time t (the timing when the next EoU token comes). At this time, if there are three tokens in the speech recognition delay portion, the recognized token sequence will be "How about XX?" and the subsequent token sequence will be "Deshou ka?". <eou>"," "Is it <eou>" "Is that something? <eou>5 shows an example of the timing when the utterance has just ended at the current time t, but the same applies to the midway stage of the utterance.

[0070] The example in Figure 6 shows a method for extending the token string one token at a time, by selecting multiple tokens (two in this figure as an example) with high probability when the next token is output together with its probability, and creating multiple token strings. If the probability that the next token will be token A is 0.6, the probability that it will be token B is 0.4, and the probability that the next token after token A will be an EoU token is 0.8, and the probability that it will be token D is 0.2, then the token string will be "A+ <eou>The probability of the token sequence being "B+E" is 0.4×0.3=0.12, and the probability of the token sequence being "B+F" is 0.4×0.7=0.28. Therefore, for the four token sequences, "A+ <eou>"," "A+D," "B+E," and "B+F" are selected in descending order of probability, and a predetermined number (two in this example) of token strings are selected. <eou>" or "B+F".

[0071] And, "A+ <eou>" cannot be extended any further, so the token sequence "B+F" is extended. If the probability that the next token in the token sequence "B+F" will be token I is 0.3 and the probability that it will be token J is 0.7, then the probability that the token sequence will be "B+F+I" is 0.28×0.3=0.08, and the probability that it will be "B+F+J" is 0.28×0.7=0.20. Since there is no EoU, these token sequences are extended further. If the probability that the next token in the token sequence "B+F+I" will be token K is 0.6 and the probability that it will be token L is 0.4, then the probability that the token sequence will be "B+F+I+K" is 0.08×0.6=0.05, and the probability that it will be "B+F+I+L" is 0.08×0.4=0.04. In addition, if the probability that the next token in the token sequence "B+F+J" will be token M is 0.3 and the probability that it will be an EoU token is 0.7, then the probability that the token sequence will be "B+F+J+M" is 0.20 x 0.3 = 0.06, and the probability that it will be "B+F+J+ <eou>" is 0.20 × 0.7 = 0.14. Therefore, for the four token sequences, "B+F+I+K", "B+F+I+L", "B+F+J+M", and "B+F+J+ <eou>Among them, a predetermined number of token sequences (two in this example) are selected in descending order of probability, and the following are selected: "B+F+J+M", "B+F+J+ <eou>". Two subsequent token sequences are estimated, and the most probable subsequent token sequence (first candidate) is "A+ <eou>" and the second candidate is "B+F+J+" with a score of 0.14. <eou>" and two complementary token sequences are generated by concatenating these with the recognized token sequence.

[0072] In addition, when the next multiple (two in this example) tokens with the highest probability (the tokens on the right side of the vertical dotted line in FIG. 6) are selected, the probabilities of the selected multiple (two) tokens may be corrected so that the sum of their probabilities becomes 1. After correction, the value becomes the value in parentheses. For example, the probability of the EoU token on the right side of the vertical dotted line on the most upstream side (left side in FIG. 6) is corrected to 0.63, and the probability of token F is corrected to 0.37. The same applies to the multiple (two) tokens selected further downstream. Therefore, since corrections (corrections that increase the value, i.e., each correction factor is 1 or more) are accumulated, when comparing the probabilities of the selected (estimated) multiple subsequent token strings, it is necessary to divide by the accumulated correction factor (the value obtained by multiplying each correction factor by the number of times corrections have been made) before comparing.

[0073] (Configuration of the language feature extraction means 32) The language feature extraction means 32 executes a process of extracting language features from the complementary token sequence generated by the sentence generator 31. In this embodiment, as an example, the language feature extraction means 32 uses BERT by Jacob Devlin et al., which is described in the above-mentioned non-patent document 7 and is pre-trained. That is, the complementary token sequence generated by the sentence generator 31 is input to BERT, and a sentence vector is generated by averaging the vectors for each token obtained as the output, and this sentence vector is used as the language feature. However, the language feature extraction means 32 is not limited to BERT, and may be any means capable of extracting language features from a complementary token sequence.

[0074] (Configuration of the Syntactic Completion Estimator 33: Figures 5, 6, and 7) The syntax completion estimator 33 executes a process of calculating a probability distribution of the number of utterance completion tokens or the utterance completion time length, which indicates how many tokens or how much time is required for the utterance to be completed, or a cumulative probability distribution that accumulates the probability of the number of utterance completion tokens or the utterance completion time length, using a series of multiple subsequent tokens estimated by the sentence generator 31. These probability distributions or cumulative probability distribution vectors are sent to the utterance timing estimator 50.

[0075] Here, the vector of the probability distribution of the number of utterance completed tokens is a 10-dimensional vector of (the probability that the 1st token of the subsequent token sequence will be the utterance end, the probability that the 2nd token will be the utterance end, ..., the probability that the 10th token will be the utterance end). It is called 10-dimensional because, in this embodiment, the upper limit of token generation by the sentence generator 31 is set to 10 tokens as an example. Also, the reason why the first dimensional element of the vector is the probability that the 1st token will be the utterance end, rather than the 0th token, is because the number of tokens is counted including the EoU token, and the probability that the nth token of the subsequent token sequence will be the utterance end means the probability that the nth token in the subsequent token sequence is an EoU token. In the example of FIG. 6, the first candidate subsequent token sequence "A+ <eou>” (the second token is at the end of the utterance) has a probability of 0.48, and the second candidate for the subsequent token sequence “B+F+J+ <eou>" (the fourth token is at the end of the utterance) has a probability of 0.14, so these numerical values ​​become the values ​​of the elements of the second and fourth dimensions of the vector, and the vector becomes (0, 0.48, 0, 0.14, 0, 0, 0, 0, 0, 0). Note that the number of tokens may not include EoU tokens.

[0076] In addition, the vector of the cumulative probability distribution of the number of completed utterance tokens is a 10-dimensional vector of (the probability that the subsequent token sequence will be the end of the utterance by the first token, the probability that it will be the end of the utterance by the second token, ..., the probability that it will be the end of the utterance by the tenth token). In the example of Figure 6, the first candidate subsequent token sequence "A+ <eou>” (the second token is at the end of the utterance) has a probability of 0.48, and the second candidate for the subsequent token sequence “B+F+J+ <eou>Since the probability of "(the fourth token is at the end of an utterance)" is 0.14, the probability that it will be at the end of an utterance by the first token is 0, the probability that it will be at the end of an utterance by the second token is 0.48, the probability that it will be at the end of an utterance by the third token is also 0.48, and the probability that it will be at the end of an utterance by the fourth token is 0.48+0.14=0.62, and the vector is (0,0.48,0.48,0.62,0.62,0.62,0.62,0.62,0.62,0.62,0.62). FIG. 7 shows an example of the output of the syntax completion estimator 33, where the horizontal axis is the number of tokens and the vertical axis is the average cumulative probability. Since it is cumulative, the closer to 10 tokens, the higher the probability. The bottom row of Figure 5 also shows the cumulative probability at the current time t for the recognized token sequence up to "How about XX?", but this number is just an image (a conceptual explanation that when an EoU token appears, the cumulative probability increases) and does not represent accurate data.

[0077] Furthermore, when the probability distribution or cumulative probability distribution of the utterance completion time length is to be a vector, for example, the relationship between each token and the average time (ms) required to pronounce each token can be stored in advance in a token-time correspondence table (not shown) made of a non-volatile memory, and the table can be used to convert each of a series of multiple subsequent tokens into the time required for utterance, and further, the required time can be assigned to one of a number of time segments (e.g., 10) that have been formed by dividing time in advance, thereby performing vectorization processing. In this case, the multiple time segments correspond to the elements of each dimension of the vector. Specifically, for example, in the case of a probability distribution, the first-dimension element of the vector is the probability that the utterance will be completed within a time length of 0 to less than T1, the second-dimension element is the probability that the utterance will be completed within a time length of T1 or more and less than T2, the third-dimension element is the probability that the utterance will be completed within a time length of T2 or more and less than T3, ..., the ninth-dimension element is the probability that the utterance will be completed within a time length of T8 or more and less than T9, the tenth-dimension element is the probability that the utterance will be completed within a time length of T9 or more and less than T10, etc. On the other hand, in the case of cumulative probability distribution, the first dimensional element of the vector is the probability that the utterance is completed in less than time length T1, the second dimensional element is the probability that the utterance is completed in less than time length T2, the third dimensional element is the probability that the utterance is completed in less than time length T3, ..., the ninth dimensional element is the probability that the utterance is completed in less than time length T9, the tenth dimensional element is the probability that the utterance is completed in less than time length T10, etc. Note that the boundaries of each time segment may be less than, less than, or greater than, or greater than. Then, as in the case of the number of completed utterance tokens, the probability of the subsequent token sequence is fitted to the time segment to which the required time for utterance obtained by converting the subsequent token sequence belongs.

[0078] In addition, the user's speaking speed (mora / second) may be measured in real time, or an average speaking speed calculated from the user's past speech data may be stored in a user information storage means (not shown) consisting of a non-volatile memory, and the time required for speaking obtained by converting the subsequent token sequence may be adjusted by the user's speaking speed (real time or past average speaking speed).

[0079] (Structure of the dialogue act estimator 40: Fig. 1, Fig. 3) 1 and 3, the dialog act estimator 40 receives as input data information including linguistic features extracted by the linguistic feature extraction means 32 using the complementary token sequence generated by the sentence generator 31 and acoustic features obtained from the audio signal of the user utterance by the processing by the dialog act estimation acoustic feature extraction means 24, and performs pattern recognition processing to estimate the dialog act (DA) of the user utterance, or to estimate the dialog act (DA) of the next system utterance in addition to the dialog act (DA) of the user utterance. Note that in this embodiment, the dialog act estimator 40 estimates the dialog acts (DA) of both the user utterance and the next system utterance.

[0080] The dialogue act estimator 40 is composed of a neural network, and first inputs the acoustic features from the dialogue act estimation acoustic feature extraction means 24 and the language features from the language feature extraction means 32 to a linear layer, then branches into two paths to estimate the dialogue act of the user utterance and the dialogue act of the next system utterance using separate transformer encoders and linear layers. Then, intermediate states between the front part 41 and the rear part 42 in the two paths for estimating the dialogue act of the user utterance and for estimating the dialogue act of the next system utterance are extracted, and these intermediate features are sent to the speech timing estimator 50. Therefore, the shared linear layer, the branched transformer encoder for user speech, and the linear layer constitute the front-stage unit for user speech 41A, and the shared linear layer, the branched transformer encoder for system speech, and the linear layer constitute the front-stage unit for system speech 41B. Also, the back-stage unit for user speech 42A and the back-stage unit for system speech 42B are constituted by separate linear layers.

[0081] During training, we optimize the cross entropy loss for each pass for estimating dialogue acts of a user utterance and the next pass for estimating dialogue acts of a system utterance.

[0082] The labels of the dialogue acts may be arbitrarily defined, and may be, for example, "question, response, interjection, request for clarification, request for repetition, understanding, non-understanding, indifference, or other," which are listed as examples of user speech intentions in the aforementioned Patent Document 1 by the applicant of the present application.

[0083] In the effectiveness confirmation experiment described later, the HARPER VALLEY BANK corpus by M. Wu et al. described in the above-mentioned non-patent document 3 is used, and the labels of the dialogue acts of the user utterances there are 16 types, for example, "yes" response, greeting, response, data confirmation, procedure explanation, data question, closing, data communication, "bear with me" response, acknowledgement, data response, filler disfluency, thanks, open question, problem description, and other. In addition, the labels of the dialogue acts of the system utterances are the same 16 types as the user utterances, but in the effectiveness confirmation experiment described later, the case where the system does not utter on its own is defined as "waiting" and added to the labels.

[0084] (Configuration of speech timing estimator 50: Fig. 1, Fig. 2) In FIG. 1 and FIG. 2, the speech timing estimator 50 estimates the elapsed time (ETAU)h(E) from the end of the user's speech measured by the elapsed time measurement means 22. t and the acoustic feature quantity h(A) for utterance timing estimation extracted by the acoustic feature quantity extraction means 23 for utterance timing estimation. t and the linguistic feature h(L) extracted by the linguistic feature extraction means 32. t and a vector h(ESC) of the probability distribution or cumulative probability distribution of the number of utterance completion tokens (the number of tokens until utterance completion) or the utterance completion time length (the time length until utterance completion) output from the syntax completion estimator 33. t and the dialogue act estimation result output from the dialogue act estimator 40 or the intermediate feature value h(DA) leading to the estimation result. t The system uses information including the above as input data, and repeatedly executes pattern recognition processing to identify whether the user's right to speak, which indicates that the user has the status or position to speak, at a period that is not dependent on the timing of execution of the voice recognition processing by the voice recognizer 30, and estimates the timing to start the next system utterance.

[0085] Among these, the output vector h(ESC) from the syntactic completion estimator 33 t is regarded as the information (a) as the prediction information in the present invention, and is the output from the dialogue act estimator 40 or its intermediate feature value h(DA) t is the information (b) as prediction information in the present invention, and the linguistic feature quantity h(L) t is regarded as information (c) as predictive information in the present invention.

[0086] Note that "termination" of a user's speech right includes both abandonment and transfer. Abandonment is when one simply ends one's speech, while transfer is when one ends one's speech and encourages the other party to start speaking, such as when asking a question to the other party.

[0087] As a result of this pattern recognition process, as shown in Fig. 2, a likelihood indicating the certainty of the user's right to speak being maintained or terminated is obtained, so a threshold is determined in advance, and when the likelihood is below the threshold or equal to or less than the threshold, it is determined that the user's right to speak is being maintained, and the speech timing estimator 50 outputs the estimation result that it is not the timing to start a system utterance (0). On the other hand, when the likelihood is above or exceeds the threshold, it is determined that the user's right to speak has terminated, and the speech timing estimator 50 outputs the estimation result that it is the timing to start a system utterance (1). This is the same as in the dialogue system by the applicant described in the above-mentioned Patent Document 1. Note that 1 and 0, and "above" and "below", and "below" and "exceed" in threshold determination may be set inversely.

[0088] Here, in this embodiment, elements that are expected to improve the estimation accuracy of the speech timing are listed as input data to the speech timing estimator 50, but the estimation accuracy can be improved even if not all of these elements are used as input data.

[0089] That is, first, the elapsed time from the end of the utterance (ETAU)h(E) t As already described in detail in the description of the configuration of the means 22 for measuring the elapsed time since the end of the user's utterance, the elapsed time here includes time features, and may be a substitute value indicating the length of the elapsed time (for example, a count number corresponding to the period of the pattern recognition process by the utterance timing estimator 50). t However, in order to improve the accuracy of estimating the utterance timing, h(E) may be omitted. t It is preferable to input

[0090] Next, the linguistic feature value h(L) extracted by the linguistic feature value extraction means 32 t In this embodiment, is a linguistic feature extracted from the complementary token sequence generated by the sentence generator 31, but may be a linguistic feature extracted from the recognized token sequence output by the speech recognizer 30. However, from the viewpoint of improving the accuracy of estimating the speech timing, it is preferable to use a linguistic feature extracted from the complementary token sequence. Furthermore, when information on the dialogue act (intermediate feature or estimation result) is added to the input data to the speech timing estimator 50, speech timing estimation is possible without using the linguistic feature, so in that case the input of the linguistic feature may be omitted. However, from the viewpoint of improving the accuracy of estimating the speech timing, it is preferable to also input the linguistic feature.

[0091] Also, the vector h(ESC) of the probability distribution or cumulative probability distribution of the number of utterance completion tokens or the utterance completion time length output from the syntactic completion estimator 33 t However, in order to improve the accuracy of estimating the utterance timing, h(ESC) t It is preferable to input

[0092] 1 and 3, the intermediate feature value h(DA) extracted from the dialogue act estimator 40 t However, it is also possible to input the estimation result output from the dialogue act estimator 40. However, from the viewpoint of improving the estimation accuracy of the speech timing, it is preferable to input the intermediate feature.

[0093] 1, the dialogue act estimator 40 is supplied with linguistic features extracted by the linguistic feature extraction means 32 using the complemented token sequence generated by the sentence generator 31. Alternatively, instead of the complemented token sequence, linguistic features extracted by the linguistic feature extraction means 32 using the recognized token sequence output by the speech recognizer 30 may be input to the dialogue act estimator 40, and the output or intermediate features of the dialogue act estimator 40 may be input to the speech timing estimator 50. However, from the viewpoint of improving the accuracy of estimating speech timing, it is preferable to input the linguistic features extracted from the complemented token sequence to the dialogue act estimator 40, and the output or intermediate features of the dialogue act estimator 40 to the speech timing estimator 50. Note that the input of dialogue act information (intermediate features or estimation results) to the speech timing estimator 50 may be omitted.

[0094] This speech timing estimator 50 is composed of a neural network, and as shown in Fig. 2, takes into account the temporal characteristics of the input sequence using one layer of LSTM, and outputs a likelihood (probability) indicating the likelihood of the user's right to speak being maintained or terminated for each frame using a linear layer and a sigmoid function. As described above, the pattern recognition process of the speech timing estimator 50 is executed at a period that is not dependent on the execution timing of the speech recognition process by the speech recognizer 30. Then, the language feature h(L) by the language feature extraction means 32 is calculated. t As shown in FIG. 1, the extraction of the linguistic feature quantity h(L) input to the speech timing estimator 50 is performed in synchronization with the speech recognition output downstream of the speech recognizer 30. t is not necessarily updated for each analysis frame of the utterance timing estimator 50, and if it is not updated, the same as that of the previous frame is used, and this state is shown in FIG. 2. In addition, since the processing of the syntactic completeness estimator 33 is also performed downstream of the speech recognizer 30 as shown in FIG. 1, the output vector h(ESC) of the syntactic completeness estimator 33 is similarly t The linguistic feature value h(L) calculated by the linguistic feature extraction unit 32 is not necessarily updated for each analysis frame of the speech timing estimator 50. t is also input to the dialogue act estimator 40. In this sense, as shown in FIG. 1, the dialogue act estimator 40 is disposed downstream of the speech recognizer 30. The output of the dialogue act estimator 40 (the period of the pattern recognition process) is the linguistic feature h(L) t 2, the intermediate feature value h(DA) extracted from the dialogue act estimator 40 is not necessarily updated at the same timing. t is the linguistic feature h(L) t It does not describe a similar situation.

[0095] (Configuration of next utterance preparation means 60) The next utterance preparation means 60 executes a preparation process to acquire or generate content data of the system's next utterance (in this embodiment, content data of multiple next utterance candidates that are candidates for the next utterance) using subject data stored in the subject data storage means 70 or subject data stored in an external system (not shown) connected via a network, at a timing that is not dependent on the period of the pattern recognition processing by the utterance timing estimator 50 and before the start timing of the system utterance is detected by the utterance timing estimator 50, and using the complementary token sequence generated by the sentence generator 31, or using the complementary token sequence as well as at least a portion of the dialogue history information between the user and the system stored in the dialogue history storage means 72, and stores the prepared content data of the next utterance (in this embodiment, content data of multiple next utterance candidates) in the next utterance candidate storage means 71.

[0096] At this time, since a plurality of complementary token strings are generated by the sentence generator 31, a plurality of next utterance candidates are prepared and stored in the next utterance candidate storage means 71. Even if there is only one complementary token string, since the next utterance selection means 61 executes a selection process using prosodic information and the like, a plurality of next utterance candidates may be prepared and stored in the next utterance candidate storage means 71. Furthermore, the next utterance preparation means 60, if necessary, uses a dictionary means for kanji conversion (not shown) to convert the complementary token string that has been generated without including kanji into a character string including kanji.

[0097] Such preparation processing by the next utterance preparation means 60 is also performed in the dialogue system by the applicant of the present application described in the above-mentioned Patent Document 1, but the difference is that in the above-mentioned Patent Document 1, the next utterance preparation processing is performed using a recognized token sequence up to the middle of the user utterance, whereas in the present application, the next utterance preparation processing is performed using a complementary token sequence. This makes it possible to prepare the next utterance more quickly and with more appropriate content than in the case of the above-mentioned Patent Document 1.

[0098] More specifically, the next utterance preparation means 60 first executes an initial preparation process for acquiring or generating content data of a plurality of next utterance candidates of the system, and an exchange process for exchanging the content data of the initially prepared plurality of next utterance candidates or the content data of the plurality of next utterance candidates prepared in the previous exchange with content data of another plurality of next utterance candidates. These processes are also performed in the dialogue system by the applicant described in the above-mentioned Patent Document 1. If there is a dialogue history of system utterance S(1) and user utterances U(1), S(2), U(2), ..., S(N), and the user utterance U(N) is about to start, the specific process contents are as follows. Note that a specific example of the content data of the system utterance (for example, the specific contents of the scenario data, etc.) may be the same as that in the above-mentioned Patent Document 1, and therefore a description thereof will be omitted in this application.

[0099] That is, in the initial preparation process by the next utterance preparation means 60, when preparation for the next system utterance S(N+1) is started, at least a part of the dialogue history information up to that point (dialogue history information up to the system utterance S(N)) stored in the dialogue history storage means 72, i.e., at least a part of S(1), U(1), S(2), U(2), ..., S(N), is used to select and acquire content data of multiple system next utterance candidates S(N+1) from among the subject data and its components stored in the subject data storage means 70 or an external system (not shown), and store it in the next utterance candidate storage means 71. However, when preparation for the first system utterance S(1) is started, only one S(1) needs to be selected and acquired. Also, it is not always necessary to select and acquire multiple next utterance candidates, and there may be a case where only one next utterance candidate is selected and acquired as a result.

[0100] Here, using at least a part of the dialogue history information (S(1), U(1), S(2), U(2), ..., S(N)) includes using the entire dialogue history information (S(1), U(1), S(2), U(2), ..., S(N)), using only the immediately preceding system utterance S(N), using earlier system utterances S(1) to S(N-1) or user utterances U(1) to U(N-1) without using the immediately preceding system utterance S(N) (for example, in response to a user request such as "I'd like to hear more about that XXX you just said..."), using only the immediately preceding user utterance U(N-1), etc.

[0101] On the other hand, in the replacement process by the next utterance preparation means 60, when it is determined that replacement is necessary in a state in which content data of multiple next utterance candidates S(N+1) are already stored in the next utterance candidate storage means 71, the content data of another multiple next utterance candidates S(N+1) of the system is selected and acquired from the subject data and its components stored in the subject data storage means 70 or an external system (not shown) using the language information of the currently ongoing user utterance U(N), i.e., the recognized token sequence obtained as a result of the speech recognition process by the speech recognizer 30, and the content data of the multiple next utterance candidates S(N+1) stored in the next utterance candidate storage means 71 is replaced by overwriting the content data of the multiple next utterance candidates S(N+1) stored in the next utterance candidate storage means 71. However, it is not always necessary to select and acquire multiple next utterance candidates, and there may be a case in which the selected and acquired next utterance candidates end up being one. The replacement process by the next utterance preparation means 60 may use not only the user utterance U(N) but also the dialogue history information S(1), U(1), . . . , S(N) as in the initial preparation process.

[0102] The "acquisition" in "acquiring or generating content data for the system's next utterance" by the next utterance preparation means 60 includes selective acquisition of necessary subject data (subject data that is to be used or has the potential to be used) from among multiple subject data stored in the subject data storage means 70 or an external system (not shown) connected via a network, and selective acquisition of necessary components (components that are to be used or have the potential to be used) from among the components of any one subject data stored in the subject data storage means 70 or the external system (not shown).

[0103] Furthermore, the above-mentioned "generation" in the next utterance preparation means 60 includes processing (partial conversion and adjustment of endings, combination, etc.) of acquired language information (subject data or text data that is a component thereof). However, it is preferable that the subject data is processed in advance at the stage where it is stored in the subject data storage means 70 or an external system (not shown). Furthermore, "generation" also includes conversion from text data to voice data (voice synthesis). Note that if voice data (e.g., a wav file, etc.) is already prepared as the subject data or a component thereof in the subject data storage means 70 or an external system (not shown), voice synthesis processing by the next utterance preparation means 60 does not have to be performed.

[0104] Furthermore, the "content data of the system's next utterance" prepared by the next utterance preparation means 60 may be text data and corresponding voice data, or text data only. However, from the viewpoint of reducing the processing load of the utterance generation means 62 and preventing delays, it is preferable for the next utterance preparation means 60 to also prepare voice data. In addition, when playing back video (video) or still images as incidental information during a dialogue, the "content data of the system's next utterance" may be accompanied by video data or image data, and when playing back music during a dialogue, the "content data of the system's next utterance" may include music data.

[0105] (Configuration of next utterance selection means 61) After the start timing of a system utterance is detected by the utterance timing estimator 50 (when the next utterance selection means 61 receives a judgment result that it is the start timing of a system utterance from the utterance timing estimator 50), the next utterance selection means 61 executes a process of selecting content data of the next utterance to be used in the utterance generation means 62 from among the content data of multiple (however, there may be only one) next utterance candidates obtained by the preparation process by the next utterance preparation means 60 and stored in the next utterance candidate storage means 71, using a combination of the estimated result of the dialogue act (DA) of the user utterance obtained by the processing by the dialogue act estimator 40 or additionally the estimated result of the dialogue act (DA) of the system utterance (here, the estimated result is used instead of the intermediate feature value) and the complementary token sequence generated by the sentence generator 31 (since the user utterance has ended, it may ultimately become a recognized token sequence for the entire user's utterance section. In other words, it may be in a state where there is no subsequent token sequence), and sending the selected content data of the next utterance to the utterance generation means 62.

[0106] Note that if the content data of the next utterance can be selected using only either the estimated result of the dialogue act (DA) of the user utterance or the system utterance obtained by the processing by the dialogue act estimator 40 or the complementary token string generated by the sentence generator 31, it is not necessary to use them in combination. Furthermore, the next utterance selection means 61 converts the complementary token string generated without containing kanji into a character string including kanji, if necessary, using a dictionary means for kanji conversion (not shown).

[0107] The process by the next utterance selection means 61 is also performed in the dialogue system by the applicant described in the above-mentioned Patent Document 1. However, the above-mentioned Patent Document 1 uses the result of the speech recognition process (the recognized token sequence) by the speech recognizer 30 after the end of the user's speech, whereas the present application uses the complementary token sequence generated by the sentence generator 31. However, since the user's right to speak has ended, the complementary token sequence used in this next utterance selection process may be only the recognized token sequence for the entire speech section of the user (i.e., there is no subsequent token sequence), and this may be the case if the speech recognizer 30 has low latency. In addition, the above-mentioned Patent Document 1 calls the means for obtaining the identification result of the user's speech intention by performing prosodic analysis, etc., a "next utterance selection information generation means," but the present application provides a dialogue act estimator 40 as a corresponding means.

[0108] Specifically, when the next utterance selection means 61 selects the next utterance using the complemented token sequence generated by the sentence generator 31, it can perform keyword matching using each word included in the complemented token sequence and each word included in each of the content data of the multiple next utterance candidates stored in the next utterance candidate storage means 71, and select the content data of the matching next utterance candidate as the content data of the next utterance. In addition, complex matching combining language processing and machine learning may be performed. Furthermore, the similarity between the complemented token sequence and each of the content data (character strings) of the multiple next utterance candidates may be found by doc2vec or the like, and the content data of the next utterance candidate with the highest similarity may be selected as the content data of the next utterance.

[0109] In addition, in cases where the system asks a user questions and the user answers them, such as in a voice dialogue in an automatic telephone response, the contents of the user's utterance are limited to options given by the system, and the contents of the user's utterance can be predicted. In this case, if each of the content data of the multiple next utterance candidates stored in the next utterance candidate storage means 71 is accompanied by corresponding predicted data of the user's utterance, it is possible to determine which of the accompanying predicted data of the user's utterance matches the complemented token string, and select the content data of the next utterance candidate corresponding to the matching predicted data of the user's utterance as the content data of the next utterance.

[0110] For example, suppose that the system utterance S(N) is "Which do you support, the XXX party or the YYY party?", and that the content data "Which politician from the XXX party do you think will become the Prime Minister?" and the associated predicted data of the user utterance U(N) "XXX party" and the content data "Which politician from the YYY party is suitable to be the party leader?" and the associated predicted data of the user utterance U(N) "YYY party" are prepared by the next utterance preparation means 60 and stored in the next utterance candidate storage means 71. At this time, if the result of the speech recognition process of the user utterance U(N) is "XXX party", it matches the predicted data of the user utterance U(N) "XXX party", so the corresponding "Which politician from the XXX party do you think will become the Prime Minister?" is selected as the system utterance S(N+1) and is reproduced by the utterance generation means 62.

[0111] Furthermore, when the next utterance selection means 61 selects the next utterance using the estimated result of the dialogue act (DA) of the user utterance or the next system utterance obtained by the processing by the dialogue act estimator 40, the system utterance type corresponding to the dialogue act (DA) is determined for each of the content data of the multiple next utterance candidates stored in the next utterance candidate storage means 71, so that the content data of the next utterance candidate, which is the system utterance type corresponding to the estimated result of the obtained dialogue act (DA), can be selected as the content data of the next utterance. Note that, in the dialogue system by the applicant of the present application described in the above-mentioned Patent Document 1, the next utterance is selected using the identification result of the user's utterance intention (question, backchannel, etc.), but this is the same as the case of selecting the next utterance using the estimated result of the dialogue act (DA) of the present application.

[0112] (Configuration of the speech generation means 62) The utterance generation means 62 executes a system utterance generation process including reproducing the audio signal of the system utterance, using the content data of the next utterance selected by the next utterance selection means 61. This utterance generation means 62 includes a speaker as hardware, and may also include a display for reproducing video and still images as an additional function.

[0113] Here, the "system utterance generation process including reproduction of the audio signal of the system utterance" in the utterance generation means 62 includes a voice synthesis process when the voice synthesis of the text data of the next utterance has not been completed yet. As described above, from the viewpoint of reducing the processing load of the utterance generation means 62 and preventing delays, it is preferable that the audio data (e.g., a wav file, etc.) is prepared in the preparation process by the next utterance preparation means 60. Furthermore, when the "content data of the next utterance of the system" prepared by the next utterance preparation means 60 is accompanied by video data or image data, the "system utterance generation process" in the utterance generation means 62 also includes a reproduction process of the video or image, and when the "content data of the next utterance of the system" includes music data, the "system utterance generation process" also includes a reproduction process of the music.

[0114] (Configuration of the subject data storage means 70) The subject data storage means 70 stores subject data such as, for example, scenario data in which various topics such as news are made into a scenario, various topic data that are not made into a scenario, dictionary data, encyclopedia data, data for guidance on how to use devices and facilities, data for questionnaire surveys, data for assisting in the operation of devices and equipment, educational data, etc. This is similar to the case of the dialogue system by the applicant of the present application described in the above-mentioned Patent Document 1. The same is true for subject data stored in an external system (not shown) connected via a network.

[0115] (Configuration of next utterance candidate storage means 71) The next utterance candidate storage means 71 stores content data of a plurality of next utterance candidates prepared by the next utterance preparation means 60 in association with identification information of the data (e.g., scenario ID, speech clause ID, etc.). The stored content data of the next utterance candidates includes, in addition to text data, audio data (e.g., wav files, etc.), and may further include video data, still image data, or music data.

[0116] Further, each of the content data of the multiple next utterance candidates is stored in association with a system utterance type indicating the attribute of the data (for example, whether the scenario data is a main plan or a sub-plan). The system utterance type is used in the processing of the next utterance selection means 61.

[0117] (Configuration of dialogue history storage means 72) The dialogue history storage means 72 stores dialogue history information between the system and the user. Specifically, it stores the content data (text data) of the system utterance S(1), the content data (text data) of the user utterance U(1), and similarly the content data (text data) of S(2), U(2), S(3), U(3), ... in the order of the dialogue. It may start with the user utterance. Note that the ongoing user utterance U(N), i.e., multiple complementary token strings (the recognized token string and multiple estimated subsequent token strings connected thereto), are stored in the main memory, and after the utterance is ended, the recognized token strings of the entire utterance section are stored in the dialogue history storage means 72.

[0118] <Processing flow when taking turns from user to system: Figure 8>

[0119] In this embodiment, the dialogue system 10 using the speech timing estimation system 20 processes speaker turn-taking from the user to the system as follows. However, since the processing cycle of each element varies in length and some processing is performed asynchronously, the flow below shows an outline of the flow without considering the length of those cycles or synchronicity.

[0120] In FIG. 8, first, the speech signal acquisition means 21 acquires the user's speech (step S1). Next, the time elapsed since the end of the user's speech measurement means 22 measures the time elapsed since the end of the user's speech (step S2). The speech timing estimation acoustic feature extraction means 23 extracts acoustic features for estimating speech timing from the user's speech signal (step S3). The dialogue act estimation acoustic feature extraction means 24 extracts acoustic features for estimating dialogue acts from the user's speech signal (step S4). Then, the speech recognizer 30 performs speech recognition processing using the user's speech signal to create a recognized token sequence (step S5). In this case, in the speech recognition processing, acoustic features are extracted from the user's speech signal and the acoustic features are input to the encoder. Alternatively, the dialogue act estimation acoustic features obtained in the above step S4 may be input to the encoder.

[0121] Then, a sentence generator 31 configured with a language model such as GPT-2, 3 or 4 estimates multiple subsequent token sequences using the recognized token sequence obtained by the speech recognizer 30, and concatenates each of these subsequent token sequences to the recognized token sequence to generate multiple complementary token sequences (step S6).

[0122] Next, the syntax completion estimator 33 creates a vector of a probability distribution or cumulative probability distribution of the number of tokens or the time length until the completion of the utterance, using the multiple subsequent token strings estimated by the sentence generator 31 (step S7). In addition, the language feature extraction means 32 extracts language features using the complementary token string generated by the sentence generator 31 (step S8). Furthermore, the next utterance preparation means 60 prepares content data of multiple next utterance candidates, using the complementary token string generated by the sentence generator 31, the subject data stored in the subject data storage means 70 or an external system (not shown) connected via the Internet, and the dialogue history information stored in the dialogue history storage means 72, and stores the content data in the next utterance candidate storage means 71 (step S9).

[0123] Next, the dialogue act estimator 40 (see Figures 1 and 3) estimates dialogue acts (DA) of the user utterances and the system utterances using the linguistic features extracted by the linguistic feature extraction means 32 and the acoustic features for dialogue act estimation extracted by the dialogue act estimation acoustic feature extraction means 24, and extracts intermediate features on the way to the estimation result (step S10).

[0124] Then, the speech timing estimator 50 (see Figures 1 and 2) estimates whether the user's right to speak will be maintained or ended, using the elapsed time from the end of the user utterance measured by the elapsed time measurement means 22 after the end of the user utterance, the acoustic features for estimating speech timing extracted by the acoustic feature extraction means for estimating speech timing 23, the vector of the probability distribution or cumulative probability distribution of the number of utterance completion tokens or the speech completion time length created by the syntax completion estimator 33, the linguistic features extracted by the linguistic feature extraction means 32, and the intermediate features of the dialogue acts of the user utterance and the system utterance extracted by the dialogue act estimator 40 (step S11).

[0125] Then, based on the result of the speech timing estimator 50's determination of whether the user's right to speak is maintained or terminated, if the user's right to speak is maintained, the process returns to step S1, and if the user's right to speak has terminated, it is determined that it is time to start a system speech (step S12).

[0126] If it is determined that it is time to start a system utterance, the next utterance selection means 61 uses a combination of the dialogue act (DA) estimation results of the user utterance and system utterance by the dialogue act estimator 40 and the complementary token sequence generated by the sentence generator 31 to select content data of the next utterance to be used by the utterance generation means 62 from the content data of multiple (although there may be only one) next utterance candidates obtained by the preparation process by the next utterance preparation means 60 and stored in the next utterance candidate storage means 71 (step S13), and the utterance generation means 62 uses the content data of the next utterance to perform processing such as playing the system utterance (step S14).

[0127] <Effects of this embodiment>

[0128] The dialogue system 10 using the speech timing estimation system 20 of this embodiment has the following effects. That is, since the speech timing estimation system 20 includes the sentence generator 31, it can estimate a subsequent token string from the recognized token string (up to the middle of the sentence) obtained by the speech recognizer 30, and generate a complementary token string equivalent to the entire sentence. Therefore, by using the subsequent token string and the complementary token string, the estimation accuracy of the speech timing estimation system 20 can be improved, and speaker changes can be made at appropriate timing, thereby realizing a smooth dialogue by the dialogue system 10. A person in a dialogue also measures the timing of his / her own speech while predicting, in the middle of the other person's speech, what the other person is trying to convey and how much time it will take to convey that. Therefore, the present invention can be said to incorporate this prediction function into the system.

[0129] Specifically, the language feature extraction means 32 extracts language features using the complementary token sequence generated by the sentence generator 31, and the language features are input to the dialogue act estimator 40 to estimate the dialogue acts of the user utterances and the system utterances, thereby improving the accuracy of estimating the dialogue acts. Then, intermediate features are extracted from the dialogue act estimator 40 and input to the speech timing estimator 50 to estimate the maintenance or end of the user's right to speak, thereby improving the accuracy of estimating the start timing of the system utterance. Therefore, the estimation accuracy of the speech timing estimator 50 can be improved by inputting, to the utterance timing estimator 50, not the estimation result by the dialogue act estimator 40, but intermediate features along the way to the estimation result. In addition, by inputting to the dialogue act estimator 40 language features extracted from the complemented token sequence, rather than from the recognized token sequence up to the middle of the user's utterance, the estimation accuracy of the dialogue act estimator 40 is improved by inputting to the dialogue act estimator 40 intermediate features extracted from the dialogue act estimator 40 with improved estimation accuracy. This also improves the estimation accuracy of the speech timing estimator 50.

[0130] Furthermore, the language feature extraction means 32 extracts language features not from the recognized token sequence but from the complemented token sequence, and inputs these language features to the speech timing estimator 50. This also improves the estimation accuracy of the speech timing estimator 50.

[0131] Furthermore, the speech timing estimation system 20 is equipped with a syntax completion estimator 33, which uses a plurality of subsequent token strings estimated by the sentence generator 31 to create a vector of a probability distribution or cumulative probability distribution of the number of tokens or time length until speech completion, and inputs this vector to the speech timing estimator 50. Therefore, in this respect as well, the estimation accuracy of the speech timing estimator 50 can be improved.

[0132] Furthermore, the speech timing estimation system 20 includes an elapsed time since end of user utterance measurement means 22, and the time elapsed since the end of user utterance measured by this elapsed time since end of user utterance measurement means 22 is input to the speech timing estimator 50. Therefore, in this respect as well, the estimation accuracy of the speech timing estimator 50 can be improved.

[0133] Furthermore, the dialogue system 10 includes a next utterance preparation means 60, which executes the next utterance preparation process using the complementary token sequence generated by the sentence generator 31, rather than the recognized token sequence. This allows the dialogue system 10 to prepare a next utterance candidate with more appropriate content earlier and store it in the next utterance candidate storage means 71, as compared to the dialogue system of the present applicant described in the above-mentioned Patent Document 1. This reduces the probability of an inconvenience such as the next system utterance not being able to be generated because the next utterance candidate has not yet been prepared even though the user's right to speak has ended. This allows the dialogue system 10 to take turns speaking at an appropriate time, thereby realizing a smooth dialogue.

[0134] The dialogue system 10 also includes a next utterance selection means 61, which executes a next utterance selection process using the complementary token sequence generated by the sentence generator 31. This allows for more appropriate next utterance selection compared to the dialogue system of the present applicant described in the above-mentioned Patent Document 1. In this case, the next utterance selection process by the next utterance selection means 61 is performed after the speech timing estimator 50 has determined that the user's right to speak has ended. Therefore, if the speech recognizer 30 has a low delay, the complementary token sequence may end up being only the recognized token sequence for the entire speech section of the user, that is, there may be no subsequent token sequence. However, since it is not possible to completely eliminate the delay, if there is a speech recognition delay portion in the portion where the user has finished speaking (see FIG. 4), the complementary token sequence provides linguistic information for that speech recognition delay portion, so the next utterance selection means 61 can obtain more linguistic information and realize more appropriate next utterance selection.

[0135] The next utterance selection means 61 can also select the next utterance using the estimation result of the dialogue act estimator 40. In this case, since the estimation accuracy of the estimation result of the dialogue act estimator 40 is improved by using the complementary token sequence, more appropriate next utterance selection can be realized.

[0136] Furthermore, if a low-delay speech recognizer, for example, the multi-look-ahead speech recognizer (MLA-ASR) by the present inventors described in the above-mentioned Non-Patent Document 5, is used as the speech recognizer 30, the estimation accuracy of the speech timing estimator 50 can be further improved. As shown in Fig. 4, the complementary token sequence generated by the sentence generator 31 complements the linguistic information between the speech recognition delay portion and the planned speech portion, but if a low-delay speech recognizer 30 is used, the speech recognition delay portion becomes smaller (however, the delay is not completely zero), and therefore the effect of using the complementary token sequence is reduced accordingly, and the effect of reducing the delay of the speech recognizer 30 and the effect of using the complementary token sequence coexist. The complementary token sequence is linguistically supplemented by the language model, whereas the low latency of the speech recognizer 30 is achieved by processing techniques based on acoustic features obtained from the user's speech signal. Therefore, the estimation accuracy of the speech timing estimator 50 improves when the recognized token sequence is longer due to the low latency of the speech recognizer 30, that is, when the subsequent token sequence estimated by the language model is shorter.

[0137] <Effect verification experiment: Figures 9 to 16>

[0138] The experimental data used was the HARPER VALLEY BANK corpus described in the aforementioned Non-Patent Document 3. This corpus is a simulated conversation that mimics a simple exchange with a consumer finance company, with speakers split into the roles of customers and agents. It is a data set of approximately 23 hours of speech that includes 1,446 conversations between people by 59 speakers.

[0139] As shown in Fig. 9, the precision, which is an evaluation index, is defined from the viewpoint of whether the start time T2 of the estimated system utterance is taken as the base point and whether the start time T3 of the utterance (correct answer) of the speaker regarded as the system is within a range of ΔT2 around T2. In other words, the precision is the value obtained by dividing the "number of times that the speaker regarded as the system started speaking within the allowable error ΔT2" by the "number of times that the system utterance started in the experiment." The precision Px.xx in Fig. 11 indicates the precision when the allowable error ΔT2 = X.XX seconds.

[0140] As shown in Fig. 10, recall, an evaluation index, is defined from the viewpoint of whether the start time T2 of the estimated system utterance falls within a range ΔT3 around T3, based on the start time T3 of the utterance (correct answer) of the speaker regarded as the system. In other words, recall is the value obtained by dividing "the number of times that the system utterance started within the allowable error ΔT3 in the experiment" by "the number of times that the speaker regarded as the system started speaking." The recall Rx.xx in Fig. 11 indicates the recall when the allowable error ΔT3 = X.XX seconds.

[0141] Moreover, F1x.xx in FIG. 11 indicates the F1 value when the allowable error ΔT2=ΔT3=X.XX seconds.

[0142] Fig. 11 shows the evaluation results of speech timing estimation when the dialogue act estimator 40 is used. The baseline in Fig. 11 is the case where the intermediate features extracted from the dialogue act estimator 40 are not input to the speech timing estimator 50, and below it are the cases where the intermediate features extracted from the dialogue act estimator 40 are input and where the intermediate features when the dialogue act estimation result becomes a true value are input. In this experiment, the acoustic features and linguistic features are input to the speech timing estimator 50, but the sentence generator 31 and the syntax completion estimator 33 are not used. As already mentioned in the explanation of the configuration of the dialogue act estimator 40, there are 16 types of labels for the dialogue acts of the user utterances, and the labels for the dialogue acts of the system utterances are the same 16 types plus the label "waiting."

[0143] According to FIG. 11, the evaluation result with a tolerance of 1.00 seconds is better, but since it is desirable to achieve high performance with a small tolerance, 1.00 seconds is too long to be worthy of evaluation. Looking at the evaluation result with a tolerance of 0.25 seconds, it can be seen that the performance is improved by 5 to 6% when the intermediate feature extracted from the dialogue act estimator 40 is input to the speech timing estimator 50 compared to the baseline. In addition, when the intermediate feature when the dialogue act estimation result becomes the true value is input to the speech timing estimator 50, the performance is improved. This shows that if the dialogue act can be estimated more accurately, the performance when the intermediate feature extracted from the dialogue act estimator 40 is input to the speech timing estimator 50 may be further improved.

[0144] The evaluation results of speech timing estimation (1) are shown in Fig. 12, the evaluation results of dialogue act estimation (1) in Fig. 13, the evaluation results of speech timing estimation (2) in Fig. 14, and the evaluation results of dialogue act estimation (2) in Fig. 15, each shown in a line graph. In each line graph, "〇〇 / △△" indicates which token sequence was used for "learning / inference" and which language model's complementary token sequence was used.

[0145] W: The entire correct sentence P: Recognized token sequence (the recognition result itself) C-GPT2: Completion token sequence by GPT-2 language model C-LSTM: Completion of tokens by LSTM language model Therefore, for example, “training / inference” = “W / C-GPT2” is the case where learning is done on the entire correct sentence and inference (operation) is done on the complementary token sequence by the GPT-2 language model.

[0146] According to Figures 12 and 13, the F1 values ​​of "W / C-GPT2" and "W / C-LSTM" are good for "W / P", so it can be seen that by using the complementary token sequence generated by the sentence generator 31 composed of a language model during inference (operation), the estimation accuracy of the speech timing (see Figure 12) and the estimation accuracy of the dialogue act (see Figure 13) are improved.

[0147] According to Figure 14, the F1 values ​​of "C-GPT2 / C-LSTM" and "W / C-LSTM" are equivalent, so when training with a complement token sequence from the GPT-2 language model, the same performance can be obtained in terms of the accuracy of estimating speech timing as when training with the entire correct sentence. However, since the F1 value of "C-LSTM / C-LSTM" is lower than that of "W / C-LSTM", it can be seen that the performance deteriorates when training with a complement token sequence from the LSTM language model. Note that, as can be seen from Figure 15, there is not much difference in the accuracy of estimating dialogue acts.

[0148] In FIG. 16, GMF (Gated Multimodal Fusion) is a model by Jiudong Yang et al. described in the above-mentioned Non-Patent Document 2, and as described in [Background Art], it is a timing estimation method (ES-TD) synchronized with the detection of the end of utterance (EoU). The baseline is a case where neither the syntactic completeness estimator 33 nor the low-delay speech recognizer 30 is used. Below that, the evaluation results are shown for the case where the low-delay speech recognizer 30 (MLA-ASR) by multi-look-ahead is used, the case where the output vector from the syntactic completeness estimator 33 is used, and the case where both the low-delay speech recognizer 30 and the syntactic completeness estimator 33 are used. According to FIG. 16, it can be seen that the estimation accuracy of the utterance timing is improved when the low-delay speech recognizer 30 or the syntactic completeness estimator 33 is used, and the estimation accuracy of the utterance timing is further improved by combining both of these.

[0149] <Transformation Form>

[0150] The present invention is not limited to the above-described embodiment, and modifications within the scope of the present invention are included in the present invention.

[0151] For example, the dialogue system 10 of the embodiment includes the speech timing estimation system 20 of the present invention, but the dialogue system of the present invention does not necessarily need to include the speech timing estimation system of the present invention, and may include, for example, a speech timing estimator (called a system speech timing detection means in Patent Document 1) constituting the dialogue system by the applicant of the present application described in the above-mentioned Patent Document 1. In short, it is sufficient that the next utterance preparation means 60 prepares the next utterance (multiple next utterance candidates) using the complementary token sequence generated by the sentence generator 31. Also, it is preferable that the next utterance selection means 61 can select the next utterance using the complementary token sequence, and further, can select the next utterance using the estimation result of the dialogue act estimator 40 whose estimation accuracy has been improved by using the complementary token sequence. [Industrial Applicability]

[0152] As described above, the speech timing estimation system and dialogue system, and program of the present invention are suitable for use in, for example, a news dialogue system that conveys the contents of an article to a user using scenario data generated from article data describing various topics such as news, columns, history, etc.; a guidance dialogue system that explains how to use a device to a user and guides users around a facility; a questionnaire dialogue system that conducts surveys of various user trends such as election conditions and consumer orientation; an information retrieval dialogue system that allows users to search for information on stores, products, travel destinations, songs they want to listen to, etc.; an operation dialogue system that allows users to operate various devices and equipment such as home appliances and cars; an education dialogue system that educates users such as children, students, new employees, etc.; and an information identification dialogue system that allows a system to identify information such as user attributes. [Explanation of symbols]

[0153] 10 Dialogue Systems 20 Speech Timing Estimation System 22. Means for measuring the time elapsed after the end of user utterance 30 Voice Recognizer 31 Sentence Generator 33 Syntactic completion estimator 40 Dialogue Act Estimator 50 Speech timing estimator 60 Next utterance preparation means 62 Speech generation means 70 Subject data storage means< / eou> < / eou> < / eou> < / eou> < / eou> < / eou> < / eou> < / eou> < / eou> < / eou> < / eou> < / eou> < / eou> < / eou> < / eou> < / eou>

Claims

1. An utterance timing estimation system configured by a computer that executes a process of estimating a timing of a change from a user utterance to a system utterance, a sentence generator that uses a recognized token sequence constituting a portion of a user's utterance obtained by speech recognition processing to estimate a subsequent token sequence constituting a portion following the recognized token sequence of the user's utterance by using a Generative Pretrained Transformer 2, 3, or 4 (GPT-2, 3, or 4) or other language model, or that generates a complementary token sequence by adding the subsequent token sequence to the recognized token sequence in addition to the estimation; an utterance timing estimator that receives as input data information including prediction information obtained from the subsequent token sequence estimated by the sentence generator or the generated complementary token sequence and acoustic features obtained from a voice signal of a user utterance, and executes a pattern recognition process to estimate a timing to start a next system utterance, The prediction information to be input to the speech timing estimator is (a) providing a syntax completion estimator that uses the multiple subsequent token strings estimated by the sentence generator to calculate a probability distribution of the number of utterance completion tokens or the utterance completion time length, which indicates how many tokens or how much time is required for the utterance to be completed, or a cumulative probability distribution that accumulates the probability of the number of utterance completion tokens or the utterance completion time length, and obtaining vector data of the probability distribution or cumulative probability distribution of the number of utterance completion tokens or the utterance completion time length calculated by the syntax completion estimator; (b) providing a dialogue act estimator that receives as input data information including linguistic features obtained from the complementary token sequence generated by the sentence generator and acoustic features obtained from a voice signal of a user utterance, and executes pattern recognition processing to estimate a dialogue act of the user utterance, or to estimate a dialogue act of a next system utterance in addition to the dialogue act of the user utterance, and outputs an estimated result of the dialogue act from the dialogue act estimator or intermediate features leading to the estimated result; (c) a linguistic feature obtained from the complemented token sequence generated by the sentence generator; At least one of the following information is The speech timing estimation system according to the present invention is characterized in that

2. a user utterance end elapsed time measuring means for measuring an elapsed time from the end of the user utterance or an alternative value indicating the length of the elapsed time by using a voice signal of the user utterance, The input data of the speech timing estimator includes the elapsed time from the end of the user's speech measured by the means for measuring the elapsed time from the end of the user's speech or a substitute value thereof.

2. The speech timing estimation system according to claim 1.

3. A dialogue system configured by a computer that executes processing for a voice dialogue with a user, The speech timing estimation system according to claim 1 or 2, A voice signal acquisition means for acquiring a voice signal of a user's speech; a voice recognizer that executes a voice recognition process on the voice signal of the user's speech acquired by the voice signal acquisition means; a next utterance preparation means for executing a preparation process for acquiring or generating content data of the next utterance of the system, using the subject data stored in the subject data storage means or the subject data stored in the external system connected via a network, at a timing that is not dependent on the period of the pattern recognition process by the utterance timing estimator constituting the utterance timing estimation system and before the start timing of the system utterance is detected by the utterance timing estimator, and using the complementary token sequence generated by the sentence generator constituting the utterance timing estimation system, or using at least a part of the dialogue history information between a user and the system in addition to the complementary token sequence; an utterance generating means for executing a system utterance generating process including reproducing a voice signal of the system utterance, using content data of a next utterance obtained by a preparation process by the next utterance preparing means, after a start timing of the system utterance is detected by the utterance timing estimator; A dialogue system comprising:

4. A dialogue system configured by a computer that executes processing for a voice dialogue with a user, A voice signal acquisition means for acquiring a voice signal of a user's speech; a voice recognizer that executes a voice recognition process on the voice signal of the user's speech acquired by the voice signal acquisition means; a sentence generator that uses a recognized token sequence constituting a portion of a user's utterance up to a middle point obtained by the speech recognition process of the speech recognizer, and estimates a subsequent token sequence constituting a portion following the recognized token sequence of the user's utterance by using a Generative Pretrained Transformer 2, 3 or 4 (GPT-2, 3 or 4) or other language model, and generates a complementary token sequence by adding the subsequent token sequence to the recognized token sequence; a next utterance preparation means for executing a preparation process for acquiring or generating content data for a next utterance of the system by using the subject data stored in a subject data storage means or the subject data stored in an external system connected via a network, and using the complement token string generated by the sentence generator, or by using the complement token string and at least a part of the dialogue history information between the user and the system; an utterance generating means for executing a system utterance generating process including reproducing a voice signal of the system utterance by using the content data of the next utterance obtained by the preparation process by the next utterance preparing means; A dialogue system comprising:

5. A program for causing a computer to function as the speech timing estimation system according to claim 1 or 2.

6. A program for causing a computer to function as the dialogue system according to claim 4.

Images