NON-VERBAL INFORMATION GENERATION DEVICE, NON-VERBAL INFORMATION GENERATION METHOD, AND PROGRAM

Abstract

To automate the association of at least one of audio information and text information with non-verbal information. [Solution] The non-verbal information generation device 40 includes a non-verbal information generation unit 64 that generates non-verbal information with time information corresponding to text features with time information based on text features with time information and a trained non-verbal information generation model, the text features with time information including features extracted from the text and time information representing the time assigned to each predetermined unit of the text, the non-verbal information is information for controlling an expression unit that expresses behavior according to the text, and the text features include features based on dialogue acts extracted from the text.

Description

[Technical field]

[0001] The present invention relates to a non-verbal information generation device, a non-verbal information generation method, and a program. This application claims priority based on Japanese Patent Application No. 2018-026516 filed in Japan on February 16, 2018, Japanese Patent Application No. 2018-026517 filed in Japan on February 16, 2018, Japanese Patent Application No. 2018-097338 filed in Japan on May 21, 2018, Japanese Patent Application No. 2018-097339 filed in Japan on May 21, 2018, and Japanese Patent Application No. 2018-230310 filed in Japan on December 7, 2018, the contents of which are incorporated herein by reference. [Background technology]

[0002] In communication, in addition to verbal behavior, non-verbal behavior has an important function in conveying emotions and intentions. Therefore, it is desirable for communication robots and communication agents to express non-verbal behavior in order to communicate smoothly with users. In light of this, a method has been proposed in which non-verbal behaviors that match utterances are registered in advance in a database (DB), and the non-verbal behaviors are expressed in accordance with the playback of the utterances (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2003-173452 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the method described in Patent Document 1 requires information on speech and non-verbal behavior (action commands consisting of pairs of gestures and lines) to be created and registered in advance, resulting in high data creation costs.

[0005] The present invention has been made in consideration of the above-mentioned circumstances, and aims to provide a non-verbal information generation device, a non-verbal information generation method, and a program that can automate the association of at least one of speech information or text information with non-verbal information. [Means for solving the problem]

[0006] In order to achieve the above-mentioned object, a non-language information generation device according to a first aspect is a non-language information generation device including a non-language information generation unit that generates non-language information with time information corresponding to text features with time information based on text features with time information and a trained non-language information generation model, wherein the text features with time information include features extracted from text and time information representing the time assigned to each predetermined unit of the text, and the non-language information is information for controlling an expression unit that expresses an action corresponding to the text.

[0007] In addition, a non-language information generation device according to a second aspect is a non-language information generation device including a non-language information generation unit that generates non-language information with time information corresponding to speech features with time information based on speech features with time information and a trained non-language information generation model, wherein the speech features with time information include features extracted from speech information and time information representing the time per predetermined unit when the speech information is uttered, and the non-language information is information for controlling an expression unit that expresses behavior corresponding to the speech information.

[0008] The non-verbal information generation device of the present invention can further include a feature extraction unit that performs text analysis on input text, assigns the playback time, assuming that the text is to be output externally as audio, to the text for each predetermined unit as time information, and extracts text features with the time information.

[0009] The text features may include at least one of dialogue acts and thesaurus information extracted from the text.

[0010] The non-language information generation device of the present invention further includes an expression unit that expresses the non-language information, and the non-language information generation unit can control the expression unit so that the non-language information with time information is expressed from the expression unit based on the time information assigned to the non-language information.

[0011] A non-verbal information generation model learning device according to a third aspect includes a learning information acquisition unit that acquires text information representing a text corresponding to a speaker's voice and time information representing a time assigned to each predetermined unit of the text; a non-verbal information acquisition unit that acquires non-verbal information representing information regarding the speaker's behavior when the speaker made an utterance corresponding to the text and time information representing the time when the behavior was performed and the time information corresponds to the non-verbal information, and creates non-verbal information with time information; a learning feature extraction unit that extracts text features with time information from the text information acquired by the learning information acquisition unit and the time information corresponding to the text information, the text features with time information representing features of the text information; and a learning unit that learns a non-verbal information generation model for generating the non-verbal information with time information acquired by the non-verbal information acquisition unit based on the text features with time information extracted by the learning feature extraction unit.

[0012] In addition, the non-language information generation model learning device of the fourth aspect includes a learning information acquisition unit that acquires speech information corresponding to the speech of a speaker and time information representing the time for each predetermined unit when the speech information is uttered; a non-language information acquisition unit that acquires non-language information representing information regarding the speaker's behavior when the speaker made an utterance corresponding to the speech and time information representing the time when the behavior was performed and which corresponds to the non-language information, and creates non-language information with time information; a learning feature extraction unit that extracts speech features with time information from the speech information acquired by the learning information acquisition unit and the time information corresponding to the speech information; and a learning unit that learns a non-language information generation model for generating the non-language information with time information acquired by the non-language information acquisition unit based on the speech features with time information extracted by the learning feature extraction unit.

[0013] In addition, a non-verbal information generation method according to a fifth aspect includes a step in which a non-verbal information generation unit generates non-verbal information with time information corresponding to the text features with time information based on text features with time information and a trained non-verbal information generation model, the text features with time information including features extracted from text and time information representing the time assigned to each predetermined unit of the text, and the non-verbal information is information for controlling an expression unit that expresses an action corresponding to the text.

[0014] In addition, a non-language information generation method according to a sixth aspect includes a step in which a non-language information generation unit generates non-language information with time information corresponding to the speech features with time information based on the speech features with time information and a trained non-language information generation model, the speech features with time information including features extracted from speech information and time information representing the time per predetermined unit when the speech information is uttered, and the non-language information is information for controlling an expression unit that expresses an action corresponding to the speech information.

[0015] Further, a program of the present invention is a program for causing a computer to function as each unit included in the non-language information generation device.

[0016] A non-verbal information generation model learning device according to a seventh aspect includes a learning information acquisition unit that acquires text information representing text corresponding to a speaker's voice and time information representing the time assigned to each predetermined unit of the text; a non-verbal information acquisition unit that acquires non-verbal information representing information regarding the behavior of a listener to the speaker's utterance when the speaker makes an utterance corresponding to the text and time information representing the time when the behavior was performed and which corresponds to the non-verbal information, and creates non-verbal information with time information; a learning feature extraction unit that extracts text features with time information from the text information acquired by the learning information acquisition unit and the time information corresponding to the text information, the text features with time information representing features of the text information; and a learning unit that learns a non-verbal information generation model for generating the non-verbal information with time information acquired by the non-verbal information acquisition unit based on the text features with time information extracted by the learning feature extraction unit.

[0017] The text features may include at least one of dialogue acts and thesaurus information extracted from the text.

[0018] In addition, a non-language information generation model learning device according to an eighth aspect includes a learning information acquisition unit that acquires speech information corresponding to a speaker's speech and time information representing the time for each predetermined unit when the speech information is uttered; a non-language information acquisition unit that acquires non-language information representing information regarding the behavior of a listener to the speaker's utterance when the speaker makes an utterance corresponding to the speech and time information representing the time when the behavior was performed and which corresponds to the non-language information, and creates non-language information with time information; a learning feature extraction unit that extracts speech features with time information from the speech information acquired by the learning information acquisition unit and the time information corresponding to the speech information; and a learning unit that learns a non-language information generation model for generating the non-language information with time information acquired by the non-language information acquisition unit based on the speech features with time information extracted by the learning feature extraction unit.

[0019] In addition, a non-language information generation model training method according to a ninth aspect includes a step of a training information acquisition unit acquiring text information representing text corresponding to the speaker's voice and time information representing the time assigned to each predetermined unit of the text; a step of a non-language information acquisition unit acquiring non-language information representing information regarding the behavior of a listener of the speaker's utterance when the speaker makes an utterance corresponding to the text and time information representing the time when the behavior was performed and the time information corresponds to the non-language information, and creating non-language information with time information; a step of a training feature extraction unit extracting text features with time information representing features of the text information from the text information acquired by the training information acquisition unit and the time information corresponding to the text information; and a step of a learning unit training a non-language information generation model for generating the non-language information with time information acquired by the non-language information acquisition unit based on the text features with time information extracted by the training feature extraction unit.

[0020] In addition, a non-language information generation model training method according to a tenth aspect includes a step of a training information acquisition unit acquiring speech information corresponding to the speech of a speaker and time information representing the time for each predetermined unit when the speech information is uttered; a step of a non-language information acquisition unit acquiring non-language information representing information regarding the behavior of a listener of the speaker's utterance when the speaker makes an utterance corresponding to the speech and time information representing the time when the behavior was performed and the time information corresponds to the non-language information, and creating non-language information with time information; a step of a training feature extraction unit extracting speech features with time information representing features of the speech information from the speech information acquired by the training information acquisition unit and the time information corresponding to the speech information; and a step of a learning unit training a non-language information generation model for generating the non-language information with time information acquired by the non-language information acquisition unit based on the speech features with time information extracted by the training feature extraction unit.

[0021] Further, a program of the present invention is a program for causing a computer to function as each unit included in the non-linguistic information generation model learning device. Effect of the Invention

[0022] As described above, the non-language information generation device, non-language information generation method, and program according to the present invention can automate the association of at least one of speech information and text information with non-language information. [Brief description of the drawings]

[0023] [Figure 1] 1 is a block diagram showing an example of a configuration of a non-linguistic information generation model learning device according to a first embodiment. [Diagram 2] FIG. 11 is an explanatory diagram for explaining a method of acquiring learning data. [Diagram 3] FIG. 2 is an explanatory diagram for explaining speech features and text features used in the present embodiment. [Figure 4]1 is a block diagram showing an example of a configuration of a non-language information generation device according to a first embodiment. [Diagram 5] 5 is a flowchart showing an example of the flow of a learning process according to the first embodiment. [Figure 6] 10 is a flowchart showing an example of the flow of non-language information generation processing according to the first embodiment. [Figure 7] FIG. 11 is a block diagram showing an example of a configuration of a non-language information generation model learning device according to a second embodiment. [Figure 8] 13 is a diagram illustrating an example of a detailed configuration of a learning information acquisition unit according to the second embodiment. FIG. [Figure 9] FIG. 11 is a block diagram showing an example of a configuration of a non-language information generation device according to a second embodiment. [Figure 10] 13 is a diagram illustrating an example of a detailed configuration of an information acquisition unit according to the second embodiment. FIG. [Figure 11] 10 is a flowchart showing an example of the flow of a learning process according to the second embodiment. [Figure 12] 13 is a flowchart showing an example of the flow of a non-language information generation process according to the second embodiment. [Figure 13A] FIG. 13 is an explanatory diagram for explaining a learning information acquisition unit and a learning feature extraction unit of the third embodiment. [Figure 13B] FIG. 13 is an explanatory diagram for explaining an information acquiring unit and a feature amount extracting unit according to the third embodiment. [Figure 14A] FIG. 13 is an explanatory diagram for explaining a modified example of the third embodiment. [Figure 14B] FIG. 13 is an explanatory diagram for explaining a modified example of the third embodiment. [Figure 15] FIG. 11 is an explanatory diagram for explaining a combination of configurations according to other embodiments. [Figure 16] FIG. 11 is an explanatory diagram for explaining a combination of configurations according to other embodiments. [Figure 17] FIG. 13 is a block diagram showing an example of the configuration of a non-linguistic information generation model learning device according to the seventh embodiment. [Figure 18] FIG. 4 is an explanatory diagram for explaining additional information used in the present embodiment. [Figure 19] FIG. 13 is a block diagram showing an example of the configuration of a non-language information generation device according to a seventh embodiment. [Figure 20] 20 is a flowchart showing an example of the flow of a learning process according to the seventh embodiment. [Figure 21] 23 is a flowchart showing an example of the flow of non-language information generation processing according to the seventh embodiment. [Figure 22] 13 is a block diagram showing an example of a configuration of a non-linguistic information generation model learning device according to another example of the seventh embodiment. [Figure 23] FIG. 23 is a block diagram showing an example of the configuration of a non-language information generation device according to another example of the seventh embodiment. [Figure 24] 11 is a diagram for explaining a method of changing time information using additional information. FIG. [Diagram 25] FIG. 13 is a block diagram showing an example of a configuration of a non-linguistic information generation model learning device according to an eighth embodiment. [Figure 26A] FIG. 13 is a diagram for explaining a method for adding detailed time information. [Figure 26B] FIG. 23 is a block diagram showing an example of the configuration of a non-language information generation device according to an eighth embodiment. [Figure 27] 23 is a flowchart showing an example of the flow of a learning process according to the eighth embodiment. [Figure 28] 23 is a flowchart showing an example of the flow of non-language information generation processing according to the eighth embodiment. [Figure 29] 23 is a flowchart showing an example of the flow of a learning process according to the ninth embodiment. [Diagram 30] 23 is a flowchart showing an example of the flow of non-language information generation processing according to the ninth embodiment. [Diagram 31] FIG. 23 is a block diagram showing an example of the configuration of a non-language information generation device according to an eleventh embodiment. [Diagram 32] FIG. 23 is a diagram showing an example of a display screen according to the eleventh embodiment. [Diagram 33] FIG. 11 is a diagram for explaining a change instruction for time information. [Diagram 34] FIG. 11 is a diagram for explaining a change instruction for time information. [Diagram 35] 23 is a flowchart showing an example of the flow of non-language information generation processing according to the eleventh embodiment. [Diagram 36] 23 is a flowchart showing an example of the flow of a display control process according to an eleventh embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] Hereinafter, an example of an embodiment of the present invention will be described in detail with reference to the drawings.

[0025] <Outline of this embodiment>

[0026] In an embodiment of the present invention, a characteristic that, when humans communicate, the transmission of voice information and linguistic information contained in the voice information occurs simultaneously with non-verbal behavior is utilized. Specifically, in this embodiment, at least one of voice information of an utterance and text information representing the contents of the utterance is input X, non-verbal information representing the speaker's non-verbal behavior generated together with the speaker's utterance is output Y, and the output Y is generated from the input X by machine learning. The non-verbal information is information about behavior other than language itself. Examples of non-verbal behavior include, for example, head movement, gaze direction, hand gestures, upper body movement, and types (classes) of lower body movement.

[0027] The non-verbal information obtained in this embodiment is used for generating gestures in CG (Computer Graphics) animations used in communication robots, communication agents, games, and interactive systems that have physical characteristics similar to humans and communicate with humans.

[0028] [First embodiment]

[0029] <Configuration of a learning device for non-linguistic information generation model>

[0030] Fig. 1 is a block diagram showing an example of the configuration of a non-verbal information generation model learning device 10 according to the first embodiment. As shown in Fig. 1, the non-verbal information generation model learning device 10 according to this embodiment is configured by a computer including a CPU (Central Processing Unit), a RAM (Random Access Memory), and a ROM (Read Only Memory) that stores a program for executing a learning processing routine described later. The non-verbal information generation model learning device 10 functionally includes a learning input unit 20 and a learning calculation unit 30.

[0031] The learning input unit 20 receives learning speech information and learning non-verbal information that represents information related to behavior other than language.

[0032] The learning data representing a combination of learning speech information and learning non-verbal information, which is input data to the non-verbal information generation model learning device 10 of this embodiment, is created, for example, by acquiring speech information (X) of a speaker who is speaking and at the same time acquiring non-verbal information (Y) of the speaker who is speaking using a predetermined measurement device in a scene such as that shown in Fig. 2. Note that the speech information (X) corresponds to the speech information when the speaker speaks to the outside.

[0033] The learning calculation unit 30 generates a non-language information generation model for generating non-language information with time information, based on the learning data received by the learning input unit 20. As shown in FIG. 1, the learning calculation unit 30 includes a learning information acquisition unit 31, a learning feature extraction unit 32, a non-language information acquisition unit 33, a generation parameter extraction unit 34, a learning unit 35, and a trained model storage unit 36.

[0034] The learning information acquisition unit 31 acquires the learning voice information accepted by the learning input unit 20. The learning information acquisition unit 31 also acquires time information indicating the time from the start time to the end time when the learning voice information is uttered.

[0035] The learning feature extraction unit 32 extracts learning audio features with time information from the learning audio information and time information acquired by the learning information acquisition unit 31, which represent the features of the learning audio information.

[0036] For example, the learning feature extraction unit 32 performs a predetermined speech information processing on the learning speech information acquired by the learning information acquisition unit 31, and extracts fundamental frequency (F0), power, MFCC (Mel Frequency Cepstral Coefficients), etc. as speech features. These speech features are extracted using a window width T A,w As shown in Figure 3, these multi-dimensional speech features can be calculated using the speech information.

[0037]

number

[0038] Here,

[0039]

number

[0040] is the time t A,s From, window width T A,w The speech features are calculated from the speech information corresponding to the time. The window width does not need to be the same for all speech features, and the features may be extracted separately. The method of extracting these speech features is common, and various methods have already been proposed (see Reference 1, for example). Therefore, any method may be used.

[0041] [Reference 1]: Seiichi Nakagawa, "Speech and Natural Language Processing", March 1, 2013, Corona Publishing

[0042] The non-verbal information acquisition unit 33 acquires the non-verbal information for learning accepted by the learning input unit 20, and acquires time information representing the time from the start time to the end time when the behavior represented by the non-verbal information for learning is performed.

[0043] The non-verbal information acquisition unit 33 acquires information on nodding, face direction, hand gestures, gaze, facial expression, body posture, etc. as non-verbal information for learning. An example of parameters representing information on nodding, face direction, hand gestures, gaze, facial expression, body posture, etc. is shown below.

[0044] [Table 1]

[0045] Note that FACS stands for Facial Action Coding System, and AU stands for Action Unit. For example, non-verbal information is expressed by labels, such as AU1 being "lifting the inner eyebrows (AU1)." Other non-verbal information includes, for example, the speaker's gaze behavior, head movement, breathing movement, and changes in mouth shape.

[0046] As described above, the non-verbal information may be any parameter related to the event, such as the joints, positions, and movements of the body. Various measurement methods are considered, and any method may be used (see, for example, References 2 and 3).

[0047] [Reference 2]: Masaaki Makikawa, Masayuki Nanbu, Shigehiro Shiozawa, Shima Okada, Masaki Yoshida, "Measurement Technology for Human Mental and Physical State - Daily Measurement for Human-Friendly Product Development", October 1, 2010, Corona Publishing [Reference 3]: Shihong Xia, Lin Gao, Yu-Kun Lai, Ming-Ze Yuan, Jinxiang Chai, "A Survey on Human Performance Capture and Animation", Journal of Computer Science and Technology, Volume 32, Issue 3, pp 536-554, (2017).

[0048] As shown in FIG. 2 above, for example, the face orientation can be measured by a behavior capture device such as a head measurement device (head tracker) in the head direction of a human or an interactive humanoid. Also, it is possible to obtain the face orientation from animation data such as CG. Further, as shown in FIG. 2, the face orientation is represented by the angles of three axes, Yaw, Roll, and Pitch, for example.

[0049] The generation parameter extraction unit 34 discretizes the parameters (sensory scales) represented by the non-verbal information for learning acquired by the non-verbal information acquisition unit 33, and extracts the discretized non-verbal information with time information. For example, although the face orientation is represented by the angle information of Yaw, Roll, and Pitch, arbitrary thresholds α and β (α < β) may be determined in advance and converted into a nominal scale as shown below. Note that only Yaw is presented in the example shown below.

[0050] -α < Yaw < α: Front

[0051] α ≤ Yaw < β: Slightly turned to the left

[0052] β ≤ Yaw: Greatly turned to the left

[0053] -β < Yaw ≤ -α: Slightly turned to the right

[0054] -β ≥ Yaw: Greatly turned to the right

[0055] In this way, the non-language information for learning acquired by the non-language information acquisition unit 33 is discretized and time information is added to it, and the discretized information is expressed as a multidimensional vector

[0056]

number

[0057] Let t N,s ,t N,e are the start and end times when the non-verbal information was obtained.

[0058] The learning unit 35 learns a non-language information generation model for generating non-language information with time information from the speech features with time information, based on the learning speech features with time information extracted by the learning feature extraction unit 32 and the discretized non-language information with time information acquired by the generation parameter extraction unit 34.

[0059] Specifically, the learning unit 35 extracts the learning speech features with time information extracted by the learning feature extraction unit 32.

[0060]

number

[0061] The input is non-verbal information with time information.

[0062]

number

[0063] A non-verbal information generation model that outputs the following is constructed. Any machine learning method may be used to construct the non-verbal information generation model, but in this embodiment, a Support Vector Machine (SVM) is used. For example, the following is obtained by using the SVM:

[0064]

number

[0065] We construct classifiers for the parameters of each dimension in the model, and construct regression models using SVR (Support Vector Machine for Regression), which applies SVM to regression.

[0066] Furthermore, in this embodiment, for each type of behavior represented by non-verbal information, an SVM model is created for estimating the presence or absence of that type of behavior.

[0067] In addition, the non-verbal information generation model can estimate non-verbal information at any time resolution and using parameters at any time.

[0068]

number

[0069] An example of the feature used for estimating the following is shown below. The speech feature obtained at the time T1 to T2 to be estimated is

[0070]

number

[0071] and the output gesture

[0072]

number

[0073] The trained non-verbal information generation model is then called a trained model.

[0074]

number

[0075] Let us assume that.

[0076] The trained model storage unit 36 ​​stores a trained non-language information generation model trained by the training unit 35. The trained non-language information generation model generates non-language information with time information from speech features with time information.

[0077] <System configuration of non-verbal information generation device>

[0078] Fig. 4 is a block diagram showing an example of the configuration of a non-language information generation device 40 according to the first embodiment. As shown in Fig. 4, the non-language information generation device 40 according to this embodiment is configured by a computer equipped with a CPU (Central Processing Unit), a RAM (Random Access Memory), and a ROM (Read Only Memory) that stores a program for executing a non-language information generation processing routine described below. Functionally, the non-language information generation device 40 is equipped with an input section 50, a calculation section 60, and a development section 70.

[0079] The input unit 50 receives voice information and time information indicating the time from the start time to the end time when the voice information is uttered.

[0080] The calculation unit 60 includes an information acquisition unit 61, a feature extraction unit 62, a trained model storage unit 63, and a non-language information generation unit 64.

[0081] The information acquiring unit 61 acquires, received by the input unit 50, the voice information and time information indicating the time from the start time to the end time when the voice information is uttered.

[0082] The feature extraction unit 62, like the learning feature extraction unit 32, extracts, from the voice information and time information acquired by the information acquisition unit 61, voice features with time information that represent the features of the voice information.

[0083] The trained model storage unit 63 stores a trained non-language information generation model that is the same as the trained non-language information generation model stored in the trained model storage unit 36.

[0084] The non-language information generation unit 64 generates non-language information with time information corresponding to the voice features with time information extracted by the feature extraction unit 62 based on the voice features with time information extracted by the feature extraction unit 62 and the trained non-language information generation model stored in the trained model memory unit 63.

[0085] For example, the non-language information generation unit 64 uses the trained non-language information generation model stored in the trained model storage unit 63.

[0086]

number

[0087] Using the above, we can extract any feature as a speech feature with time information.

[0088]

number

[0089] The input is gestures as non-verbal information with time information.

[0090]

number

[0091] Get the.

[0092] Then, the non-language information generating unit 64 controls the expressing unit 70 so that the generated non-language information with time information is output from the expressing unit 70 based on the time information added to the non-language information.

[0093] Specifically, the non-verbal information generating unit 64 generates a gesture

[0094]

number

[0095] is reflected as the motion of an arbitrary object (e.g., an animation character, a robot, etc.).

[0096] The expression unit 70, under the control of the non-language information generation unit 64, expresses the speech information received by the input unit 50 and the non-language information generated by the non-language information generation unit 64.

[0097] Examples of the expression unit 70 include a communication robot, a communication agent displayed on a display, and CG animation used in games and interactive systems.

[0098] <Function of the Non-verbal Information Generation Model Learning Device 10>

[0099] Next, a description will be given of the operation of the non-verbal information generation model learning device 10 according to this embodiment. First, when learning data representing a combination of a plurality of pieces of speech information for learning and a plurality of pieces of non-verbal information for learning are input to the learning input unit 20 of the non-verbal information generation model learning device 10, the non-verbal information generation model learning device 10 executes a learning process routine shown in FIG.

[0100] First, in step S100, the learning information acquisition unit 31 acquires learning voice information from the multiple learning data accepted by the learning input unit 20 and time information representing the time from the start time to the end time when the learning voice information is emitted.

[0101] In step S102, the non-verbal information acquisition unit 33 acquires non-verbal information for learning from the multiple learning data accepted by the learning input unit 20 and time information representing the time from the start time to the end time when the behavior represented by the non-verbal information for learning is performed.

[0102] In step S104, the learning feature extraction unit 32 extracts learning speech features with time information from the learning speech information and time information acquired in step S100.

[0103] In step S106, the generation parameter extraction unit 34 extracts discretized non-language information with time information from the learning non-language information and time information acquired in step S102.

[0104] In step S108, the learning unit 35 learns a non-language information generation model for generating non-language information with time information from the speech features with time information, based on the learning speech features with time information extracted in step S104 and the non-language information with time information acquired in step S106.

[0105] In step S110, the learning unit 35 stores the trained non-language information generation model obtained in step S108 in the trained model storage unit 36, and ends the learning process routine.

[0106] <Function of the non-verbal information generation device 40>

[0107] Next, the operation of the non-language information generation device 40 according to this embodiment will be described. First, when the trained non-language information generation model stored in the trained model storage unit 36 ​​of the non-language information generation model learning device 10 is input to the non-language information generation device 40, the trained non-language information generation model is stored in the trained model storage unit 63 of the non-language information generation device 40. Then, when speech information for which non-language information is to be generated is input to the input unit 50, the non-language information generation device 40 executes a non-language information generation processing routine shown in FIG. 6.

[0108] In step S200, the information acquiring unit 61 acquires, received by the input unit 50, the voice information and the time information indicating the time from the start time to the end time when the voice information is uttered.

[0109] In step S202, the feature extracting unit 62, like the learning feature extracting unit 32, extracts speech features with time information from the speech information and time information acquired in step S200.

[0110] In step S204, the non-language information generation unit 64 reads out the trained non-language information generation model stored in the trained model storage unit 63.

[0111] In step S206, the non-language information generation unit 64 generates non-language information with time information corresponding to the speech features with time information extracted in step S202 based on the speech features with time information extracted in step S202 and the trained non-language information generation model read out in step S204.

[0112] In step S208, the non-verbal information generation unit 64 controls the expression unit 70 so that the non-verbal information with time information generated in step S206 above is output from the expression unit 70 based on the time information assigned to the non-verbal information, and then terminates the non-verbal information generation processing routine.

[0113] As described above, the non-language information generation device 40 according to the first embodiment extracts speech features with time information from speech information and time information, and generates non-language information with time information corresponding to the speech features with time information based on the extracted speech features with time information and a trained non-language information generation model for generating non-language information with time information. This makes it possible to automatically associate speech information with non-language information, thereby reducing the cost involved.

[0114] When making a communication robot or dialogue agent perform gestures or other actions in accordance with the spoken voice or the corresponding text, it is necessary to determine what actions and when to make them perform the actions in accordance with the speech. Conventionally, these were all created and set manually as scenarios, which resulted in high creation costs.

[0115] In contrast, in this embodiment, a trained non-language information generation model is generated for generating non-language information with time information from speech features with time information, so that speech information is input and non-language information with time information corresponding to the action corresponding to the input (non-language information with output timing attached) is output.

[0116] As a result, according to this embodiment, non-verbal information can be automatically generated from voice information, and there is no need to register non-verbal information for each utterance separately as in the conventional technology, which significantly reduces costs. In addition, by using this embodiment, non-verbal behavior can be generated at a natural, human-like timing in response to input voice information, and the effects include improving the human-likeness and naturalness of agents and robots, facilitating the communication of intentions through non-verbal behavior, and activating conversations.

[0117] In addition, by using a pre-trained non-verbal information generation model, spoken voice or text is input, and information on the actions and timing corresponding to the input is output. This reduces the cost of creating scenarios. In addition, since the actions are generated based on the actual actions of humans, movements can be reproduced with more natural timing.

[0118] Furthermore, according to the non-language information generation model learning device 10 of the first embodiment, a non-language information generation model for generating non-language information with time information from speech features with time information is learned based on speech features for learning with time information and non-language information for learning with time information. This makes it possible to obtain a non-language information generation model for generating non-language information from speech features while reducing the cost of associating speech information with non-language information.

[0119] In addition, by using a trained non-verbal information generation model, non-verbal behavior can be generated at natural timing.

[0120] In the above first embodiment, a case where non-verbal information is generated from speech features has been described as an example. In the above first embodiment, it is possible to generate non-verbal information with a minimum necessary configuration by using information (e.g., emotions) expressed as speech features as a clue, without going into the content of what is being said.

[0121] In the above first embodiment, the content of what is being said is not addressed (language information is not used), so for example, sensors may be attached to an animal to obtain non-verbal information and audio information (e.g., cries, etc.) and then used to move an animal-type robot.

[0122] [Second embodiment]

[0123] <Configuration of a learning device for non-linguistic information generation model>

[0124] Next, a second embodiment of the present invention will be described. Note that parts having the same configuration as the first embodiment are given the same reference numerals and the description thereof will be omitted.

[0125] In the second embodiment, text information is used as an input instead of speech information. The second embodiment differs from the first embodiment in that a non-verbal information generation model that generates non-verbal information from text information is trained. Note that the text information used in the second embodiment is text information that represents the contents of an utterance when a speaker speaks to the outside by voice.

[0126] Fig. 7 is a block diagram showing an example of the configuration of a non-verbal information generation model learning device 210 according to the second embodiment. As shown in Fig. 7, the non-verbal information generation model learning device 210 according to the second embodiment is configured by a computer including a CPU, a RAM, and a ROM storing a program for executing a learning processing routine described later. The non-verbal information generation model learning device 210 functionally includes a learning input unit 220 and a learning calculation unit 230.

[0127] The learning input unit 220 receives learning text information and learning non-language information.

[0128] The learning calculation unit 230 generates a non-language information generation model for generating non-language information with time information, based on the learning data received by the learning input unit 220. As shown in Fig. 7, the learning calculation unit 230 includes a learning information acquisition unit 231, a learning feature extraction unit 232, a non-language information acquisition unit 33, a generation parameter extraction unit 34, a learning unit 235, and a trained model storage unit 236.

[0129] The learning information acquisition unit 231 acquires learning voice information corresponding to learning text information, and acquires time information indicating the time from the start time to the end time when the voice information is uttered. As shown in Fig. 8, the learning information acquisition unit 231 includes a learning text analysis unit 237 and a learning voice synthesis unit 238.

[0130] The learning text analysis unit 237 performs a predetermined text analysis on the learning text information and obtains a text analysis result. For example, the learning text analysis unit 237 performs text analysis, including morpheme analysis, on the learning text information to extract, for each morpheme, word notation (morpheme) information, parts of speech, category information, evaluation expressions, emotional expressions, sensibility expressions, onomatopoeia, mimetic words, named entities, themes, number of characters, position, thesaurus information, and the like, and extracts, for each sentence, dialogue acts of utterances, and the like. Note that, instead of for each morpheme, word notation (morpheme) information, parts of speech, category information, evaluation expressions, emotional expressions, named entities, themes, number of characters, position, thesaurus information, and the like may be extracted for each phrase. Also, word notation (morpheme) information, parts of speech, category information, evaluation expressions, emotional expressions, sensibility expressions, onomatopoeia, mimetic words, named entities, themes, number of characters, position, thesaurus information, and the like may be extracted for any unit other than morpheme or phrase. For example, extraction may be performed on a character basis, or in the case of English, on a character string basis separated by a space or on a phrase basis. Extraction of the subject may be performed on a sentence basis or on an utterance basis. Here, the dialogue act is an abstraction of the intention in an utterance, abstracted as a label. The subject is information that indicates the topic or focus of a text. The number of characters is the number of characters in a morpheme or phrase. The position is the position of a morpheme or phrase from the beginning or end of a sentence. Thesaurus information is thesaurus information of words in a morpheme or phrase based on the Japanese Lexicon. These text feature extraction methods may be general, and various methods have already been proposed (see Reference 1 above and References 4 to 6 below). In this embodiment, a case in which word notation (morpheme) information, parts of speech, dialogue acts, number of characters, position, and thesaurus information are used among these pieces of information will be described as an example.

[0131] [Reference 4]: R Higashinaka, K Imamura, T Meguro, C Miyazaki, N Kobayashi, H Sugiyama, T Hirano, T Makino, Y Matsuo, "Towards an open-domain conversational system fully based on natural language processing", In Proceedings of International conference on Computational linguistics, pp. 928-939, 2014

[0132] [Reference 5]: Japanese Patent Publication No. 2014-222399

[0133] [Reference 6]: Japanese Patent Publication No. 2015-045915

[0134] The training voice synthesis unit 238 synthesizes training voice information corresponding to the training text information based on the text analysis result acquired by the training text analysis unit 237. For example, the training voice synthesis unit 238 performs voice synthesis using the text analysis result to generate an utterance corresponding to the text information, and sets the result as training voice information corresponding to the training text information.

[0135] Furthermore, the learning voice synthesis unit 238 acquires time information indicating the time from the start time to the end time when the learning voice information is uttered. Specifically, the learning voice synthesis unit 238 acquires time information corresponding to the start time to the end time of the speech generated by voice synthesis. This time information corresponds to each morpheme of the text information corresponding to the speech. Note that the start time and the end time may be acquired for each character included in the text information.

[0136] The training feature extraction unit 232 extracts training text features with time information, which represent the features of the training text information, from the training text information and time information acquired by the training speech synthesis unit 238. Specifically, the training feature extraction unit 232 adds time information to the text information for each predetermined analysis unit, and extracts text features with time information.

[0137] Specifically, the training feature extraction unit 232 performs sentence division on the training text information output by the training speech synthesis unit 238. Next, the training feature extraction unit 232 extracts text features related to the dialogue acts obtained by the training text analysis unit 237.

[0138]

number

[0139] Extract each sentence.

[0140]

number

[0141] are the start and end times of an utterance corresponding to a sentence.

[0142] Furthermore, for each of the multiple morphemes constituting each sentence obtained by division, the learning feature extraction unit 232 extracts at least word notation information from among the word notation information, part of speech, category information (e.g., noun, proper expression, and predicate), evaluation expression, emotional expression, proper expression, number of characters, position in the sentence, thesaurus information, etc. of the morpheme. Then, the learning feature extraction unit 232 extracts these multidimensional features as

[0143]

number

[0144] In addition,

[0145]

number

[0146] are the start and end times of the speech corresponding to the morpheme unit. An example of text features with time information is shown in Figure 3. As shown in Figure 3, the start and end times of each morpheme of the text information are obtained.

[0147] For each of the multiple phrases constituting each sentence obtained by division, the word notation information, part of speech, category information (e.g., noun, proper expression, and predicate), evaluation expression, emotional expression, proper expression, number of characters, position in the sentence, thesaurus information, etc. of the phrase may be extracted. Then, the learning feature extraction unit 232 extracts these multidimensional features as

[0148]

number

[0149] In addition,

[0150]

number

[0151] are the start and end times of the speech corresponding to the phrase unit.

[0152] In the learning information acquisition unit 231, information obtained when performing voice recognition and voice synthesis may be used.

[0153] The learning unit 235 learns a non-language information generation model for generating non-language information with time information from text features with time information, based on the learning text features with time information extracted by the learning feature extraction unit 232 and the learning discretized non-language information with time information extracted by the generation parameter extraction unit 34.

[0154] Specifically, the learning unit 235 extracts the learning text feature with time information extracted by the learning feature extraction unit 232.

[0155]

number

[0156]

number

[0157] With input, non-linguistic information

[0158]

number

[0159] When constructing the non-language information generation model, any machine learning method may be used, but in this embodiment, SVM is used.

[0160] In addition, the non-verbal information generation model can estimate non-verbal information at any time resolution and using parameters at any time.

[0161]

number

[0162] An example of the feature used for estimating is shown below. The language feature obtained at the time T1 to T2 to be estimated is

[0163]

number

[0164] and the output gesture

[0165]

number

[0166] The trained non-verbal information generation model is then called a trained model.

[0167]

number

[0168] As a method of setting T1 and T2, for example, when non-linguistic information is estimated on a morpheme-by-morpheme basis, the start time and end time of each morpheme are set as T1 and T2. In this case, the window width from T2 to T1 differs for each morpheme.

[0169] The trained model storage unit 236 stores a trained non-language information generation model trained by the training unit 235. The trained non-language information generation model generates non-language information with time information from text features with time information.

[0170] <Configuration of non-verbal information generation device>

[0171] Fig. 9 is a block diagram showing an example of the configuration of a non-language information generation device 240 according to the second embodiment. As shown in Fig. 9, the non-language information generation device 240 according to this embodiment is configured by a computer equipped with a CPU (Central Processing Unit), a RAM (Random Access Memory), and a ROM (Read Only Memory) that stores a program for executing a non-language information generation processing routine described below. Functionally, the non-language information generation device 240 is equipped with an input section 250, a calculation section 260, and a development section 70.

[0172] The input unit 250 accepts text information.

[0173] The calculation unit 260 includes an information acquisition unit 261 , a feature extraction unit 262 , a trained model storage unit 263 , and a non-language information generation unit 264 .

[0174] The information acquisition unit 261 acquires text information accepted by the input unit 250. The information acquisition unit 261 also acquires voice information corresponding to the text information, and acquires time information indicating the time from the start time to the end time when the voice information is uttered. As shown in Fig. 10, the information acquisition unit 261 includes a text analysis unit 265 and a voice synthesis unit 266.

[0175] The text analysis unit 265, like the training text analysis unit 237, performs a predetermined text analysis on the text information received by the input unit 250, and obtains the text analysis result.

[0176] The voice synthesis unit 266, like the training voice synthesis unit 238, synthesizes voice information corresponding to text information based on the text analysis result acquired by the text analysis unit 265. Then, the voice synthesis unit 266 acquires time information corresponding to the start time to the end time of the spoken voice generated by voice synthesis.

[0177] Similar to the learning feature extraction unit 232, the feature extraction unit 262 extracts, from the text information and time information acquired by the information acquisition unit 261, text features with time information that represent the features of the text information.

[0178] The trained model storage unit 263 stores a trained non-language information generation model that is the same as the trained non-language information generation model stored in the trained model storage unit 236.

[0179] The non-language information generation unit 264 generates non-language information with time information corresponding to the text features with time information extracted by the feature extraction unit 262, based on the text features with time information extracted by the feature extraction unit 262 and the trained non-language information generation model stored in the trained model memory unit 263.

[0180] For example, the non-language information generation unit 264 uses the trained non-language information generation model stored in the trained model storage unit 263.

[0181]

number

[0182] Using the above, we can extract any feature as a text feature with time information.

[0183]

number

[0184] The input is the gesture as a generation parameter corresponding to non-verbal information with time information.

[0185]

number

[0186] Get the.

[0187] Then, the non-language information generating section 264 controls the expressing section 70 so that the generated generation parameters with the time information are output from the expressing section 70.

[0188] Specifically, the non-verbal information generating unit 264 generates a gesture

[0189]

number

[0190] is reflected as the motion of an arbitrary object (e.g., an animation character, a robot, etc.).

[0191] The expression section 70, under the control of the non-language information generation section 264, expresses speech information corresponding to the text information received by the input section 250 and the non-language information generated by the non-language information generation section 264.

[0192] <Function of the Non-verbal Information Generation Model Learning Device 210>

[0193] Next, the operation of the non-language information generation model learning device 210 according to the second embodiment will be described. First, when learning data representing a combination of a plurality of pieces of text information for learning and a plurality of pieces of non-language information for learning is input to the learning input unit 220 of the non-language information generation model learning device 210, the non-language information generation model learning device 210 executes a learning process routine shown in FIG.

[0194] First, in step S300, the learning information acquiring unit 231 acquires learning text information from among the multiple pieces of learning data accepted by the learning input unit 220 (specifically, pairs of text information and non-language information).

[0195] In step S303, the training text analysis unit 237 performs a predetermined text analysis on the training text information acquired in step S300, and acquires a text analysis result. The training voice synthesis unit 238 synthesizes training voice information corresponding to the training text information, based on the text analysis result acquired by the training text analysis unit 237. The training voice synthesis unit 238 then acquires time information indicating the time from the start time to the end time when the training voice information is uttered.

[0196] In step S304, the learning feature extraction unit 232 extracts learning text features with time information from the learning text information and time information acquired in step S303.

[0197] In step S308, the learning unit 235 learns a non-linguistic information generation model for generating generation parameters with time information from the text features with time information, based on the learning text features with time information extracted in step S304 and the learning generation parameters with time information acquired in step S106.

[0198] <Function of the non-verbal information generating device 240>

[0199] Next, the operation of the non-language information generation device 240 according to the second embodiment will be described. First, when the trained non-language information generation model stored in the trained model storage unit 236 of the non-language information generation model learning device 210 is input to the non-language information generation device 240, the trained non-language information generation model is stored in the trained model storage unit 263 of the non-language information generation device 240. Then, when text information for which non-language information is to be generated is input to the input unit 250, the non-language information generation device 240 executes a non-language information generation processing routine shown in FIG.

[0200] In step S400, information acquisition unit 261 acquires text information accepted by input unit 250.

[0201] In step S401, the text analysis unit 265 performs a predetermined text analysis on the text information acquired in step S400, and acquires a text analysis result. In addition, the voice synthesis unit 266 synthesizes voice information corresponding to the text information based on the text analysis result acquired by the text analysis unit 265. Then, the voice synthesis unit 266 acquires time information indicating the time from the start time to the end time when the voice information is uttered.

[0202] In step S402, the feature extraction unit 262 extracts text features with time information from the text information and time information acquired in step S401.

[0203] In step S404 , the non-language information generation unit 264 reads out the trained non-language information generation model stored in the trained model storage unit 263 .

[0204] In step S406, the non-language information generation unit 264 generates generation parameters with time information corresponding to the text features with time information extracted in step S402 based on the text features with time information extracted in step S402 and the trained non-language information generation model read out in step S404.

[0205] Other configurations and operations of the non-language information generation device and the non-language information generation model learning device according to the second embodiment are similar to those of the first embodiment, and therefore description thereof will be omitted.

[0206] As described above, the non-language information generation device 240 according to the second embodiment acquires voice information corresponding to text information, acquires time information indicating the time from the start time to the end time when the voice information is uttered, and generates non-language information with time information corresponding to the text feature with time information based on the text feature with time information and a trained model for generating non-language information with time information. This makes it possible to automatically associate text information with non-language information, thereby reducing the cost involved.

[0207] In addition, in this embodiment, a trained non-verbal information generation model is generated for generating non-verbal information with time information from text features with time information, so that text information is input and non-verbal information with time information corresponding to the action corresponding to the input (non-verbal information with output timing attached) is output.

[0208] As a result, according to this embodiment, non-verbal information can be automatically generated from text information, and there is no need to register non-verbal information for each utterance separately as in the conventional technology, which significantly reduces costs. In addition, by using this embodiment, non-verbal behavior can be generated at a natural, human-like timing in response to input text information, and the effects of this embodiment include improving the human-likeness and naturalness of agents and robots, facilitating the communication of intentions through non-verbal behavior, and activating conversations.

[0209] Furthermore, according to the non-language information generation model learning device 210 of the second embodiment, a non-language information generation model for generating non-language information with time information from text features with time information is learned based on learning text features with time information and learning non-language information with time information. This makes it possible to obtain a non-language information generation model for generating non-language information from text features while reducing the cost of associating text information with non-language information.

[0210] In the above second embodiment, a case where non-verbal information is generated from text features has been described as an example. In the above second embodiment, it is possible to generate non-verbal information using information such as word notation, parts of speech, and dialogue acts as clues. By using this configuration, it is possible to generate non-verbal information with a minimum necessary configuration when input is not accompanied by voice, such as in a chat dialogue.

[0211] [Third embodiment]

[0212] Next, a third embodiment of the present invention will be described. Note that parts having the same configuration as the first or second embodiment are given the same reference numerals and the description thereof will be omitted.

[0213] In the third embodiment, both voice information and text information are used as input. The third embodiment differs from the first and second embodiments in that non-verbal information is generated from the voice information and the text information. Note that the text information used in the third embodiment is text information that represents the contents of an utterance when a speaker speaks to the outside by voice.

[0214] FIG. 13A shows an example of the configuration of the learning information acquisition unit 331 and the learning feature extraction unit 332 in the non-language information generation model learning device of the third embodiment.

[0215] As shown in FIG. 13A, when training voice information is input, the training voice recognition unit 337 performs a predetermined voice recognition process on the training voice information and obtains text information corresponding to the training voice information (hereinafter referred to as training recognition text).

[0216] Then, the training text analysis unit 338 performs a predetermined text analysis on the training recognition text and obtains the text analysis result.

[0217] The training feature extraction unit 332 then extracts speech features with time information from the training speech information, and also extracts text features with time information from the training recognized text.

[0218] A learning unit (not shown) of the third embodiment learns a non-language information generation model based on the learning speech features with time information, the learning text features with time information, and the learning non-language information. This makes it possible to obtain a non-language information generation model for generating non-language information with time information from the learning speech features with time information and the learning text features with time information.

[0219] FIG. 13B shows an example of the configuration of the information acquisition unit 361 and the feature extraction unit 362 in the non-language information generation device of the third embodiment.

[0220] 13B, when voice information is input, the voice recognition unit 365 performs a predetermined voice recognition process on the voice information to obtain text information corresponding to the voice information (hereinafter, referred to as recognized text). Then, the text analysis unit 366 performs a predetermined text analysis on the recognized text to obtain a text analysis result.

[0221] The feature extraction unit 362 then extracts speech features with time information from the speech information, and also extracts text features with time information from the recognized text.

[0222] A non-language information generation unit (not shown) of the third embodiment generates non-language information with time information based on the speech features with time information and the text features with time information and the trained non-language information generation model. This makes it possible to appropriately generate non-language information using both the speech features with time information and the text features with time information.

[0223] When generating non-language information using both speech features and text features, it is desirable that the time axis of the time information corresponds to and matches the time axis of the speech features and the text features, as shown in Figure 3 above.

[0224] In addition, when non-verbal information is generated using both speech features and text features and expressed by the expression unit, it is also possible to present it together with the input speech information and text information, synthetic speech obtained from the text information, and recognized text obtained from the speech information.

[0225] As described above, the non-language information generation device according to the third embodiment generates non-language information with time information based on speech features with time information and text features with time information, and a trained non-language information generation model for generating non-language information with time information. This makes it possible to reduce the cost of associating speech information and text information with non-language information.

[0226] Furthermore, according to the non-language information generation model learning device of the third embodiment, a non-language information generation model for generating non-language information with time information from the speech features with time information and the text features with time information can be obtained based on the speech features with time information for learning and the text features with time information for learning.

[0227] In the above third embodiment, the input information is voice information, but the present invention is not limited to this. For example, the input information may be text information.

[0228] FIG. 14A shows a configuration example of the learning information acquisition unit 431 and learning feature extraction unit 432 in the non-language information generation model learning device when the input information is text information.

[0229] As shown in FIG. 14A, when learning text information is input, learning text analysis unit 437 performs a predetermined text analysis process on the learning text information and obtains a text analysis result corresponding to the learning text information.

[0230] Then, the training speech synthesis unit 438 performs a predetermined speech synthesis process on the text analysis result to obtain training speech information.

[0231] The training feature extraction unit 432 then extracts speech features with time information from the training speech information, and also extracts text features with time information from the training text information.

[0232] A learning unit (not shown) learns a non-language information generation model based on the learning speech features with time information, the learning text features with time information, and the learning non-language information. This makes it possible to obtain a non-language information generation model for generating non-language information with time information from the learning speech features with time information and the learning text features with time information.

[0233] FIG. 14B shows an example of the configuration of information acquisition section 461 and feature extraction section 462 in the non-language information generation device when the input information is text information.

[0234] As shown in FIG. 14B, when text information is input, the text analysis unit 465 performs a predetermined text analysis process on the text information and acquires a text analysis result corresponding to the text information.

[0235] Then, the voice synthesis unit 466 performs a predetermined voice synthesis process on the text analysis result to obtain voice information.

[0236] The feature extraction unit 462 then extracts voice features with time information from the voice information, and also extracts text features with time information from the text information.

[0237] Then, a non-language information generation unit (not shown) acquires generation parameters representing non-language information based on the speech features with time information and the text features with time information, and a non-language information generation model that has been trained in advance.

[0238] The present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the spirit and scope of the present invention.

[0239] For example, the configurations corresponding to the combination of the information acquisition unit (or the learning information acquisition unit) and the feature extraction unit (or the learning feature extraction unit) in each of the above embodiments are a total of four patterns shown in Fig. 15. Also, the possible variations in the combination of configurations during learning and during non-language information generation are the patterns shown in Fig. 16. Note that the accuracy of non-language information generation will be higher if the features during learning and generation match.

[0240] The present invention can also be realized by installing a program on a well-known computer via a medium or a communication line.

[0241] In addition, the above-mentioned device has an internal computer system, but if the "computer system" uses a WWW (World Wide Web) system, it also includes the homepage providing environment (or display environment).

[0242] Furthermore, in the present specification, an embodiment in which a program is pre-installed has been described, but the program can also be provided by being stored on a computer-readable recording medium.

[0243] Other embodiments will be described below.

[0244] [Outline of other embodiments]

[0245] The learning data used in the non-verbal information generation model learning device is created, for example, in a scene such as that shown in FIG. 2 above, by acquiring speech information (X) of a speaker who is currently speaking and, at the same time, acquiring non-verbal information (Y) of a conversation partner with whom the speaker is currently speaking using a measurement device.

[0246] By training a non-verbal information generation model based on the training data created in this way, it becomes possible to realize agents, robots, etc. that react non-verbally (e.g., nodding) at appropriate times to input information such as voice information or text information.

[0247] Furthermore, in a scene like that shown in Figure 2 above, learning data can be created by acquiring the speech information (X) of the speaker who is currently speaking while also acquiring the non-verbal information (Y) of the other participants using a measuring device.

[0248] By training a non-verbal information generation model in this way based on the non-verbal information of each of the multiple participants, it is possible to have multiple robots or agents react appropriately and at different times to the input information, which is voice information or text information.

[0249] In this case, a combination of speech information acquired from a speaker and non-verbal information representing information on the behavior of the other party of the speaker (for example, a listener or participant in the conversation) is used as training data to train a trained non-verbal information generation model. Here, the other party of the speaker may include not only the listener or participant in the conversation, but also an observer of the conversation, and these "entities that show some kind of reaction in response to the speaker's voice (and its contents)" are also referred to as "listeners of the speech".

[0250] In addition, when text information is the target, a combination of text information corresponding to the voice information obtained from a speaker and non-verbal information representing information regarding the behavior of the speaker's other party (e.g., a listener or participant in the conversation) is used as training data to train a trained non-verbal information generation model.

[0251] The following fourth to sixth embodiments are first to third modified examples.

[0252] [Fourth embodiment]

[0253] In the fourth embodiment, the target is voice information, and the non-verbal information generation model learning device learns a non-verbal information generation model for generating non-verbal information with time information from the voice feature with time information, based on the learning voice feature with time information extracted from the learning voice information output from the speaker ("speaker" in FIG. 2) and the learning non-verbal information representing information on the behavior of the speaker's partner ("other participants" or "speaker's conversation partner" in FIG. 2). As a result, a trained non-verbal information generation model expressing the behavior of the speaker's partner is obtained according to the voice information acquired from the speaker. As shown in FIG. 2, the "speaker" refers to a person who makes a sound such as an utterance. Also, the "speaker's partner" refers to a person who listens to the utterance, etc., made by the speaker, and corresponds to the "other participants" and the "speaker's conversation partner" shown in FIG. 2. Also, the behavior of the speaker's partner is, for example, the reaction of the speaker's partner to the sound made by the speaker.

[0254] In addition, when speech information for which non-verbal information is to be generated is input, the non-verbal information generation device generates non-verbal information with time information based on speech features with time information extracted from the speech information acquired from the speaker and the trained non-verbal information generation model.

[0255] Other configurations and operations of the non-language information generation model learning device and the non-language information generation device of the fourth embodiment are similar to those of the first embodiment, and therefore description thereof will be omitted.

[0256] [Fifth embodiment]

[0257] In the fifth embodiment, the target is text information, and the non-verbal information generation model training device trains a non-verbal information generation model for generating non-verbal information with time information from text features with time information, based on training text features with time information extracted from text corresponding to training speech information acquired from a speaker, and training non-verbal information representing information on the behavior of the other party of the speaker. This allows for the acquisition of a trained non-verbal information generation model that expresses the behavior of the other party of the speaker according to text information corresponding to the speech information acquired from the speaker.

[0258] In addition, when text information for which non-verbal information is to be generated is input, the non-verbal information generation device generates non-verbal information with time information based on text features with time information extracted from text information corresponding to the speech information obtained from the speaker and the trained non-verbal information generation model.

[0259] Other configurations and operations of the non-language information generation model learning device and the non-language information generation device of the fifth embodiment are similar to those of the second embodiment, and therefore description thereof will be omitted.

[0260] [Sixth embodiment]

[0261] In the sixth embodiment, both speech information and text information are targeted, and the non-verbal information generation model training device trains a non-verbal information generation model for generating non-verbal information with time information from speech features with time information and text features with time information, based on training speech features with time information and training text features with time information, and training non-verbal information representing information on the behavior of the other party of the speaker. This provides a trained non-verbal information generation model that expresses the behavior of the other party of the speaker according to the speech information and text information.

[0262] In addition, when both speech information and text information for which non-language information is to be generated are input, the non-language information generation device generates non-language information with time information based on speech features with time information and text features with time information, and the trained non-language information generation model.

[0263] Other configurations and operations of the non-language information generation model learning device and the non-language information generation device of the sixth embodiment are similar to those of the third embodiment, and therefore description thereof will be omitted.

[0264] [Seventh embodiment]

[0265] <Configuration of a learning device for non-linguistic information generation model>

[0266] Next, a seventh embodiment of the present invention will be described. Note that parts having the same configuration as the first embodiment are given the same reference numerals and the description thereof will be omitted.

[0267] In the seventh embodiment, a gesture corresponding to an input utterance is generated based on a trained machine learning model. In this case, not only the presence or absence of an action but also information on the size, number of times, and ratio of pause length is generated as a gesture. Also, the seventh embodiment differs from the first embodiment in that a non-verbal information generation model that generates non-verbal information is trained using variables that affect the generation of the gesture as "additional information" in addition to the speech or text used as input.

[0268] The text information used in the seventh embodiment is text information that represents the contents of a speech made by a speaker to the outside through voice.

[0269] Fig. 17 is a block diagram showing an example of the configuration of a non-verbal information generation model learning device 710 according to the seventh embodiment. As shown in Fig. 17, the non-verbal information generation model learning device 710 according to the seventh embodiment is configured by a computer including a CPU, a RAM, and a ROM storing a program for executing a learning processing routine described later. The non-verbal information generation model learning device 710 functionally includes a learning input unit 720 and a learning calculation unit 730.

[0270] The learning input unit 720 receives learning data including a combination of learning text information, learning non-language information, and learning additional information.

[0271] The non-verbal information received includes not only the presence or absence of a behavior, but also the size, frequency, and proportion of pause length.

[0272] Specifically, the non-verbal information includes any of the behaviors included in the behavior lists shown in Tables 2 to 4 below.

[0273] [Table 2]

[0274] [Table 3]

[0275] [Table 4]

[0276] Here, the text information for learning was prepared by manually transcribing each individual's speech based on the speech voice data, and the speech sections were separated by silent sections of 200 msec or more.

[0277] The non-verbal information for learning is added in consideration of the movements made during speech corresponding to the speech voice data, such as nodding, facial direction, hand gestures, gaze, facial expression, body posture, body joints, position, movement, pupil diameter, and blinking. The hand gestures included in the non-verbal information for learning are, for example, annotations that take into account the hand movements during speech as well as the content of the speech. The hand gestures included in the non-verbal information for learning may also be obtained by automatic recognition. Many methods have been proposed for automatic recognition of hand gestures, such as those using image processing (e.g., Reference 7), and any method may be used.

[0278] [Reference 7] Siddharth S. Rautaray, Anupam Agrawal, "Vision based hand gesture recognition for human computer interaction: a survey", Artificial Intelligence Review, January 2015, Volume 43, Issue 1, pp. 1-54.

[0279] Here, the hand gesture types may be the same as those described in Reference 8. For example, detailed hand gesture types are (A) Iconic, (B) Metaphoric, (C) Beat, (D) Deictic, (E) Feedback, (F) Compellation, (G) Hesitate, and (H) Others.

[0280] [References 8] D. McNeill, "Hand and Mind: What Gestures Reveal About Thought", Chicago: University of Chicago Press, 1992

[0281] (A) Iconic is a gesture used to describe a scene or express an action. (B) Metaphoric is a pictorial or graphic gesture, like Iconic, but the contents indicated are abstract matters or concepts (such as the flow of time). (C) Beat is a gesture used to adjust the tone of speech or emphasize a statement, such as shaking the hand or waving the hand in time with the speech. (D) Deictic is a gesture that directly indicates a direction, place, or thing, such as pointing. (E) Feedback is a gesture made in sync with, agreement with, or response to another person's speech, a gesture that accompanies a speech in response to another person's previous speech or gesture, or a gesture made by imitating the other person's gesture. (F) Compellation is a gesture made to call out to the other person. (G) Hesitate is a gesture made when one is hesitant to speak. (H) Others is a gesture that is difficult to judge, but may have some meaning.

[0282] In addition, in the annotation of the hand gesture, in addition to the above-mentioned hand gesture type, annotations indicating the four states of Prep, Hold, Stroke, and Return may also be added.

[0283] "Prep" indicates the state of raising the hand from the home position to perform a gesture, and "Hold" indicates the state of keeping the hand raised in the air (waiting time until the gesture starts). "Stroke" indicates the state of performing a gesture, and the above types of annotations (A) to (H) are given as detailed information about this state. "Return" indicates the state of returning the hand to the home position.

[0284] The nodding gestures included in the non-verbal information for training were annotated by, for example, the nodding section from when the person bows their head to when they speak until they return it to their original position. The action of putting the head forward and returning it to their original position, or pulling it back and returning it to their original position, was also annotated as a nodding action. The action of tilting the head or shaking it from side to side was not annotated as a nodding action.

[0285] If a participant nodded two or more times in a row without a pause, the consecutive segments were grouped together and the number of nods was assigned. The number of nods was classified as "1, 2, 3, 4, 5 or more."

[0286] The additional information for learning that is accepted is additional information for each predetermined processing unit (for example, for each morpheme). At least one of personal attributes, environmental variables, physical characteristics, posture of the action target, content of the dialogue, human relationships, and emotions is accepted as the additional information. Specifically, the personal attributes include gender, age, personality, nationality, and cultural area, and the environmental variables include the number of people during the dialogue (one-to-one, one-to-many, many-to-many), temperature, indoors / outdoors, land / air / underwater, light / dark, and the like. The physical characteristics include three-headedness, clothing: pockets, wearing a skirt, wearing a hat, and other things that affect the action, and the posture of the action target includes standing, sitting, and holding something in the hand. The contents of the dialogue include discussion, chat, explanation, and the like, and the human relationships include the human relationships between the person generating the gesture and the dialogue partner, which one is in a dominant position, whether they have favor, and the like. The emotions represent internal states, including joy, anger, sadness, and other mental states such as tension / relaxation. In addition, if the additional information does not change for each predetermined processing unit (for example, gender), it is not necessary to accept it for each predetermined processing unit. In this case, when the additional information is accepted, it is sufficient for the device to expand it into additional information for each predetermined processing unit.

[0287] The learning calculation unit 730 generates a non-language information generation model for generating non-language information with time information, based on the learning data received by the learning input unit 720. As shown in Fig. 17, the learning calculation unit 730 includes a learning information acquisition unit 231, a learning additional information acquisition unit 731, a learning feature extraction unit 732, a non-language information acquisition unit 33, a generation parameter extraction unit 34, a learning unit 735, and a trained model storage unit 736.

[0288] The learning information acquisition unit 231 acquires learning voice information corresponding to learning text information, and acquires time information indicating the time from the start time to the end time when the voice information is uttered.

[0289] The learning additional information acquisition unit 731 acquires additional information for learning.

[0290] The learning feature extraction unit 732 extracts learning text features with time information, which represent the features of the learning text information, from the learning text information and time information acquired by the learning information acquisition unit 231. Specifically, the learning feature extraction unit 732 assigns time information to the text information for each predetermined analysis unit, and extracts text features with time information. An example of text features with time information is shown in Fig. 18. As shown in Fig. 18, the start time and end time of each morpheme of the text information are acquired.

[0291] Furthermore, the learning feature extraction unit 732 generates additional information for learning with time information from the additional information for learning acquired by the additional information for learning acquisition unit 731 and the time information acquired by the information for learning acquisition unit 231. Specifically, the learning feature extraction unit 732 adds time information to the additional information for each predetermined analysis unit to generate additional information with the time information.

[0292] An example of additional information with time information is shown in Fig. 18. As shown in Fig. 18, the start time and end time of each morpheme are acquired in the additional information of the morpheme. In addition, when there are multiple types of additional information (for example, the number of conversation participants, emotions, and temperature), the additional information is represented as a vector array storing multiple types of additional information.

[0293] The learning feature extraction unit 732 extracts the vector array of the additional information

[0294]

number

[0295] In addition,

[0296]

number

[0297] are the start and end times of the speech corresponding to the morpheme unit.

[0298] The learning unit 735 learns a non-language information generation model for generating non-language information with time information from the text features and additional information with time information, based on the learning text features with time information and additional information with time information extracted by the learning feature extraction unit 732, and the learning discretized non-language information with time information extracted by the generation parameter extraction unit 34.

[0299] Specifically, the learning unit 735 extracts the learning text feature with time information extracted by the learning feature extraction unit 732.

[0300]

number

[0301]

number

[0302] , and additional information for learning with time information

[0303]

number

[0304] With the input, non-linguistic information

[0305]

number

[0306] A non-verbal information generation model that outputs the following is constructed. When constructing the non-verbal information generation model, any machine learning method may be used, but in this embodiment, SVM is used. For example, for each of the above types (A) to (H), an SVM model is created to estimate which behavior of the type the gesture belongs to. That is, for each of the above types (A) to (H), an SVM model is created to estimate which behavior of the type the gesture belongs to among the multiple behaviors listed as the content belonging to the type in the behavior list table.

[0307] In addition, the non-verbal information generation model can estimate non-verbal information at any time resolution and using parameters at any time.

[0308]

number

[0309] An example of the feature used for estimating is shown below. The language feature obtained at the time T1 to T2 to be estimated is

[0310]

number

[0311] , and additional information

[0312]

number

[0313] and the output gesture

[0314]

number

[0315] are paired, and learning is performed using a plurality of pieces of learning data that are data of these pairs. The learned non-verbal information generation model is

[0316]

Number

[0317] set as. Note that as a method for setting T1 and T2, for example, when estimating non-verbal information in units of morphemes, the start time and end time of each morpheme are set as T1 and T2. In this case, the window widths from T2 to T1 are different for each morpheme.

[0318] Also,

[0319]

Number

[0320] may be the average value of non-verbal information obtained within T1 to T2, the combination of non-verbal information that has appeared, or a pattern considering the order of appearance. For example, the hand gesture ID

[0321]

Number

[0322] is within T1 to T3 (T1 < T3 < T2)

[0323]

Number

[0324] and within T3 to T2

[0325]

Number

[0326] and when

[0327]

number

[0328] As a result, the IDs with the highest occurrence times and combination information

[0329]

number

[0330] , as ngram patterns

[0331]

number

[0332] will be adopted.

[0333] When using n-gram patterns, the time information of non-linguistic information in the n-gram patterns (in the above example,

[0334]

number

[0335] The time information of each non-language information (e.g., the start time of each of the non-language information) is assigned by a predetermined method set in advance for each non-language information. However, the time information of the non-language information in this n-gram pattern may also be estimated. In this case, the time information is estimated based on the learning data using the feature amount used to estimate the n-gram pattern and the estimated n-gram.

[0336] The trained model storage unit 736 stores a trained non-language information generation model trained by the training unit 735. The trained non-language information generation model generates non-language information with time information from text features with time information.

[0337] <Configuration of non-verbal information generation device>

[0338] Fig. 19 is a block diagram showing an example of the configuration of a non-language information generation device 740 according to the seventh embodiment. As shown in Fig. 19, the non-language information generation device 740 according to this embodiment is configured by a computer equipped with a CPU (Central Processing Unit), a RAM (Random Access Memory), and a ROM (Read Only Memory) that stores a program for executing a non-language information generation processing routine described later. The non-language information generation device 740 functionally comprises an input section 750, a calculation section 760, and a development section 70.

[0339] The input unit 750 receives text information and additional information. The received additional information is additional information for each predetermined processing unit (for example, for each morpheme).

[0340] The calculation unit 760 includes an information acquisition unit 261 , an additional information acquisition unit 761 , a feature extraction unit 762 , a trained model storage unit 763 , and a non-language information generation unit 764 .

[0341] The information acquiring unit 261 acquires text information accepted by the input unit 750. The information acquiring unit 261 also acquires voice information corresponding to the text information, and acquires time information indicating the time from the start time to the end time when the voice information is uttered. The additional information acquisition unit 761 acquires the additional information accepted by the input unit 750 .

[0342] Similar to the learning feature extraction unit 732, the feature extraction unit 762 extracts text features with time information, which represent the features of the text information, from the text information and time information acquired by the information acquisition unit 261. Also, similar to the learning feature extraction unit 732, the feature extraction unit 762 generates additional information with time information from the additional information acquired by the additional information acquisition unit 761 and the time information acquired by the information acquisition unit 261.

[0343] The trained model storage unit 763 stores a trained non-language information generation model that is the same as the trained non-language information generation model stored in the trained model storage unit 736.

[0344] The non-language information generation unit 764 generates non-language information with time information corresponding to the text features with time information and additional information extracted by the feature extraction unit 762, based on the text features with time information and additional information extracted by the feature extraction unit 762, and the trained non-language information generation model stored in the trained model memory unit 763.

[0345] For example, the non-language information generation unit 764 uses the trained non-language information generation model stored in the trained model storage unit 763.

[0346]

number

[0347] Using the above, we can extract any feature as a text feature with time information.

[0348]

number

[0349] , and additional information with time information

[0350]

number

[0351] The input is the gesture as a generation parameter corresponding to non-verbal information with time information.

[0352]

number

[0353] Get the.

[0354] Then, the non-language information generating section 764 controls the expressing section 70 so that the generated generation parameters with time information are output from the expressing section 70 based on the time information added to the generation parameters.

[0355] Specifically, the non-verbal information generating unit 764 generates a gesture

[0356]

number

[0357] is reflected as the motion of an arbitrary object (e.g., an animation character, a robot, etc.).

[0358] The expression section 70 expresses, under the control of the non-language information generation section 764, speech information corresponding to the text information received by the input section 750 and the non-language information generated by the non-language information generation section 764.

[0359] <Function of the Non-verbal Information Generation Model Learning Device 710>

[0360] Next, the operation of the non-verbal information generation model learning device 710 according to the seventh embodiment will be described. First, when learning data representing a combination of a plurality of pieces of text information for learning, a plurality of pieces of additional information for learning, and a plurality of pieces of non-verbal information for learning is input to the learning input unit 720 of the non-verbal information generation model learning device 710, the non-verbal information generation model learning device 710 executes a learning process routine shown in FIG.

[0361] First, in step S300, the learning information acquisition unit 231 acquires learning text information from among the multiple learning data (specifically, pairs of text information, additional information, and non-language information) accepted by the learning input unit 720.

[0362] In step S102, the non-verbal information acquisition unit 33 acquires non-verbal information for learning from the multiple learning data accepted by the learning input unit 720 and time information representing the time from the start time to the end time when the behavior represented by the non-verbal information for learning is performed.

[0363] In step S303, the training text analysis unit 237 performs a predetermined text analysis on the training text information acquired in step S300, and acquires a text analysis result. The training voice synthesis unit 238 synthesizes training voice information corresponding to the training text information, based on the text analysis result acquired by the training text analysis unit 237. The training voice synthesis unit 238 then acquires time information indicating the time from the start time to the end time when the training voice information is uttered.

[0364] In step S700, the learning-use additional information acquisition unit 731 acquires additional information for learning from among the multiple pieces of learning data accepted by the learning-use input unit 720.

[0365] In step S702, the learning feature extraction unit 732 extracts learning text features with time information from the learning text information and time information acquired in step S303. In addition, the learning feature extraction unit 732 generates learning additional information with time information from the learning additional information acquired in step S700 and the time information acquired in step S303.

[0366] In step S106, the generation parameter extraction unit 34 extracts discretized non-language information with time information from the learning non-language information and time information acquired in step S102.

[0367] In step S708, the learning unit 735 learns a non-language information generation model for generating generation parameters with time information from the text features and additional information with time information, based on the learning text features with time information and learning additional information with time information extracted in step S702 above, and the learning generation parameters with time information acquired in step S106.

[0368] In step S110, the learning unit 735 stores the trained non-language information generation model obtained in step S708 above in the trained model storage unit 736, and ends the learning process routine.

[0369] <Function of the non-verbal information generating device 740>

[0370] Next, the operation of the non-language information generation device 740 according to the seventh embodiment will be described. First, when a trained non-language information generation model stored in the trained model storage unit 736 of the non-language information generation model learning device 710 is input to the non-language information generation device 740, the trained non-language information generation model is stored in the trained model storage unit 763 of the non-language information generation device 740. Then, when text information and additional information for which non-language information is to be generated are input to the input unit 750, the non-language information generation device 740 executes a non-language information generation processing routine shown in FIG.

[0371] In step S 400 , information acquisition unit 261 acquires text information accepted by input unit 750 .

[0372] In step S401, the text analysis unit 265 performs a predetermined text analysis on the text information acquired in step S400, and acquires a text analysis result. In addition, the voice synthesis unit 266 synthesizes voice information corresponding to the text information based on the text analysis result acquired by the text analysis unit 265. Then, the voice synthesis unit 266 acquires time information indicating the time from the start time to the end time when the voice information is uttered.

[0373] In step S 750 , the additional information acquisition unit 761 acquires the additional information accepted by the input unit 750 .

[0374] In step S752, the feature extraction unit 762 extracts text features with time information from the text information and time information acquired in step S401, and generates additional information with time information from the additional information acquired in step S750 and the time information acquired in step S401.

[0375] In step S754, the non-language information generation unit 764 reads out the trained non-language information generation model stored in the trained model storage unit 763.

[0376] In step S756, the non-language information generation unit 764 generates generation parameters with time information corresponding to the text features and additional information with time information extracted in step S752, based on the text features and additional information with time information extracted in step S752 and the trained non-language information generation model read out in step S754.

[0377] In step S208, the non-verbal information generation unit 764 controls the expression unit 70 so that the non-verbal information with time information generated in step S756 above is output from the expression unit 70 based on the time information assigned to the non-verbal information, and then terminates the non-verbal information generation processing routine.

[0378] Other configurations and operations of the non-language information generation device and the non-language information generation model learning device according to the seventh embodiment are similar to those of the first embodiment, and therefore description thereof will be omitted.

[0379] As described above, the non-language information generating device 740 according to the seventh embodiment acquires additional information, and generates non-language information with time information corresponding to the text feature with time information based on the text feature with time information, the additional information, and a trained model for generating non-language information with time information including the number of times or the magnitude of an action. This makes it possible to automatically associate text information with non-language information including the number of times or the magnitude of an action, thereby reducing the cost.

[0380] In addition, by setting the type of non-verbal information in detail, including the magnitude and frequency of actions, more detailed nuances can be expressed, making it easier to express intentions through non-verbal information. In addition, by combining time information with additional information, changes in actions due to differences in additional information can be expressed in more detail, making it easier to express emotions, for example.

[0381] Furthermore, according to the non-language information generation model learning device 710 of the seventh embodiment, a non-language information generation model for generating non-language information with time information from the text feature with time information and the additional information, based on learning text feature and additional information with time information, and learning non-language information with time information including the number of times or the magnitude of an action, is learned. This makes it possible to obtain a non-language information generation model for generating non-language information from the text feature with time information, while reducing the cost of associating text information with non-language information including the number of times or the magnitude of an action.

[0382] In the above embodiment, the case where the additional information is input has been described as an example, but the present invention is not limited to this, and the additional information may be estimated. In this case, as shown in Fig. 22, the learning calculation unit 730A of the non-verbal information generation model learning device 710A may include a learning additional information estimation unit 731A that estimates the additional information from text information included in the learning data received by the learning input unit 720 and outputs the additional information to the learning additional information acquisition unit 731. For example, the contents of the dialogue and emotions can be estimated from the text information.

[0383] Moreover, as shown in FIG. 23, a calculation section 760A of a non-language information generation device 740A may include an additional information estimation section 761A that estimates additional information from text information received by an input section 750 and outputs the additional information to an additional information acquisition section 761.

[0384] When audio information or video information is input instead of text information, the additional information may be estimated from the audio information or video information. For example, age, sex, environmental variables, physical characteristics, etc. may be estimated from video information, and the number of people in the conversation, emotions, nationality, indoor / outdoor, etc. may be estimated from audio information.

[0385] As described above, when estimating additional information, time information can be automatically added to the additional information. Also, when estimating additional information, an estimated value of non-language information can be generated for each unit of estimation.

[0386] In addition, in the non-verbal information generating device, other additional information may be changed according to specific additional information obtained by estimation. Specifically, other additional information may be changed so as to switch to designated information defined in advance for specific additional information. For example, when it is estimated from the voice information that the indoor / outdoors has been switched, other additional information may be changed so as to change the designated content of clothing, which is a physical characteristic. Also, the speech speed of the voice synthesis or the display speed of text may be changed according to emotions. In addition, when a full stomach state is detected, the additional information may be changed so as to set the body shape to be plump, and when an angry emotion is detected, the additional information may be changed so as to change the content of the dialogue to "discussion."

[0387] Furthermore, when emotion labels of a given processing unit are arranged in the order of "normal" and "anger" in chronological order as a result of estimating additional information, and voice is output simultaneously with a gesture, the additional information for each given processing unit is passed as a parameter (speech rate) to a voice synthesis unit (not shown). As a result, for a given processing unit in which the emotion label "anger" is estimated, the speech rate is increased by a given number or multiplied by a given number, and the time information of the text information is shortened (FIG. 24). In FIG. 24, the horizontal axis with an arrow indicates the time axis.

[0388] In addition, the time information of the voice feature with time information during the period in which "anger" was estimated may be changed.

[0389] Further, the non-verbal information has been described as including the magnitude and number of times of the action, but is not limited to this, and may further include the proportion of the pause length.

[0390] [Eighth embodiment]

[0391] Next, an eighth embodiment of the present invention will be described. Note that parts having the same configuration as the first embodiment are given the same reference numerals and the description thereof will be omitted.

[0392] <Outline of the Eighth Embodiment>

[0393] When the input is text, detailed time information for the text may not be available. For example, this may occur when (1) the speech synthesis (or speech recognition) does not have the functionality to output time information of the required granularity corresponding to each piece of text, or (2) when only text is output during gesture generation (no need for speech synthesis) and there are no resources to perform speech synthesis or the time required for speech synthesis processing cannot be secured (when the input is speech, there is no need to output the original speech information, or there are insufficient resources, etc.).

[0394] The above case (1) occurs when using DNN (Deep Neural Network)-based synthetic voice, while the above case (2) occurs due to constraints in actual services.

[0395] Therefore, in this embodiment, when time information of the required granularity (e.g., characters or morphemes) as in (1) above cannot be obtained, time information of the required granularity is obtained using the speech duration of each phrase (the closest one, though coarser than the required granularity) that can be obtained when generating speech corresponding to the text using a speech synthesizer. Specifically, the units from which time information can be obtained are divided by the number of units (the number of moras in Japanese, the number of accents in English, etc.) that correspond to the pronunciation characteristics of the target language, and the units are used as time information of the required unit.

[0396] <Configuration of a learning device for non-linguistic information generation model>

[0397] As shown in FIG. 25, the learning calculation unit 830 of the non-language information generation model learning device 810 of the eighth embodiment includes a learning information acquisition unit 231, a learning additional information acquisition unit 731, a learning feature extraction unit 732, a non-language information acquisition unit 33, a generation parameter extraction unit 34, a learning data creation unit 833, a learning unit 735, and a learned model memory unit 736.

[0398] The learning data creation unit 833 obtains detailed time information for the learning text information acquired by the learning information acquisition unit 231, and generates text information with detailed time information.

[0399] Here, the detailed time information is given based on the result of dividing the time range when the learning text is output by the number of divisions when the text is divided into a predetermined unit.

[0400] For example, when the time information obtained for the text information is time information for each utterance and time information for a predetermined unit (for each mora) is to be obtained, the time information is obtained as follows.

[0401] First, the speech length of the text is normalized to 1, and the speech length of the unit from which time information can be obtained is divided by a predetermined number of units (in this case, the number of moras) (see FIG. 26A). Then, the time information obtained by dividing is used as the time information of the predetermined unit.

[0402] In addition, the learning data creation unit 833 uses the detailed time information obtained above to generate learning text features, learning additional information, and learning non-language information with detailed time information for the learning text features and additional information with time information extracted by the learning feature extraction unit 732, and the learning discretized non-language information with time information extracted by the generation parameter extraction unit 34.

[0403] The learning data creation unit 833 outputs a combination of learning text features with detailed time information, learning additional information with detailed time information, and learning non-language information with detailed time information to the learning unit 735 as learning data.

[0404] <Configuration of non-verbal information generation device>

[0405] As shown in FIG. 26B, the calculation unit 860 of the non-language information generation device 840 of the eighth embodiment includes an information acquisition unit 261, an additional information acquisition unit 761, a feature extraction unit 762, a trained model memory unit 763, a non-language information generation unit 764, and a control unit 870.

[0406] The expression unit 70 outputs text information as in the first and second embodiments, and expresses the behavior indicated by the generated non-verbal information with time information in accordance with the time information. Also, a voice corresponding to the text information may be output.

[0407] The feature extraction unit 762, like the learning feature extraction unit 732, extracts text features with time information, which represent the features of the text information, from the text information and time information acquired by the information acquisition unit 261. In addition, like the learning feature extraction unit 732, the feature extraction unit 762 generates additional information with time information from the additional information acquired by the additional information acquisition unit 761 and the time information acquired by the information acquisition unit 261.

[0408] Furthermore, the feature extraction unit 762 obtains detailed time information for the text information in the same manner as the learning data creation unit 833, and obtains detailed time information for the text information.

[0409] Furthermore, the feature extraction unit 762 generates text feature values ​​and additional information with detailed time information using the detailed time information obtained above for the text feature values ​​and additional information with time information.

[0410] The non-language information generation unit 764 generates non-language information with detailed time information corresponding to the text features with detailed time information and additional information extracted by the feature extraction unit 762, based on the text features with detailed time information and additional information with detailed time information generated by the feature extraction unit 762, and the trained non-language information generation model stored in the trained model memory unit 763.

[0411] Then, the control unit 870 controls the expression unit 70 so that generation parameters corresponding to the generated non-verbal information with detailed time information are output from the expression unit 70, and also controls the expression unit 70 to output text information in accordance with the time information.

[0412] In accordance with the control of the control unit 870, the expression unit 70 outputs text information or audio information corresponding to the text information received by the input unit 750 in accordance with detailed time information, and expresses the non-verbal information generated by the non-verbal information generation unit 764 in accordance with the detailed time information.

[0413] <Function of the Non-verbal Information Generation Model Learning Device 810>

[0414] Next, the operation of the non-language information generation model learning device 810 according to the eighth embodiment will be described. First, when learning data representing a combination of a plurality of pieces of text information for learning, a plurality of pieces of additional information for learning, and a plurality of pieces of non-language information for learning is input to the learning input unit 720 of the non-language information generation model learning device 810, the non-language information generation model learning device 810 executes a learning process routine shown in FIG.

[0415] First, in step S300, the learning information acquisition unit 231 acquires learning text information from among the multiple learning data (specifically, a combination of text information, non-language information, and additional information) accepted by the learning input unit 720.

[0416] In step S102, the non-verbal information acquisition unit 33 acquires non-verbal information for learning from the multiple learning data accepted by the learning input unit 720 and time information representing the time from the start time to the end time when the behavior represented by the non-verbal information for learning is performed.

[0417] In step S303, the training text analysis unit 237 performs a predetermined text analysis on the training text information acquired in step S300, and acquires a text analysis result. The training voice synthesis unit 238 synthesizes training voice information corresponding to the training text information, based on the text analysis result acquired by the training text analysis unit 237. The training voice synthesis unit 238 then acquires time information indicating the time from the start time to the end time when the training voice information is uttered.

[0418] In step S700, the learning-use additional information acquisition unit 731 acquires additional information for learning from among the multiple pieces of learning data accepted by the learning-use input unit 720.

[0419] In step S702, the learning feature extraction unit 732 extracts learning text features with time information from the learning text information and time information acquired in step S303. In addition, the learning feature extraction unit 732 generates learning additional information with time information from the learning additional information acquired in step S700 and the time information acquired in step S303.

[0420] In step S106, the generation parameter extraction unit 34 extracts discretized non-language information with time information from the learning non-language information and time information acquired in step S102.

[0421] In step S800, the learning data creation unit 833 obtains detailed time information for the learning text information from the learning text information and time information acquired in step S303 above.

[0422] In addition, the learning data creation unit 833 uses the detailed time information obtained above to generate learning text features, learning additional information, and learning non-language information with detailed time information for the text features and additional information with time information extracted by the learning feature extraction unit 732, and the discretized learning non-language information with time information extracted by the generation parameter extraction unit 34.

[0423] The learning data creation unit 833 outputs a combination of learning text features with detailed time information, learning additional information with detailed time information, and learning non-language information with detailed time information to the learning unit 735 as learning data.

[0424] In step S708, the learning unit 735 learns a non-linguistic information generation model for generating generation parameters with detailed time information from the text features with detailed time information and the additional information based on the learning text features with detailed time information and the learning additional information with detailed time information generated in step S800, and the learning generation parameters with detailed time information acquired in step S800.

[0425] In step S110, the learning unit 735 stores the trained non-language information generation model obtained in step S708 above in the trained model storage unit 736, and ends the learning process routine.

[0426] <Function of the non-verbal information generating device>

[0427] Next, the operation of the non-language information generation device 840 according to the eighth embodiment will be described. First, when the trained non-language information generation model stored in the trained model storage unit 736 of the non-language information generation model learning device 810 is input to the non-language information generation device 840, the trained non-language information generation model is stored in the trained model storage unit 763 of the non-language information generation device 840. Then, when text information and additional information for which non-language information is to be generated are input to the input unit 750, the non-language information generation device 840 executes a non-language information generation processing routine shown in FIG.

[0428] In step S 400 , information acquisition unit 261 acquires text information accepted by input unit 750 .

[0429] In step S401, the text analysis unit 265 performs a predetermined text analysis on the text information acquired in step S400, and acquires a text analysis result. In addition, the voice synthesis unit 266 synthesizes voice information corresponding to the text information based on the text analysis result acquired by the text analysis unit 265. Then, the voice synthesis unit 266 acquires time information indicating the time from the start time to the end time when the voice information is uttered.

[0430] In step S 750 , the additional information acquisition unit 761 acquires the additional information accepted by the input unit 750 .

[0431] In step S752, the feature extraction unit 762 extracts text features with time information from the text information and time information acquired in step S401, and generates additional information with time information from the additional information acquired in step S750 and the time information acquired in step S401.

[0432] In step S850, the feature extraction unit 762 obtains detailed time information for the text information from the text information and time information obtained in step S401. In addition, the feature extraction unit 762 generates text features with detailed time information for the text features with time information by using the detailed time information obtained above.

[0433] Furthermore, the feature extraction unit 762 generates additional information with detailed time information by using the detailed time information obtained above with respect to the additional information with time information.

[0434] In step S754, the non-language information generation unit 764 reads out the trained non-language information generation model stored in the trained model storage unit 763.

[0435] In step S756, the non-language information generation unit 764 generates generation parameters with time information corresponding to the text features with detailed time information and additional information generated in step S850, based on the text features and additional information with detailed time information generated in step S850 and the trained non-language information generation model read out in step S754.

[0436] In step S852, the control unit 870 controls the expression unit 70 so that the text acquired in step S400 and the non-verbal information with time information generated in step S756 are output from the expression unit 70 in accordance with the time information, and then terminates the non-verbal information generation processing routine.

[0437] Other configurations and operations of the non-language information generation device and the non-language information generation model learning device according to the eighth embodiment are similar to those of the first embodiment, and therefore description thereof will be omitted.

[0438] As described above, according to the non-verbal information generation device of the eighth embodiment, by dividing the time information of the text information into time information for each predetermined unit and adding it, even if time information of the required granularity cannot be obtained for the text information, the text information can be output together with the manifestation of the behavior indicated by the non-verbal information according to the time information for each predetermined unit.

[0439] In the above embodiment, the time information is divided using the number of moras as a unit number suited to the pronunciation characteristics of the target language, but the present invention is not limited to this, and in the case of English, the time information may be divided using the number of accents. Furthermore, the time information may be divided and assigned using the number of parts of speech, the number of categories in a thesaurus, or the like, in addition to the number of moras or accents.

[0440] Also, weighting may be performed after dividing the time information into a predetermined number of units. The weighting may be determined by machine learning, or a weighting DB (in which weights are set for each type of unit according to the pronunciation characteristics of the target language) may be prepared in advance. The learning data for machine learning may be created using a voice synthesizer capable of adding detailed time information, or may be created manually.

[0441] In addition, in the case where only text needs to be output in the non-verbal information generating device, playback speed information of the utterance text can be acquired, and the occurrence of the behavior can be synchronized with the text display of the utterance. In this case, the time information added to the non-verbal information can be expressed in accordance with the length of the utterance (or the duration of the phrase) to be played back, without dividing it into a predetermined number of units.

[0442] [Ninth embodiment]

[0443] Next, a ninth embodiment of the present invention will be described. Note that parts having the same configuration as the first embodiment are given the same reference numerals and the description thereof will be omitted.

[0444] <Overview of the ninth embodiment>

[0445] When generating non-verbal information, if time series information is not sufficiently taken into consideration, or if a non-verbal information generation model is trained for multiple actions and non-verbal information is generated for each action individually, unnatural or impossible non-verbal information will be generated. Unnatural non-verbal information is an action that is inappropriate when performed simultaneously, such as jumping while bowing. Impossible non-verbal information is an action that is inappropriate when viewed in chronological order, such as nodding slowly five times, which is assigned to only one morpheme. This action is assigned to only one morpheme, and is impossible because there is not enough time.

[0446] Therefore, in this embodiment, constraints are set on the non-language information to be generated, and the generated data is corrected (inserted, deleted, or replaced).

[0447] For example, in order to express natural non-language information, constraint conditions are used to delete unnatural non-language information from training data or generated non-language information, or constraint conditions are used to add non-language information.

[0448] The constraints used are at least one of the following: constraints on the non-verbal information itself, constraints on the shape of the expression (CG character / robot), and constraints using additional information. The constraints are manually defined as a set of rules.

[0449] <Configuration of a learning device for non-linguistic information generation model>

[0450] The non-language information generation model learning device according to the ninth embodiment has the same configuration as the non-language information generation model learning device 810 according to the eighth embodiment, so the same reference numerals are used and the description will be omitted.

[0451] In the non-verbal information generation model learning device of the ninth embodiment, the learning data creation unit 833 changes the learning non-verbal information or time information for the discretized learning non-verbal information with time information extracted by the generation parameter extraction unit 34 so as to satisfy constraints on the time series of the non-verbal information or constraints on the non-verbal information whose time corresponds.

[0452] For example, the minimum necessary time information or the number of text units (phrases, morphemes, etc.) for each action is set as a constraint condition. This constraint condition is set based on the shape of the expression unit 70, the possible movement speed, the current posture from the previous action, etc.

[0453] Specifically, an example will be described in which actions are generated on a per-segment basis. In the case where a constraint is set that hand gesture A must perform an action across three segments, if hand gesture A is generated for only one segment, the segment before or after, or both, will be modified to be labeled with hand gesture A. If this is not possible, the action label for hand gesture A will be deleted. Note that if it is determined that the expression of the generated non-verbal information is impossible, it is a good idea to prepare an alternative action.

[0454] Furthermore, regarding the setting of the constraint conditions, it is possible to create in advance which behavior should be generated and for how long (or in how many text units) based on actual human behavior data. In this case, as shown in FIG. 2 above, data can be used that is created by acquiring the speech information (X) of the speaker while simultaneously acquiring the non-verbal information (Y) of the speaker using a specified measuring device. This makes it possible to set constraint conditions that allow for more natural human movements. Note that, if the expression unit 70 does not simulate a human, the constraint conditions may be set so as to purposely allow unnatural movements.

[0455] Furthermore, the learning data creation unit 833 changes the learning non-verbal information or time information so as to satisfy the constraint conditions set using the additional information, taking into consideration the additional information with time information extracted by the learning feature extraction unit 732. For example, if the speech rate increases due to emotions, it may become impossible to perform actions that were previously possible, so the time information added to the learning non-verbal information is changed.

[0456] The learning unit 735 learns a non-language information generation model for generating non-language information with time information from the text features with time information and the additional information, based on the learning text features with time information and the additional information with time information extracted by the learning feature extraction unit 732, and the learning non-language information with time information modified by the learning data creation unit 833. At this time, it is preferable to learn the non-language information generation model using sequence labeling (CRF (Conditional Random Fields)).

[0457] In this case,

[0458]

number

[0459] In order to take into account the time series relationship of the labels, sequence information such as BIO (Begin, Inside, Outside) tags may be added. For example, when a certain label appears consecutively, the B (Begin) label is added to the first label, and I (Inside) is added to the subsequent labels. This increases the accuracy of estimating consecutive labels.

[0460] Using the labeled data in this way, we train a non-verbal information generation model that generates gestures using a sequence labeling method. Although SVM can be used for this, it is even better to use HMM (Hidden Markov Model) or CRF (Conditional Random Fields, Reference 9).

[0461] [Reference 9] Japanese Patent No. 5152918

[0462] <Configuration of non-verbal information generation device>

[0463] The non-language information generation device according to the ninth embodiment has the same configuration as the non-language information generation device 740 according to the seventh embodiment, so the same reference numerals are used and the description will be omitted.

[0464] In the non-language information generation device of the ninth embodiment, the non-language information generation unit 764 generates non-language information with time information corresponding to the text features with time information and additional information extracted by the feature extraction unit 762, based on the text features with time information and additional information with time information generated by the feature extraction unit 762 and a trained non-language information generation model stored in the trained model memory unit 763.

[0465] The non-language information generation unit 764, similar to the learning data creation unit 833, modifies the generated non-language information with time information or the time information assigned to the non-language information so as to satisfy constraints on the time series of the non-language information or constraints on the non-language information whose time corresponds.

[0466] Then, the non-language information generation unit 764 controls the expression unit 70 so that generation parameters corresponding to the changed non-language information with time information are output from the expression unit 70 based on the time information assigned to the generation parameters.

[0467] In accordance with the control of the non-verbal information generation unit 764, the expression unit 70 outputs text information or audio information corresponding to the text information received by the input unit 750 in accordance with detailed time information, and expresses the non-verbal information generated by the non-verbal information generation unit 764 in accordance with the detailed time information.

[0468] <Functions of the non-verbal information generation model learning device>

[0469] Next, the operation of the non-language information generation model learning device according to the ninth embodiment will be described. First, when learning data representing a combination of a plurality of pieces of text information for learning, a plurality of pieces of additional information for learning, and a plurality of pieces of non-language information for learning is input to the learning input unit 720 of the non-language information generation model learning device, the non-language information generation model learning device executes a learning process routine shown in FIG.

[0470] First, in step S300, the learning information acquisition unit 231 acquires learning text information from among the multiple pieces of learning data (specifically, a combination of text information, non-language information, and additional information) accepted by the learning input unit 720.

[0471] In step S102, the non-verbal information acquisition unit 33 acquires non-verbal information for learning from the multiple learning data accepted by the learning input unit 720 and time information representing the time from the start time to the end time when the behavior represented by the non-verbal information for learning is performed.

[0472] In step S303, the training text analysis unit 237 performs a predetermined text analysis on the training text information acquired in step S300, and acquires a text analysis result. The training voice synthesis unit 238 synthesizes training voice information corresponding to the training text information, based on the text analysis result acquired by the training text analysis unit 237. The training voice synthesis unit 238 then acquires time information indicating the time from the start time to the end time when the training voice information is uttered.

[0473] In step S700, the learning-use additional information acquisition unit 731 acquires additional information for learning from among the multiple pieces of learning data accepted by the learning-use input unit 720.

[0474] In step S702, the learning feature extraction unit 732 extracts learning text features with time information from the learning text information and time information acquired in step S303. In addition, the learning feature extraction unit 732 generates learning additional information with time information from the learning additional information acquired in step S700 and the time information acquired in step S303.

[0475] In step S106, the generation parameter extraction unit 34 extracts discretized non-language information with time information from the learning non-language information and time information acquired in step S102.

[0476] In step S900, the learning data creation unit 833 modifies the non-verbal information or the time information assigned to the non-verbal information, so as to satisfy the constraint conditions on the time series of the non-verbal information or the constraint conditions on the non-verbal information whose time corresponds, for the discretized non-verbal information with time information obtained in step S106 above.

[0477] In step S708, the learning unit 735 learns a non-language information generation model for generating generation parameters with time information from the text features and additional information with time information, based on the learning text features with time information and learning additional information with time information extracted in step S702 above, and the learning non-language information with time information changed in step S900.

[0478] In step S110, the learning unit 735 stores the trained non-language information generation model obtained in step S708 above in the trained model storage unit 736, and ends the learning process routine.

[0479] <Function of the non-verbal information generating device>

[0480] Next, the operation of the non-language information generation device according to the ninth embodiment will be described. First, when a trained non-language information generation model stored in the trained model storage unit 736 of the non-language information generation model learning device 810 is input to the non-language information generation device, the trained non-language information generation model is stored in the trained model storage unit 763 of the non-language information generation device. Then, when text information and additional information for which non-language information is to be generated are input to the input unit 750, the non-language information generation device executes a non-language information generation processing routine shown in FIG.

[0481] In step S 400 , information acquisition unit 261 acquires text information accepted by input unit 750 .

[0482] In step S401, the text analysis unit 265 performs a predetermined text analysis on the text information acquired in step S400, and acquires a text analysis result. In addition, the voice synthesis unit 266 synthesizes voice information corresponding to the text information based on the text analysis result acquired by the text analysis unit 265. Then, the voice synthesis unit 266 acquires time information indicating the time from the start time to the end time when the voice information is uttered.

[0483] In step S 750 , the additional information acquisition unit 761 acquires the additional information accepted by the input unit 750 .

[0484] In step S752, the feature extraction unit 762 extracts text features with time information from the text information and time information acquired in step S401, and generates additional information with time information from the additional information acquired in step S750 and the time information acquired in step S401.

[0485] In step S754, the non-language information generation unit 764 reads out the trained non-language information generation model stored in the trained model storage unit 763.

[0486] In step S950, the non-language information generation unit 764 generates generation parameters with time information corresponding to the text features and additional information with time information extracted in step S752, based on the text features and additional information with time information extracted in step S752 and the trained non-language information generation model read out in step S754. Then, the non-language information generation unit 764 changes the non-language information or the time information assigned to the non-language information, so that the generated generation parameters with time information satisfy the constraint conditions on the time series of the non-language information or the constraint conditions on the non-language information whose time corresponds.

[0487] In step S208, the non-verbal information generation unit 764 controls the expression unit 70 so that the non-verbal information with time information changed in step S950 above is output from the expression unit 70 based on the time information added to the non-verbal information, and then terminates the non-verbal information generation processing routine.

[0488] Other configurations and operations of the non-language information generation device and the non-language information generation model learning device according to the ninth embodiment are similar to those of the first embodiment, and therefore description thereof will be omitted.

[0489] As described above, the non-language information generation device according to the ninth embodiment acquires voice information corresponding to text information, acquires time information representing the time from the start time to the end time when the voice information is uttered, generates non-language information with time information corresponding to the text feature with time information based on the text feature with time information and a trained model for generating non-language information with time information, and changes the non-language information or the time information of the non-language information so as to satisfy the constraint conditions. This makes it possible to automatically associate text information with non-language information by eliminating non-natural non-language information, thereby reducing the cost involved.

[0490] Moreover, according to the non-language information generation model learning device of the ninth embodiment, the non-language information or the time information of the non-language information is changed so as to satisfy the constraint condition for the learning non-language information with time information. Then, based on the learning text feature with time information and the learning non-language information with time information, a non-language information generation model for generating non-language information with time information from the text feature with time information is learned. This makes it possible to obtain a non-language information generation model for generating non-language information from the text feature while eliminating non-language information that is not natural and reducing the cost of associating text information with non-language information. Note that a machine learning model for rewriting may be created based on the constraint condition.

[0491] [Tenth embodiment]

[0492] Next, a tenth embodiment of the present invention will be described. Note that parts having the same configuration as the first embodiment are given the same reference numerals and the description thereof will be omitted.

[0493] <Outline of the Tenth Embodiment>

[0494] This differs from the ninth embodiment in that when non-verbal information that is not possible due to constraints is detected, a pause is inserted in the voice data (including synthetic voice) or the time information attached to the text information is changed to change the display speed of the text (the speaking speed of the synthetic voice) so as to satisfy the constraints.

[0495] This embodiment is effective when non-verbal information is more important than voice (text), and is particularly effective when the output voice is generated by voice synthesis or when text is output.

[0496] <Configuration of a learning device for non-linguistic information generation model>

[0497] The non-language information generation model learning device according to the tenth embodiment has the same configuration as the non-language information generation model learning device 810 according to the eighth embodiment, so the same reference numerals are used and the description will be omitted.

[0498] In the non-verbal information generation model training device according to the tenth embodiment, the training data creation unit 833 changes the time information of the training text information and the time information assigned to the training text features acquired by the training feature extraction unit 732 so as to satisfy constraints on the time series of the non-verbal information.

[0499] For example, the time information attached to the learning text information and learning text features may be changed so as to insert pauses in accordance with the non-verbal information in order to satisfy constraint conditions, or the time information attached to the learning text information and text features may be changed so as to change the display speed of the text (the speaking speed of the synthetic voice) in accordance with the non-verbal information.

[0500] In addition, the learning data creation unit 833 takes into consideration the additional information with time information extracted by the learning feature extraction unit 732 and changes the time information added to the text information so as to satisfy the constraint conditions set using the additional information.

[0501] The learning unit 735 learns a non-language information generation model for generating non-language information with time information from the text features with time information and additional information based on the learning text features with time information modified by the learning data creation unit 833, the additional information with time information extracted by the learning feature extraction unit 732, and the learning non-language information with time information extracted by the generation parameter extraction unit 34.

[0502] <Configuration of non-verbal information generation device>

[0503] The non-language information generation device according to the tenth embodiment has the same configuration as the non-language information generation device 740 according to the seventh embodiment, so the same reference numerals are used and the description will be omitted.

[0504] In the non-language information generation device of the tenth embodiment, the non-language information generation unit 764 generates non-language information with time information corresponding to the text features with time information and additional information extracted by the feature extraction unit 762, based on the text features with time information and additional information with time information generated by the feature extraction unit 762 and a trained non-language information generation model stored in the trained model memory unit 763.

[0505] The non-language information generating unit 764 changes the time information for the text information and the text feature with time information, in the same manner as the learning data creating unit 833, so as to satisfy the constraints on the time series of the non-language information.

[0506] Then, the non-language information generation unit 764 controls the expression unit 70 so that generation parameters corresponding to the generated non-language information with time information are output from the expression unit 70 based on the time information assigned to the generation parameters.

[0507] In accordance with the control of the non-verbal information generation unit 764, the expression unit 70 outputs the text information received by the input unit 750 or the audio information corresponding to the text information in accordance with the changed time information, and expresses the non-verbal information generated by the non-verbal information generation unit 764 in accordance with the time information.

[0508] Other configurations and operations of the non-language information generation device and the non-language information generation model learning device according to the tenth embodiment are similar to those of the ninth embodiment, and therefore will not be described.

[0509] As described above, the non-language information generation device according to the tenth embodiment acquires voice information corresponding to text information, acquires time information representing the time from the start time to the end time when the voice information is uttered, generates non-language information with time information corresponding to the text feature with time information based on the text feature with time information and a trained model for generating non-language information with time information, and changes the time information of the text information so as to satisfy the constraints. This makes it possible to automatically associate text information with non-language information by eliminating unnatural information, thereby reducing the cost involved.

[0510] Furthermore, according to the non-language information generation model learning device of the tenth embodiment, the time information of the text features for learning with time information and the learning non-language information is changed so as to satisfy the constraint conditions. Then, based on the learning text features with time information and the learning non-language information with time information, a non-language information generation model for generating non-language information with time information from the text features with time information is learned. This makes it possible to obtain a non-language information generation model for generating non-language information from text features while eliminating unnatural information and reducing the cost of associating text information with non-language information.

[0511] [Eleventh embodiment]

[0512] Next, an eleventh embodiment of the present invention will be described. Note that parts having the same configuration as the first embodiment are given the same reference numerals and the description thereof will be omitted.

[0513] <Overview of the 11th embodiment>

[0514] When creating learning data for a gesture scenario or a non-verbal information generation model, it is essential to check and correct whether appropriate actions are performed for utterances, i.e., whether the intended actions are performed. However, when checking and correcting actions, it is difficult to know which actions are assigned to which utterances, and the work cost tends to be high. Therefore, in this embodiment, by visualizing which non-verbal information is assigned to which text information, the checking and correction of actions is made easier.

[0515] Specifically, the system displays non-verbal information in association with linguistic information (text or speech) divided into predetermined units, and allows users to confirm the actual behavior for each predetermined unit, providing an interface that makes it easy to make corrections. In addition, depending on the correction results, the system can add / correct training data, and furthermore, retrain the non-verbal information generation model. Here, the interface described in this embodiment can be used in the following five situations, for example. (1) When creating a gesture scenario, the interface according to the present embodiment is used. For example, for input text information, non-verbal information with time information is generated based on a trained non-verbal information generation model, the generated non-verbal information is modified by a user operation, and the modified result is output as a fixed scenario. (2) When the learning data is to be corrected, the interface according to the present embodiment is used. For example, the input learning data is read, text information or non-language information contained in the learning data is corrected by a user operation, and the correction result is output as the learning data. (3) When adding learning data, the interface according to the present embodiment is used. For example, for input text information, non-verbal information with time information is generated based on a trained non-verbal information generation model, the generated non-verbal information is corrected by a user operation, and the correction result is output as learning data corresponding to the non-verbal information generation model. (4) The interface according to the present embodiment is used when re-training a trained non-language information generation model. For example, for input text information, non-language information with time information is generated based on the trained non-language information generation model, the generated non-language information is corrected by a user operation, and the correction result is added as training data corresponding to the non-language information generation model, and the non-language information generation model is re-trained. (5) When generating the constraint conditions described in the ninth and tenth embodiments, the interface according to this embodiment is used. For example, the constraint conditions are specified using the correction results obtained by the same method as any one of the above (1) to (4).

[0516] <Configuration of non-verbal information generation device>

[0517] Fig. 31 is a block diagram showing an example of the configuration of a non-language information generation device 1140 according to the 11th embodiment. As shown in Fig. 31, the non-language information generation device 1140 according to the 11th embodiment is configured by a computer including a CPU, a RAM, and a ROM storing a program for executing a non-language information generation processing routine described later. The non-language information generation device 1140 functionally includes an input unit 750, a calculation unit 1160, a display unit 1190, and an output unit 1192.

[0518] The input unit 750 accepts text information and additional information. The accepted additional information is additional information for each predetermined processing unit (for example, for each morpheme or for each phrase). If the additional information does not change for each predetermined processing unit (for example, gender), it does not need to be accepted for each predetermined processing unit. In this case, when the additional information is accepted, it is sufficient for the device to expand it into additional information for each predetermined processing unit.

[0519] The calculation unit 1160 includes an information acquisition unit 261, an additional information acquisition unit 761, a feature extraction unit 762, a trained model storage unit 763, a non-verbal information generation unit 764, a control unit 1170, a training data generation unit 1172, and a re-learning control unit 1174. The additional information acquisition unit 761 may be omitted. In the above usage scenes (1) and (5), the training data generation unit 1172 and the re-learning control unit 1174 may also be omitted. In the above usage scenes (2) and (3), the re-learning control unit 1174 may also be omitted.

[0520] The information acquisition unit 261, similar to the information acquisition unit 261 of the non-language information generation device 240 relating to the second embodiment described above, acquires text information accepted by the input unit 750, acquires voice information corresponding to the text information, and acquires time information indicating the time from the start time to the end time when the voice information is uttered. The additional information acquiring section 761 acquires the additional information accepted by the input section 750, similarly to the additional information acquiring section 761 of the non-language information generation device 740 according to the seventh embodiment. The feature extraction unit 762, like the feature extraction unit 762 of the non-language information generation device 740 according to the seventh embodiment, extracts text features with time information, which represent features of the text information, from the text information and time information acquired by the information acquisition unit 261. In addition, the feature extraction unit 762 generates additional information with time information from the additional information acquired by the additional information acquisition unit 761 and the time information acquired by the information acquisition unit 261. The trained model memory unit 763, like the trained model memory unit 763 of the non-language information generation device 740 of the seventh embodiment described above, stores a trained non-language information generation model that is identical to the trained non-language information generation model stored in the trained model memory unit 736. The non-language information generation unit 764, like the non-language information generation unit 764 of the non-language information generation device 740 relating to the above-mentioned seventh embodiment, generates non-language information with time information corresponding to the text features with time information and additional information extracted by the feature extraction unit 762, based on the text features with time information and additional information with time information extracted by the feature extraction unit 762, and the trained non-language information generation model stored in the trained model memory unit 763. The control unit 1170 controls the display unit 1190 to display the non-language information with time information generated by the non-language information generation unit 764 and the text information and additional information received by the input unit 750.

[0521] The display unit 1190 includes a display screen 1190A and a manifestation unit 1190B. In the present embodiment, the manifestation unit 1190B is included in the display unit 1190, but is not limited to this and may be configured as a device (for example, a robot) separate from the display unit 1190.

[0522] The expression unit 1190B outputs a voice corresponding to the text information, and expresses the behavior indicated by the generated non-verbal information with time information in accordance with the time information. Alternatively, a speech bubble including the text information may be displayed.

[0523] At this time, an example of a display screen 1190A displayed by the display unit 1190 is shown in FIG.

[0524] Display screen 1190A displays the text divided into predetermined units, and also displays labels indicating non-language information in association with each predetermined unit of text based on the time information added to the text feature and the time information added to the non-language information. Display screen 1190A may also display the speech waveform of the speech corresponding to the text information.

[0525] The added time information is the time when the text is output, and may be added based on the result of dividing the range of times when the text is output by the number of divisions when the text is divided into a predetermined unit, as in the eighth embodiment above.

[0526] The display unit 1190 also includes an expression unit 1190B that expresses behavior indicating non-verbal information, and displays a display screen 1190A in a state in which it is possible to accept instructions to start or stop the expression of the behavior by the expression unit 1190B, or to fast-forward or rewind by a predetermined unit (for example, one morpheme or one phrase). For example, a play button, a pause button, a rewind button, and a fast-forward button are displayed on the display screen 1190A.

[0527] A slide bar capable of receiving an instruction to fast-forward or rewind the manifestation of the action by the manifestation unit 1190B may be displayed on the display screen 1190A.

[0528] When the control unit 1170 receives an instruction to start, stop, fast-forward, or rewind the manifestation, it controls the manifestation of the behavior by the manifestation unit 1190B in accordance with the instruction. The display unit 1190 may display the behavior expressed by the expression unit 1190B on the display screen 1190A so that it is possible to identify which part of the text corresponds to the behavior expressed by the expression unit 1190B. For example, a playback bar may be displayed in the corresponding part of the text corresponding to the behavior expressed by the expression unit 1190B, or the color of the square of the corresponding part of the text corresponding to the behavior expressed by the expression unit 1190B may be changed or the square may be blinked.

[0529] Furthermore, the display unit 1190 displays the display screen 1190A in a state in which the setting of additional information can be accepted. When the control unit 1170 accepts the setting of additional information, the control unit 1170 outputs the additional information to the feature extraction unit 762, and controls the display unit 1190 to display, on the display screen 1190A, a label indicating non-language information generated by the non-language information generation unit 764 using the additional information, and text.

[0530] Moreover, display unit 1190 displays display screen 1190A in a state in which an instruction to change the label indicating non-linguistic information can be accepted.

[0531] When the learning data generation unit 1172 receives an instruction to change the label indicating the non-linguistic information, it generates a combination of the text features and additional information with time information extracted by the feature extraction unit 762 and the label indicating the non-linguistic information changed in accordance with the change instruction as learning data for training the non-linguistic information generation model.

[0532] Furthermore, the display unit 1190 displays the display screen 1190A in a state in which it is ready to receive an instruction to re-learn the non-language information generation model and a setting of a weight for the learning data generated by the learning data generation unit 1172. Here, the weight for the learning data is set according to how much importance is given to the added learning data in comparison with the existing learning data at the time of re-learning. For example, when this weight is set to the maximum value, the non-language information generation model is re-learned so that Y is always generated for X of the pair (X, Y) in the added learning data.

[0533] When the re-learning control unit 1174 receives a re-learning instruction and weight settings, it causes the non-language information generation model learning device 710 to learn the non-language information generation model using the learning data generated by the learning data generation unit 1172 and the set weights.

[0534] Specifically, the user operates display screen 1190A in the following steps 1 to 5.

[0535] (Step 1) Set the text by inputting or selecting text indicating a spoken sentence for which non-verbal information is to be generated. For example, input text when generating a gesture scenario as in the above usage scene (1), or when adding learning data as in the above usage scenes (3) and (4). Also, when correcting learning data as in the above usage scene (2), a set of learning data is presented, and the text of the learning data to be corrected is selected.

[0536] (Step 2) A text representing an utterance is divided into predetermined units, and a label Y representing non-linguistic information generated for each divided unit is displayed.

[0537] (Step 3) When the start of expression is instructed, expression section 1190B moves according to the generated non-verbal information.

[0538] (Step 4) The user visually checks the operation of expression unit 1190B.

[0539] (Step 5) When the robot makes a strange movement, by clicking on the cell M (the cell M with a label or the blank cell M), the cell M can be rewritten to a label indicating the correct non-verbal information. As a result, the label can be added to the learning data as a label indicating the correct non-verbal information for the input utterance (at that time, weighting can also be set).

[0540] Note that display screen 1190A may be displayed so that time information for each predetermined unit is displayed and an instruction to change the time information can be received (see FIG. 33). For example, the value of the time information can be edited by clicking on it, and the value of the time information can be changed. Alternatively, the start time for each predetermined unit can be changed by shifting the position of the vertical bar indicating the start time for each predetermined unit to the left or right.

[0541] Also, the display screen 1190A may be displayed so as to accept a modification instruction to insert a pause into the voice data (including synthetic voice). For example, as shown in Fig. 34, a modification instruction to insert the start position of a pause P (see the dotted vertical line in Fig. 34) for each predetermined unit of text information can be accepted, and a modification instruction to change the proportion of the pause length by adjusting the start position of the pause P can be accepted. Similarly, for a pause inserted so as to satisfy the constraint conditions as described in the above tenth embodiment, a modification instruction to change the proportion of the pause length by adjusting the start position of the pause P may also be accepted.

[0542] Also, text features for each predetermined unit (e.g., phrase) and generation parameters corresponding to non-language information may be displayed so that a change instruction can be accepted. For example, as shown in Fig. 32, when a user places a mouse cursor on a cell M of text information or a cell M of non-language information and right-clicks, the text features corresponding to cell M may be overlaid and a change instruction may be accepted. When the text features corresponding to cell M of text information are overlaid, all text features extracted from the text information of cell M are displayed, and when the text features corresponding to cell M of non-language information are overlaid, the text features on which the non-language information was generated may be overlaid.

[0543] The non-language information generation model learning device according to the eleventh embodiment is similar to the non-language information generation model learning device 710 according to the seventh embodiment, so the same reference numerals are used and the description will be omitted.

[0544] <Function of the non-verbal information generating device>

[0545] Next, the operation of the non-language information generation device according to the eleventh embodiment will be described. First, when a trained non-language information generation model stored in the trained model storage unit 736 of the non-language information generation model learning device 710 is input to the non-language information generation device, the trained non-language information generation model is stored in the trained model storage unit 763 of the non-language information generation device. Then, when text information and additional information for which non-language information is to be generated are input to the input unit 750, the non-language information generation device executes a non-language information generation processing routine shown in FIG.

[0546] In step S 400 , information acquisition unit 261 acquires text information accepted by input unit 750 .

[0547] In step S401, the text analysis unit 265 performs a predetermined text analysis on the text information acquired in step S400, and acquires a text analysis result. In addition, the voice synthesis unit 266 synthesizes voice information corresponding to the text information based on the text analysis result acquired by the text analysis unit 265. Then, the voice synthesis unit 266 acquires time information indicating the time from the start time to the end time when the voice information is uttered.

[0548] In step S 750 , the additional information acquisition unit 761 acquires the additional information accepted by the input unit 750 .

[0549] In step S752, the feature extraction unit 762 extracts text features with time information from the text information and time information acquired in step S401, and generates additional information with time information from the additional information acquired in step S750 and the time information acquired in step S401.

[0550] In step S754, the non-language information generation unit 764 reads out the trained non-language information generation model stored in the trained model storage unit 763.

[0551] In step S756, the non-language information generation unit 764 generates generation parameters with time information corresponding to the text features and additional information with time information extracted in step S752, based on the text features and additional information with time information extracted in step S752 and the trained non-language information generation model read out in step S754.

[0552] In step S1100, the control unit 1170 controls the display unit 1190 to display the non-verbal information with time information generated by the non-verbal information generation unit 764, and the text information and additional information accepted by the input unit 750, and then terminates the non-verbal information generation processing routine.

[0553] The process of step S1100 is realized by a process routine shown in FIG.

[0554] First, in step S1150, control unit 1170 divides the text into predetermined units and displays them, and also displays labels indicating the non-language information on display screen 1190A, corresponding to each predetermined unit of text, based on the time information assigned to the text features and the time information assigned to the non-language information.

[0555] In step S1152, control unit 1170 determines whether or not an operation on display screen 1190A has been accepted. If control unit 1170 has accepted an operation on display screen 1190A, the process proceeds to step S1154.

[0556] In step S1154, the control unit 1170 determines whether the type of the operation received in step S1152 is a change instruction, a manifestation instruction, or a re-learning instruction. If the received operation is an instruction to change the setting of additional information or the label indicating non-verbal information, in step S1156, the control unit 1170 displays a result reflecting the change according to the change instruction on the display screen 1190A. If the received operation is the setting of additional information, the control unit 1170 further outputs the additional information to the feature extraction unit 762, and displays the label indicating the non-verbal information generated by the non-verbal information generation unit 764 using the additional information and the text on the display screen 1190A.

[0557] Furthermore, if the accepted operation is an instruction to change the label indicating non-language information, the learning data generation unit 1172 generates a combination of the text feature with time information and additional information extracted by the feature extraction unit 762 and the label indicating the non-language information changed in accordance with the change instruction as learning data for training the non-language information generation model, and outputs the combination to the non-language information generation model training device 710. Then, the process returns to the above step S1152.

[0558] Furthermore, if an instruction to start, stop, fast-forward one sentence at a time, or rewind the manifestation of behavior by the manifestation unit 1190B is received, in step S1158, the control unit 1170 controls the manifestation of behavior by the manifestation unit 1190B in accordance with the received instruction, and the process returns to step S1152 above.

[0559] Furthermore, if a re-learning instruction and weight setting are accepted, in step S1160, the control unit 1170 causes the non-language information generation model learning device 710 to learn the non-language information generation model using the learning data generated by the learning data generation unit 1172 and the set weight, and then ends the processing routine.

[0560] As described above, the user modifies the result of non-language information generated using the non-language information generation model for the input text information and additional information, whereby learning data for the non-language information generation model can be generated and learning data for the non-language information generation model can be added. In addition, the user instructs re-learning of the non-language information generation model, whereby re-learning is performed using the added learning data, and the non-language information generation model can be updated.

[0561] As described above, the non-verbal information generation device of the 11th embodiment makes it easier to modify the non-verbal information by visualizing what kind of non-verbal information is assigned to what kind of text information.

[0562] Note that a gesture scenario including text information with time information and non-verbal information generated by the user's correction may be output as a fixed scenario.

[0563] Furthermore, similar to the methods described in the ninth or tenth embodiment above, when a combination of text information with time information and non-verbal information is rewritten based on constraint conditions, the rewritten result may be modified by the user, and the data may be used as learning data to create a machine learning model for rewriting.

[0564] In addition, when a combination of text information with time information and non-verbal information is rewritten based on constraint conditions, the rewriting history and which constraint conditions have been applied may be displayed, and further, modifications to the constraint conditions themselves may be accepted.

[0565] Alternatively, a ranking of text features associated with the labels indicating the non-language information may be obtained from the training data and presented to the user. In this case, pairs of text features and labels indicating non-language information are first obtained from the training data, and for each label indicating non-language information, the type of paired text feature and its occurrence count are counted. Then, for each label indicating non-language information, the types of text features are rearranged in descending order of the count count, and the resulting ranking of text features is presented. Also, after presenting the ranking of text features, the selection of learning data may be accepted, and editing of the selected learning data may be accepted. For example, as shown in FIG. 32, when a user places the mouse cursor on a non-language information cell M and right-clicks, a ranking of features for the labels of cell M is overlaid. Then, a selection instruction (e.g., a click) of each feature name in the ranking of features can be accepted, and when a feature name is selected, learning data consisting of the non-language information label of cell M and the selected text feature is displayed. At this time, the learning data is displayed so that it can be directly edited, and the learning data is edited by deleting, adding, editing, etc. Furthermore, when the learning data is corrected, the learning data before the correction may be added as a negative example.

[0566] Also, a display may be provided that allows modifications to voice synthesis parameters (such as speech rate) to be accepted.

[0567] In the seventh to eleventh embodiments, the input information is text information, but the present invention is not limited to this. For example, the input information may be voice information. When the input information is voice information, the learning information acquisition unit in the non-language information generation model learning device is similar to the learning information acquisition unit 31 in the first embodiment. When the input information is voice information, the information acquisition unit in the non-language information generation device is similar to the information acquisition unit 61 in the first embodiment.

[0568] For example, the configurations corresponding to the combination of the information acquisition unit (or the learning information acquisition unit) and the feature extraction unit (or the learning feature extraction unit) in each of the above embodiments are a total of four patterns shown in Fig. 15. Also, the possible variations as combinations of the configurations during learning and during non-language information generation are the patterns shown in Fig. 16.

[0569] In addition, as learning data used in the non-verbal information generation model learning device described in the fourth to sixth embodiments, for example, in a scene as shown in FIG. 2, speech information (X) of a speaker who is currently speaking is acquired, and at the same time, non-verbal information (Y) of a conversation partner who is currently speaking with the speaker is acquired using a measurement device may be applied to each of the seventh to eleventh embodiments.

[0570] [Experimental Example]

[0571] Next, an experimental example related to the fifth embodiment will be described.

[0572] [About the experimental data]

[0573] We constructed corpus data including text information representing speech and non-verbal information representing accompanying nods for two-person dialogues. The participants in the two-person dialogues were Japanese men and women in their 20s to 50s who met for the first time. There were a total of 24 participants (12 pairs). The participants sat facing each other. The dialogue content was an animation explanation task in order to collect as much data on nods accompanying speech. Each participant watched an animation with a different content, and then explained the animation content to their dialogue partner. In the 10-minute dialogue session, one participant explained the content of the animation in detail to the dialogue partner. The dialogue partner was allowed to freely ask questions to the explainer and to talk freely. A directional lapel microphone was attached to the chest of each subject to record the speech. A video camera was used to record the overall appearance of the dialogue situation and the state of the participants. The video was recorded at 30 Hz. The obtained text information and non-verbal information are shown below.

[0574] Text information representing speech: After manually transcribing the spoken language from the audio information, we divided the speech into sentences. Furthermore, we divided each sentence into phrases using a dependency analysis engine (see References 10 and 11). The number of phrases divided was 11,877.

[0575] [Reference 10] Kenji Imamura, "Dependency Analysis of Semi-Spoken Japanese Using Sequence Labeling," Proceedings of the 13th Annual Conference of the Association for Natural Language Processing, pp. 518-521, 2007. [Reference 11] E. Charniak, "A Maximum-Entropy-Inspired Parser", Proceedings of the 1st North American chapter of the Association for Computational Linguistics conference, pp. 132-139, 2000.

[0576] Non-verbal information representing nodding: We manually labeled the sections in the video where nodding occurred. Continuous nodding was treated as one nodding event.

[0577] Manual labeling (annotation) was performed by integrating all the above data with a time resolution of 30 Hz.

[0578] [Non-verbal information generation model]

[0579] Using the constructed corpus data, we constructed a non-verbal information generation model that takes words, their parts of speech and thesaurus, word positions, and the dialogue acts of the entire text information as inputs, and generates non-verbal information representing nodding for each phrase. To verify the effectiveness of each piece of text information, we constructed a non-verbal information generation model using each text feature, and a non-verbal information generation model using all text features. Specifically, we implemented a non-verbal information generation model that takes text features obtained from the target phrase and the phrases before and after it for each phrase as inputs, and outputs a binary value indicating whether or not a nod is present, using the decision tree algorithm C4.5 (see Reference 12). The text features used are as follows:

[0580] [Reference 12] JR Quinlan, "Improved use of continuous attributes in c4.5", Journal of Artificial Intelligence Research, 4:77-90, 1996.

[0581] Number of characters: Number of characters in the phrase

[0582] Position: Position of the phrase from the beginning or end of the sentence

[0583] Words: Word information (Bag-of-words) within a phrase extracted by the morphological analysis tool Jtag (see Reference 13).

[0584] Part of speech: Part of speech information of words in a phrase extracted by Jtag

[0585] Thesaurus: Thesaurus information for words in the phrase based on the Japanese Lexicon (see reference 14)

[0586] Dialogue acts: Dialogue acts (33 types) extracted for each sentence using a dialogue act inference method that uses word n-grams and thesaurus information (see references 4, 15).

[0587] [Reference 13] Takeshi Fuchi and Shinichiro Takagi, "Japanese morphological analyzer using word co-occurrence -Jtag-", In Proceedings of International conference on Computational linguistics, pages 409-413, 1998.

[0588] [Reference 14] Satoru Ikehara, Masahiro Miyazaki, Satoru Shirai, Akio Yokoo, Hiromi Nakaiwa, Kentaro Ogura, Yoshifumi Oyama, and Yoshihiko Hayashi, "Japanese Vocabulary Series," Iwanami Shoten, 1997.

[0589] [Reference 15] Toyomi Meguro, Ryuichiro Higashinaka, Yasuhiro Minami, and Kohji Dohsaka, "Controlling listening-oriented dialogue using partially observable markov decision processes", In Proceedings of International conference on computational linguistics, pages 761-769, 2010.

[0590] [About the experimental results]

[0591] Of the 24 participants' data, 23 people's data was used for learning, and the remaining participant's data was used for evaluation, using a 24-way cross-validation method. This allowed us to evaluate how well nodding could be generated from other people's data. In addition, the amount of data on the presence or absence of nodding in each experiment was reduced to match the amount of data with fewer data, so that the amount of data was consistent. Therefore, the baseline chance level was 0.50. The average performance evaluation results are shown in Table 5.

[0592] Evaluation of the non-linguistic information generation model showed that the order of accuracy was best for vocabulary, parts of speech, and words. In machine learning, if the resolution of the extracted text features is too high (the number of types is large), the frequency of occurrence of each text feature becomes relatively low, and text features that have never appeared in the training data tend to appear more frequently during execution, which reduces the accuracy of generation. Conversely, the lower the resolution (reducing the number of types through abstraction), the less likely the above problems are to occur, but the differences in the data cannot be expressed, which tends to reduce the accuracy of generation.

[0593] Thesaurus information classifies words by their meanings and attributes, and since the number of types is fewer than words but more than parts of speech, it is believed that efficient learning was possible even with the amount of training data used in this experiment.Thesaurus information has a hierarchical structure, and since it is possible to superordinate (abstraction) words at multiple levels, it is easy to control the degree of abstraction according to the size of the training data.

[0594] Creating a huge amount of corpus data is costly and difficult. When sufficient training data is not available, better learning results can be expected even with a relatively small amount of data by using thesaurus information.

[0595] [Table 5] [Industrial Applicability]

[0596] The present invention can be used, for example, in a technique for expressing non-verbal behavior in accordance with playback of an utterance. According to the present invention, it is possible to automate the association of at least one of voice information and text information with non-verbal information. [Explanation of symbols]

[0597] 10,210,710,710A,810 Non-verbal information generation model learning device 20,220,720 Learning input section 30,230,730,730A,830 Learning calculation unit 31,231,331,431 Learning information acquisition section 32,232,332,432,732 Learning feature extraction unit 33 Nonverbal information acquisition unit 34 Generation parameter extraction part 35,235,735 Learning Department 36,63,236,263,736,763 Trained model memory 40,240,740,740A,840,1140 Nonverbal information generation device 50,250,750 Input section 60,260,760,760A,860,1160 Arithmetic unit 61,261,361,461 Information acquisition department 62,262,362,462,762 Feature Extraction Section 64,264,764 Nonverbal information generation unit 70,1190B Expression section 237,338,437 Learning text analysis section 238,438 Training speech synthesis unit 265,366,465 Text Analysis Section 266,466 Voice synthesis section 337 Training speech recognition unit 365 Voice Recognition Unit 731 Acquisition of Additional Information for Learning 731A Additional information estimation unit for learning 761 Additional Information Acquisition Unit 761A Additional information estimation unit 833 Learning Data Creation Department 870,1170 Control section 1172 Learning Data Generation Unit 1174 Re-learning control unit 1190 Display section 1190A display screen 1192 Output section

Claims

1. A non-language information generation device including a non-language information generation unit that generates non-language information with time information corresponding to a text feature with time information based on the text feature with time information and a trained non-language information generation model, the text feature with time information includes a feature extracted from a text and time information indicating a time assigned to each predetermined unit of the text, and the non-linguistic information is information for controlling an expression unit that expresses an action corresponding to the text, the text features include dialogue act based features extracted from the text, The dialogue act is an abstraction of the intention in the utterance, abstracted as a label. Nonverbal information generating device.

2. A non-language information generation device including a non-language information generation unit that generates non-language information with time information corresponding to a speech feature with time information based on the speech feature with time information and a trained non-language information generation model, the speech feature with time information includes a feature extracted from speech information and time information representing a time for each predetermined unit when the speech information is uttered, and the non-linguistic information is information for controlling an expression unit that expresses an action corresponding to the speech information, the speech features include dialogue act-based features extracted from the speech information, The dialogue act is an abstraction of the intention in the utterance, abstracted as a label. Nonverbal information generating device.

3. a feature extraction unit that performs text analysis on an input text, assigns a playback time when the text is assumed to be outputted to the outside as speech to the text for each predetermined unit as the time information, and extracts text features with the time information; The non-verbal information generating device according to claim 1 .

4. A non-language information generation method in which a computer generates non-language information with time information corresponding to a text feature with time information based on the text feature with time information and a trained non-language information generation model, the method comprising: the text feature with time information includes a feature extracted from a text and time information indicating a time assigned to each predetermined unit of the text, and the non-linguistic information is information for controlling an expression unit that expresses an action corresponding to the text, the text features include dialogue act based features extracted from the text, The dialogue act is an abstraction of the intention in the utterance, abstracted as a label. Nonverbal information generation method.

5. A non-language information generation method in which a computer generates non-language information with time information corresponding to a speech feature with time information, based on the speech feature with time information and a trained non-language information generation model, the method comprising: the speech feature with time information includes a feature extracted from speech information and time information representing a time for each predetermined unit when the speech information is uttered, and the non-linguistic information is information for controlling an expression unit that expresses an action corresponding to the speech information, the speech features include dialogue act-based features extracted from the speech information, The dialogue act is an abstraction of the intention in the utterance, abstracted as a label. Nonverbal information generation method.

6. A program for causing a computer to function as the non-verbal information generation device according to any one of claims 1 to 3.

Images