Natural language reaction control program, natural language reaction control system, and natural language reaction control method

Abstract

[Problem] The present invention reduces discomfort or stress experienced by a human user in a conversation between the user and an object such as a character controlled by a system. [Solution] Provided is a natural language reaction control program for controlling reaction in a natural language by an object in response to utterance by a user, the natural language reaction control program causing a computer to implement a user utterance information acquisition function for acquiring utterance information that is information indicating content uttered by the user, and a reaction control function for causing the object to produce reaction according to the utterance information. The reaction includes at least utterance by the object. In the reaction control function, the content of reaction by the object is controlled on the basis of the start or end of utterance by the user.

Description

Natural language reaction control program, natural language reaction control system, and natural language reaction control method

[0001] At least one embodiment of the present invention relates to a natural language reaction control program, a natural language reaction control system, and a natural language reaction control method.

[0002] There are services that allow human users to interact with avatars controlled by artificial intelligence.

[0003] Patent Literature 1 describes a chat system. The chat system includes a log recording unit that stores, in a log data storage unit, dialogue history log data for messages sent from a user to a chatbot and messages sent from the chatbot to the user, including the text of the messages, the time of sending, and an identifier for identifying whether the messages were sent from the user or the chatbot. The chat system includes a similar dialogue history search unit that, when a user sends a message to the chatbot, calculates a history similarity between a user's own dialogue history, which is a dialogue history for the user up to the time the message was sent, and a log dialogue history, which is a dialogue history for multiple users stored as log data in the log data storage unit, and searches for a log dialogue history similar to the user's own dialogue history in a portion of the log dialogue history excluding the most recent message. The chat system includes a message generation unit that generates a message for the user using the most recent message included in the log dialogue history searched by the similar dialogue history search unit. The chat system also includes a message sending unit that sends the message generated by the message generation unit to the user.

[0004] The similar dialogue history search unit calculates the history similarity by calculating sentence similarity regarding at least the sentences of each message included in the dialogue history, and calculates the sentence similarity according to a function designed so that the similarity between messages having the same identifier becomes smaller.

[0005] Patent Publication No. 2024-003862

[0006] When two or more speakers are conversing, their speech timing may overlap. Also, each speaker may wait for the other to speak, resulting in a matchmaking state where neither speaker speaks. Such overlapping speech and matchmaking states can also occur in conversations between a human user and an object, such as a character controlled by a system.

[0007] When the above situation occurs, the user may feel uncomfortable or stressed during the conversation.

[0008] An object of at least one embodiment of the present invention is to solve the above-mentioned problems and provide a natural language reaction control program, a natural language reaction control system, and a natural language reaction control method that can reduce the difficulty and stress felt by a human user in a dialogue between the user and an object such as a character controlled by the system.

[0009] From a non-limiting perspective, a natural language reaction control program according to one embodiment of the present invention is a natural language reaction control program that controls a natural language reaction by an object to a user's utterance, wherein the reaction includes at least an utterance by the object, and the program implements in a computer a user utterance information acquisition function that acquires utterance information, which is information indicating the content of the utterance made by the user, and a reaction control function that causes the object to react in accordance with the utterance information, and the reaction control function controls the content of the reaction by the object based on the start or end of the user's utterance.

[0010] From a non-limiting perspective, a natural language reaction control system according to one embodiment of the present invention is a natural language reaction control system that controls a reaction in natural language by an object to a user's utterance, wherein the reaction includes at least an utterance by the object, and the dialogue system includes a communication network, a server, and a user terminal, and includes a user utterance information acquisition means that acquires utterance information, which is information indicating the content of an utterance made by the user, and a reaction control means that causes the object to react in accordance with the utterance information, and the reaction control means controls the content of the reaction by the object based on the start or end of the user's utterance.

[0011] From a non-limiting perspective, a natural language reaction control method according to one embodiment of the present invention is a computer-based natural language reaction control method for controlling a reaction in natural language by an object to a user's utterance, wherein the reaction includes at least an utterance by the object, and the method includes a user utterance information acquisition process for acquiring utterance information, which is information indicating the content of the utterance made by the user, and a reaction control process for causing the object to react in accordance with the utterance information, and the reaction control process controls the content of the reaction by the object based on the start or end of the user's utterance.

[0012] Each embodiment of the present application addresses one or more of the deficiencies.

[0013] FIG. 1 is a block diagram showing an example of the configuration of a natural language reaction control system corresponding to at least one of the embodiments of the present invention. FIG. 2 is a block diagram showing the configuration of a server corresponding to at least one of the embodiments of the present invention. FIG. 3 is a flowchart showing an example of processing of a natural language reaction control program corresponding to at least one of the embodiments of the present invention. FIG. 4 is a conceptual diagram illustrating an example of a natural language reaction of an object corresponding to at least one of the embodiments of the present invention. FIG. 5 is a conceptual diagram illustrating an example of a natural language reaction of an object corresponding to at least one of the embodiments of the present invention.

[0014] Hereinafter, examples of embodiments of the present invention will be described with reference to the drawings. Note that the various components in the examples of the embodiments described below can be combined as appropriate to the extent that no inconsistencies or the like arise. Furthermore, content described as an example of one embodiment may be omitted in other embodiments. Furthermore, the content of operations and processes unrelated to the characteristic parts of each embodiment may be omitted. Furthermore, the order of various processes constituting the various flows and sequences described below may be in any order to the extent that no inconsistencies or the like arise in the processing content.

[0015] The following description will be given taking as an example a natural language reaction control program executed on a server, which is an example of a computer, but the computer may be another device such as a user terminal.

[0016] FIG. 1 is a block diagram showing an example of the configuration of a natural language reaction control system corresponding to at least one embodiment of the present invention. The natural language reaction control system 1 includes a server 10 and a user terminal 20 used by a user of the natural language reaction control system 1. Each of the user terminals 20A, 20B, and 20C is an example of the user terminal 20. The configuration of the natural language reaction control system 1 is not limited to this. For example, the natural language reaction control system 1 may be configured so that multiple users use a single user terminal. The natural language reaction control system 1 may also include multiple servers.

[0017] The server 10 and the user terminal 20 are examples of computers. The server 10 and the user terminal 20 are each communicatively connected to a communication network 30 such as the Internet. The connection between the communication network 30 and the server 10 and the connection between the communication network 30 and the user terminal 20 may be a wired connection or a wireless connection. For example, the user terminal 20 may connect to the communication network 30 by performing data communication with a base station managed by a telecommunications carrier via a wireless communication line.

[0018] The natural language reaction control system 1 includes a server 10 and a user terminal 20, thereby realizing various functions for executing various processes in response to user operations.

[0019] The server 10 may be a game processing server that controls the progress of the game. In this case, the server 10 controls the progress of the game in accordance with the player's operations. The server 10 may provide various electronic services to users that are not limited to games. The electronic services may be services that use a virtual space. For example, an electronic service that allows a user to control a player character in a virtual space is conceivable.

[0020] The server 10 includes a processor 11, a memory 12, and a storage device 13. The processor 11 is, for example, a central processing unit such as a CPU (Central Processing Unit) that performs various calculations and control. If the server 10 includes a GPU (Graphics Processing Unit), some of the calculations and control may be performed by the GPU. The server 10 uses the data read into the memory 12 to execute various information processing operations using the processor 11, and stores the obtained processing results in the storage device 13 as necessary.

[0021] The storage device 13 functions as a storage medium for storing various types of information. The configuration of the storage device 13 is not particularly limited, but may be configured to store all of the various types of information necessary for the control performed by the natural language reaction control system 1, from the viewpoint of reducing the processing load on the user terminal 20. Examples of such a configuration include an HDD and an SSD. However, the storage device for storing the various types of information only needs to have a storage area accessible by the server 10, and may be configured to have a dedicated storage area outside the server 10, for example.

[0022] The server 10 may be configured by an information processing device such as a game processing server capable of rendering game images.

[0023] The user terminal 20 is managed by a user. Examples of the user terminal 20 include a mobile phone terminal, a smartphone, a PDA (Personal Digital Assistant), a personal computer, a tablet, a stationary game device, a portable game device, etc. The user terminal 20 may also be VR goggles, AR glasses, smart glasses, AR contacts, or other wearable devices that can be operated by the user.

[0024] The user terminal 20 may be a communication terminal capable of playing a network distribution type game.

[0025] The user terminal 20 is connected to the communication network 30 and includes hardware and software for executing various processes by communicating with the server 10. The multiple user terminals 20 may be configured to be able to communicate directly with each other without going through the server 10.

[0026] The user terminal 20 may have a built-in display device. The display device may be connected to the user terminal 20 wirelessly or via a wire. Since the display device is a very common configuration, it is not illustrated here. The display device is an example of an output device. When a user plays a game using the user terminal 20, the game screen is displayed as a composite image by the display device, and the user perceives this composite image. When a user performs various information processing using the user terminal 20, the screen required for the information processing (information processing screen) is displayed as an image by the display device, and the user perceives this image. The game screen or information processing screen is displayed, for example, on a display, which is an example of a display device provided in the user terminal, or a display, which is an example of a display device connected to the user terminal. Examples of display devices include a hologram display device capable of displaying holograms, a projection device that projects images (including the game screen or information processing screen) onto a screen, and a device that performs XR display. XR includes virtual reality (VR), augmented reality (AR), mixed reality (MR), and substitutional reality (SR).

[0027] The user terminal 20 includes a processor 21, a memory 22, and a storage device 23. The processor 21 is, for example, a central processing unit such as a CPU (Central Processing Unit) that performs various calculations and control. Furthermore, if the user terminal 20 includes a GPU (Graphics Processing Unit), some of the various calculations and control may be performed by the GPU. The user terminal 20 uses the data read into the memory 22 to execute various information processing operations with the processor 21, and stores the obtained processing results in the storage device 23 as necessary. The storage device 23 functions as a storage medium for storing various information.

[0028] The user terminal 20 may have a built-in input device. The input device may be connected to the user terminal 20 wirelessly or via a wire. The input device accepts user input. In response to the user input, a processor included in the server 10 or a processor included in the user terminal 20 executes various control processes. Examples of input devices include touch panel screens, keyboards, mice, gamepads, joysticks, and other controllers found on smartphones and tablets. A camera built into or connected to the user terminal 20 may also serve as an input device. The user performs input operations by gestures such as moving their hands in front of the camera (gesture input). The user terminal 20 may have a microphone. The user can perform voice input by speaking into the microphone in natural language. The headsets shown in FIGS. 4 and 5 serve as both an input device for voice input and an output device for voice output.

[0029] Additionally, the user terminal 20 may be equipped with other output devices such as a speaker, etc. The other output devices output sounds, vibrations, and various other types of information to the user.

[0030] 2 is a block diagram showing the configuration of a server corresponding to at least one embodiment of the present invention. The server 10 includes a user utterance information acquisition unit 101, a reaction control unit 102, and an utterance feature analysis unit 103. The processor included in the server 10 refers to a natural language reaction control program stored in a storage device and executes the program to functionally realize the user utterance information acquisition unit 101, the reaction control unit 102, and the utterance feature analysis unit 103. The natural language reaction control program is a program that controls the natural language reactions of objects to user utterances.

[0031] The user utterance information acquisition unit 101 has a function of acquiring utterance information, which is information indicating the content of an utterance made by a user. The reaction control unit 102 has a function of causing an object to react in accordance with the utterance information. The reaction control unit 102 controls the content of the reaction made by the object based on the start or end of an utterance made by the user. The utterance feature analysis unit 103 has a function of analyzing the features of how the user speaks.

[0032] FIG. 3 is a flowchart showing an example of processing of a natural language reaction control program corresponding to at least one of the embodiments of the present invention.

[0033] The user utterance information acquisition unit 101 acquires utterance information, which is information indicating the content of an utterance made by a user (St101). The reaction control unit 102 causes the object to react in accordance with the utterance information (St102). This reaction may include an utterance made by the object. In step St102, the reaction control unit 102 controls the content of the reaction made by the object based on the start or end of the user's utterance.

[0034] An object refers to a character such as a non-player character (NPC) placed in a virtual space. The object is not limited to a humanoid character, and may be, for example, a virtual ship object. The object may have a virtual body represented by a 2D model or a 3D model. The object may be a virtual object in a virtual space. The object may also be an object in real space whose reactions can be controlled.

[0035] The natural language reaction includes at least an utterance by the object. The user speaks, and the object speaks in response to the user's utterance. In other words, a dialogue takes place between the user and the object. The natural language reaction control program controls the object's reaction in this dialogue.

[0036] In addition to the above-mentioned speech, the reaction in natural language may include gestures such as hand and body movements, changes in facial expression, etc. The gestures, changes in facial expression, etc. may be expressed by animation of a 2D or 3D model associated with the object.

[0037] The user utterance information acquisition unit 101 acquires utterance information, which is information indicating the content of an utterance made by a user. For example, when a user speaks into a microphone provided in the user terminal 20, the user terminal 20 acquires the voice of the utterance and transmits the voice data to the server 10. The server 10 acquires the voice data. The server 10 may acquire the voice data in real time. The server 10 may acquire the voice data continuously.

[0038] The user utterance information acquisition unit 101 converts the acquired voice data into text information using a transcription function (Speech-to-Text function), thereby acquiring utterance information that indicates the content of the user's utterance.

[0039] In the above example, the voice data is transmitted from the user terminal 20 to the server 10 and then converted into text information on the server side. However, the voice data may be converted into text information on the user terminal 20 side, and the converted text information may be transmitted to the server 10.

[0040] The reaction control unit 102 uses the speech information to generate the content of a reaction in natural language to be performed by the object. The content of the reaction here means text information of the lines included in the reaction, command information specifying animation such as gestures and facial expressions performed by the object, etc.

[0041] For example, in the case of rule-based processing, predetermined keywords and information indicating the content of the reaction by an object when the predetermined keyword is included in the utterance information may be stored in advance as tabular data in the storage device 13. In this case, the reaction control unit 102 determines whether the utterance information includes the predetermined keyword, and if so, extracts information indicating the content of the reaction corresponding to the keyword from the tabular data.

[0042] The reaction control unit 102 may also use a language model. The language model may be, for example, a large language model (LLM). The language model may be stored in the storage device 13 of the server 10. Alternatively, the language model may be stored in a storage device on a cloud that is communicatively connected to the server 10 via the communication network 30, and the server 10 may access the language model via the network.

[0043] The reaction control unit 102 generates a prompt to be input to the language model based on the acquired utterance information. The utterance information may be stored as an utterance history in the storage device 13 or the like. The reaction control unit 102 may generate a prompt using information extracted from the utterance history as the utterance information.

[0044] The storage of the utterance history in the storage device may be performed for utterances by the user or for utterances by the object.

[0045] The reaction control unit 102 determines the content of the reaction by the object based on the output information of the language model obtained by inputting the generated prompt into the language model. The output information of the language model may include, for example, the content of the utterance by the object, i.e., text information of the dialogue. The output information of the language model may also include information for specifying gestures, facial expressions, etc. to be performed by the object.

[0046] The reaction control unit 102 controls the object using information indicating the content of the reaction identified as described above. For example, if text-format dialogue and information specifying animation such as gestures and facial expressions are generated as information indicating the content of the reaction, the reaction control unit 102 converts the text-format dialogue into audio data and outputs it. The reaction control unit 102 also executes animation using a character model of gestures, facial expressions, etc. The process of converting text into audio may be performed on the server 10 side or on the user terminal 20 side. The process of converting text into audio may be performed using a voice synthesis function (text-to-speech function).

[0047] Here, the user utterance information acquisition unit 101 may continuously acquire utterance information, which is information indicating the content of an utterance made by the user. By this continuous acquisition, the reaction control unit 102 detects the timing of the start or end of the user's utterance. Note that the user terminal 20 may detect the timing of the start or end of the user's utterance and transmit a message indicating the detection to the server 10. The reaction control unit 102 controls the content of the reaction made by the object based on the detected start or end of the user's utterance. An example of the control of the content of the reaction will be described in more detail later.

[0048] [Non-verbal Reactions in Case of Overlapping Speeches] When a user speaks while an object is speaking or preparing to speak, the reaction control unit 102 causes the object to perform an additional reaction. The additional reaction is a non-verbal reaction such as eye contact or animation.

[0049] Figure 4 is a conceptual diagram illustrating a natural language reaction of an object, according to at least one embodiment of the present invention; Figure 5 is a conceptual diagram illustrating a natural language reaction of an object, according to at least one embodiment of the present invention; and Figure 6 is a conceptual diagram illustrating a natural language reaction of an object, according to at least one embodiment of the present invention.

[0050] 4, an object OBJ is displayed on a display device DISP. The object OBJ is, for example, a 3D model of an avatar. The display device DISP is connected to a user terminal 20 so as to be able to communicate with the user terminal 20.

[0051] The user speaks using a microphone built into a headset connected to the user terminal 20. The object OBJ reacts in response to the user's utterance. In this example, under the control of the reaction control unit 102, the object OBJ reacts by saying, "I prefer thick ramen noodles. The reason is..."

[0052] Assume that the user simultaneously utters "By the way, I heard a new ramen shop opened yesterday" so as to overlap with the utterance of the object OBJ.

[0053] The user utterance information acquisition unit 101 continuously acquires utterance information, which is information indicating the content of utterances made by the user. Therefore, while the object OBJ is being made to speak, it detects that the user has started speaking. The reaction control unit 102 then causes the object OBJ to perform an additional reaction, such as a wink, as shown in FIG. 5 . The additional reaction is not a verbal utterance, but a non-verbal reaction such as eye contact, facial expression, or gesture. The additional reaction may convey to the user that the object OBJ (e.g., a character) has already recognized the user's interruption. In other words, the additional reaction conveys to the user through non-verbal communication, "I (the object OBJ) recognize that you (the user) have started speaking," and "Do you still want to continue?" This provides the user with an opportunity to consider whether to continue speaking even when there is overlapping speech, thereby facilitating a smooth dialogue.

[0054] The reaction control unit 102 may then determine whether to continue or interrupt the object's utterance, and control the object's reaction in accordance with the determination. For example, as shown in Fig. 6, the object OBJ may be allowed to continue uttering "The thick noodles have a great texture" after the object has been made to perform the non-verbal reaction described above. Conversely, the subsequent utterance as shown in Fig. 6 may not be performed. The determination of whether to continue or interrupt the object's utterance may be made based on the content of the user's utterance, etc.

[0055] The timing of the non-verbal additional reaction is not limited to when the user speaks while the object is making an utterance, as described above. For example, the reaction control unit 102 may cause the object to make the non-verbal additional reaction when the user speaks while the object is preparing to make an utterance.

[0056] "Preparing to speak" refers to the timing when the object has not yet spoken, but has already started internal processing to generate the content of the speech (for example, when using an LLM, a prompt is input into the LLM and the content of the speech is determined based on the output.) Since generating the content of the speech requires time, there is a time lag between the start of preparation for speech and the start of speech.

[0057] The reaction control unit 102 may cause the object to perform an additional non-verbal reaction when the object makes a normal utterance or when the object makes a filler utterance, which will be described later.

[0058] [Determining whether to continue or interrupt speech when speech overlaps] When speech by an object and a user overlap, it is generally considered that the conversation will be more comfortable for the user if the object interrupts its speech and listens attentively to what the user is saying. However, if the user's speech when the speech overlap occurs is an interjection such as "uh-huh" or "I see," interrupting the object's speech in response to the user's speech will make the conversation unnatural.

[0059] Therefore, when a user utters an utterance while an object is speaking or preparing to speak, the reaction control unit 102 may determine whether the content of the user utterance is a backchannel response. If the content of the user utterance is a backchannel response, the reaction control unit 102 continues the reaction by the object without interruption. If the content of the user utterance is not a backchannel response, the reaction control unit 102 interrupts the reaction by the object.

[0060] Whether the content of an utterance is a backchannel can be determined by, for example, registering backchannels such as "uh-huh" in advance in a database provided in the storage device 13 or the like, and determining that the utterance is a backchannel if there is a word that matches the registered word in the database. The determination may also be based on a threshold, such as determining that the utterance is a backchannel if the number of characters in the utterance is three or less.

[0061] When the content of the user utterance is not a backseat, the reaction control unit 102 may control the content of the object's reaction in accordance with the content of the user utterance, further depending on the content of the user utterance. The object's reaction is performed based on the user's utterance information continuously acquired by the user utterance information acquisition unit 101. The utterance information of the user utterance is added to the utterance information used to generate this reaction. That is, the reaction control unit 102 adds the user utterance determined not to be a backseat to the above-mentioned utterance history, and then regenerates the content of the reaction by referring to the utterance history.

[0062] For example, when inputting a prompt into a language model to output a line to be spoken by an object, the reaction control unit 102 inputs the generated prompt into the LLM by referring to the speech history.

[0063] When the content of the user utterance is not a backchannel response, the object's reaction is interrupted when an overlapping utterance occurs between the object and the user (including an overlapping utterance while the object is preparing to speak), and the user utterance was meaningful. If such a new user utterance is added to the utterance history and the object's reaction is regenerated using LLM or the like, the object will be able to react in a more contextual way.

[0064] To explain the interruption of speech in more detail, the interruption of speech by an object includes the following two patterns of interruption: (Interruption pattern A) Stopping the speech of an object and the animation accompanying the speech midway (Interruption pattern B) Canceling the speech of an object and the animation accompanying the speech before it is executed

[0065] Interruption pattern A is a pattern that occurs when the user makes an utterance other than a backchannel response while the object is speaking, and a reaction by the object occurs before and after the interruption.

[0066] Interruption pattern B occurs when the user makes an utterance other than a backchannel response while the object is preparing to speak or while making a filler utterance associated with the preparation. In this case, the object is prevented from starting to speak based on its previous utterance history. From the user's perspective, it appears as if the object never spoke, so there is no reaction from the object before or after the interruption, but it serves to facilitate the dialogue.

[0067] An example of processing for interruption pattern B is shown below. While the user is speaking continuously, the reaction control unit 102 causes the object to utter filler, and repeats the process of generating the content of the reaction and the process of canceling the generated reaction without executing it. When the user stops speaking, the reaction control unit 102 causes the object to perform a reaction following the filler. In other words, the reaction is not canceled. When this processing is performed, the user feels that the filler uttered by the object is a response from the object.

[0068] An example of a conversation based on the above processing example is shown below. User: "I like this." Object: "(Filler) Uh-huh" (Start of Reaction 1 generation process) User: "But I like that one too" Object: (Cancels Reaction 1 generation result "I like this kind of thing") Object: "(Filler) Seriously" (Start of Reaction 2 generation process) User: "Which do you think is better?" Object: (Cancels Reaction 2 generation result "That's surprising") Object: "(Filler) Umm" (Start of Reaction 3 generation process) Object: "(Reaction 3) I think this one is better"

[0069] [Match-up Response After Overlapping Speech] When an object's speech (including speech preparation) overlaps with a user's speech, the object's speech is interrupted as described above. It is assumed that the user will also interrupt their speech at the same time the object interrupts their speech. That is, a match-up state may occur in which both the object and the user interrupt their speech, assuming that the other will speak. In such a match-up state, prompting the user to make the next utterance can facilitate a smooth dialogue for the user. Therefore, if the content of the user's utterance is not a backchannel response, the reaction control unit 102 may interrupt the object's reaction and then determine whether the user has stopped speaking. If the user has stopped speaking, the reaction by the object to prompt the user to speak is controlled.

[0070] Controlling the reaction of an object to encourage the user to speak means making the object say something like, "Please speak." Also, making the object make facial expressions or gestures that encourage the user to speak also falls under controlling the reaction of an object to encourage the user to speak.

[0071] [Filler] As described above, the reaction control unit 102 controls the reactions of objects. The content of this reaction may not be determined immediately, for example, when using LLM. For example, if the process of generating a reaction takes a long time, a time lag occurs between when the user speaks and when the object reacts to the utterance, which may make the user feel that the dialogue is not proceeding smoothly. Therefore, it is necessary to ensure a smooth reaction generation time.

[0072] Therefore, the reaction control unit 102 may cause the object to utter a filler during the time until the object starts to react in accordance with the utterance information.

[0073] Fillers are words used to fill gaps in speech, such as "umm," "ah," etc. In other words, the reaction control unit 102 generates reactions by objects, but fills in gaps until the generation of the reactions is complete by having the objects utter fillers.

[0074] The filler that the reaction control unit 102 causes the object to speak may be based on the context of the dialogue. For example, the reaction control unit 102 may determine whether the content of the utterance information related to the user's most recent utterance is positive or negative. Note that a positive / negative determination process for determining whether the content of a certain text is positive or negative is known, and therefore conventional technology may be used. The filler that the object uses may differ depending on whether the content of the utterance information is positive or negative. This allows the object to speak a filler that matches the nuance of the user's utterance.

[0075] [Determining End of Utterance According to Characteristics of User Utterance] When an object and a user have a dialogue, the object basically waits for the user to finish speaking before reacting. Therefore, the reaction control unit 102 detects the timing when the user's utterance ends based on the user's utterance information acquired by the user utterance information acquisition unit 101. Then, the reaction control unit 102 controls the reaction of the object so that the object speaks after the user's utterance ends.

[0076] For example, the reaction control unit 102 may determine that the user's speech has ended when the user does not resume speaking even after a predetermined time has elapsed since the user's speech stopped.

[0077] Each user's speech has its own characteristics. For example, some users speak quickly, while others speak slowly and intermittently. As a result, it may happen that the object begins speaking even though the user is still speaking, as it is determined that the user has finished speaking. From the user's perspective, this feels as if the object has interrupted them while they were speaking, which can leave them dissatisfied and make the dialogue feel less smooth.

[0078] To avoid such a situation, the speech feature analysis unit 103 analyzes the features of how the user speaks. The reaction control unit 102 controls the reaction of the object so that the object starts reacting after a predetermined time has elapsed since the user's speech stopped. The reaction control unit 102 then changes the value of the predetermined time according to the analysis result of the speech feature analysis unit 103. For example, if the user is the type who speaks slowly, the predetermined time is set longer than usual.

[0079] The speech feature analysis unit 103 may analyze the features of how the user speaks by inputting the user utterance acquired by the user utterance information acquisition unit 101 into a trained model that has learned the features of the user utterance using the user utterance as input and information from which a predetermined time value can be derived as output. The trained model may be stored in the storage device 13 or the like, or may be stored in a storage device or cloud accessible from the server 10 via the communication network 30.

[0080] As described above, each embodiment of the present application solves one or more deficiencies. Note that the effects of each embodiment are non-limiting effects or examples of effects.

[0081] In each of the above-described embodiments, the user terminal 20 and the server 10 execute the above-described various processes in accordance with various control programs (e.g., a natural language reaction control program) stored in their own storage devices. Furthermore, other computers, not limited to the user terminal 20 and the server 10, may execute the above-described various processes in accordance with various control programs (e.g., a natural language reaction control program) stored in their own storage devices.

[0082] Furthermore, the configuration of the natural language reaction control system 1 is not limited to the configuration described as an example of the above-mentioned embodiment. For example, a configuration may be adopted in which a server executes some or all of the processes described as processes executed by a user terminal, or a configuration may be adopted in which a user terminal executes some or all of the processes described as processes executed by a server. Also, a configuration may be adopted in which the user terminal includes some or all of the storage unit (storage device) included in the server. In other words, a configuration may be adopted in which the user terminal or the server in the natural language reaction control system 1 includes some or all of the functions included in the other.

[0083] Furthermore, the program may be configured to cause a part or all of the functions described as examples of each of the above-mentioned embodiments to be realized by a single device that does not include a communication network.

[0084] [Additional Notes] The above-described embodiments have been described so that at least the following invention can be implemented by a person having ordinary skill in the art to which the invention pertains.

[0085] [1] A natural language reaction control program that controls a reaction in natural language by an object to a user's utterance, wherein the reaction includes at least an utterance by the object, and the program is configured to implement in a computer: a user utterance information acquisition function that acquires utterance information that is information indicating the content of the utterance made by the user; and a reaction control function that causes the object to react in accordance with the utterance information, wherein the reaction control function controls the content of the reaction by the object based on the start or end of the user's utterance.

[0086] The natural language reaction control program described above can reduce the difficulty and stress felt by a human user in a dialogue between the user and an object such as a character controlled by the system.

[0087] [2] The natural language reaction control program according to [1], wherein the reaction control function causes the object to perform an additional reaction when a user speaks while the object is speaking or preparing to speak, and the additional reaction is a non-verbal reaction.

[0088] According to the above-mentioned natural language reaction control program, when there is overlapping speech between an object and a user, the object can communicate to the user through non-verbal communication that it is aware of the overlapping speech.

[0089] [3] The natural language reaction control program described in [1], wherein the reaction control function, when a user speaks while the object is speaking or preparing to speak, determines whether the content of the user utterance is a backchannel response, and if the content of the user utterance is a backchannel response, continues the reaction by the object without interruption, and if the content of the user utterance is not a backchannel response, interrupts the reaction by the object.

[0090] According to the natural language reaction control program described above, when there is overlap in speech between an object and a user, the object should basically interrupt its speech and hand over control of the dialogue to the user, thereby realizing a smooth dialogue. However, this program can avoid an unnatural dialogue in which the object interrupts its speech even when the user responds with an interjection.

[0091] [4] The natural language reaction control program described in [3], wherein the reaction control function controls the content of the reaction by the object in accordance with the content of the user utterance when the content of the user utterance is not a back-channel response and the reaction by the object is interrupted.

[0092] According to the above-mentioned natural language reaction control program, if there is overlapping speech between an object and a user, and the user's speech is meaningful rather than just a nod, the object can be made to react in a way that takes into account the context of the user's speech.

[0093] [5] The natural language reaction control program described in [3], wherein the reaction control function, if the content of the user utterance is not a backchannel response, interrupts the reaction by the object, and then determines whether the user has stopped speaking, and if the user has stopped speaking, controls the reaction by the object to encourage the user to speak.

[0094] According to the natural language reaction control program described above, when a speech matchmaking state occurs after an overlap in speech occurs between an object and a user, the user can be prompted to take the initiative in the dialogue.

[0095] [6] The natural language reaction control program according to [1], wherein the reaction control function causes the object to utter a filler during a time until the object starts to react in response to the utterance information.

[0096] According to the natural language reaction control program described above, gaps may occur in the dialogue due to the processing time required to generate the content of the reaction by the object, but these gaps can be filled with filler.

[0097] [7] The natural language reaction control program described in [6], wherein the reaction control function determines whether the content of the speech information is positive or negative, and the filler that the object speaks varies depending on whether the content of the speech information is positive or negative.

[0098] According to the natural language reaction control program described above, it is possible to make the object utter fillers that match the nuance of the user's utterance.

[0099] [8] The natural language reaction control program according to [1], further comprising: making the computer realize a speech feature analysis function that analyzes the characteristics of how a user speaks; and in the reaction control function, controlling the reaction of the object so that the reaction of the object starts after a predetermined time has elapsed since the user's speech stopped; and changing the value of the predetermined time according to the analysis result of the speech feature analysis function.

[0100] According to the above-mentioned natural language reaction control program, by changing the time it takes for the user's speech to be determined to have ended after the speech has stopped, the timing at which the object's reaction begins can be appropriately adjusted according to the characteristics of each user's speech.

[0101] [9] The natural language reaction control program described in [8], wherein the speech feature analysis function inputs a user's speech and outputs information from which the value of the predetermined time can be derived, and inputs the user's speech acquired by the user speech information acquisition function into a trained model that has learned the features of the user's speech, thereby analyzing the features of how the user speaks.

[0102] According to the above-mentioned natural language reaction control program, the time after a user's speech stops at which it is determined that the speech has ended can be determined using a trained model trained using past speech data.

[0103]

[10] A natural language reaction control system that controls a reaction in natural language by an object to a user's utterance, wherein the reaction includes at least an utterance by the object, and the dialogue system includes a communication network, a server, and a user terminal, and includes: a user utterance information acquisition means that acquires utterance information that is information indicating the content of an utterance made by the user; and a reaction control means that causes the object to react in accordance with the utterance information, and the reaction control means controls the content of the reaction by the object based on the start or end of the user's utterance.

[0104] According to the natural language reaction control system described above, it is possible to reduce the difficulty and stress felt by a human user in a dialogue between the user and an object such as a character controlled by the system.

[0105]

[11] A computer-implemented natural language reaction control method for controlling a reaction in natural language by an object to a user's utterance, wherein the reaction includes at least an utterance by the object, and the method includes: a user utterance information acquisition process for acquiring utterance information that is information indicating the content of the utterance made by the user; and a reaction control process for making the object react according to the utterance information, wherein the reaction control process controls the content of the reaction by the object based on the start or end of the user's utterance.

[0106] According to the above-described natural language reaction control method, it is possible to reduce the difficulty and stress felt by a human user in a dialogue between the user and an object such as a character controlled by the system.

[0107] According to one embodiment of the present invention, the present invention is useful as a natural language reaction control program, a natural language reaction control system, and a natural language reaction control method that can reduce the difficulty and stress felt by a human user in a dialogue between the user and an object such as a character controlled by the system.

[0108] REFERENCE SIGNS LIST 1 Natural language reaction control system 10 Server 11 Processor 12 Memory 13 Storage device 20, 20A, 20B User terminal 21 Processor 22 Memory 23 Storage device 30 Communication network 101 User utterance information acquisition unit 102 Reaction control unit 103 Utterance feature analysis unit

Claims

1. A natural language reaction control program that controls a reaction in natural language by an object to a user's utterance, wherein the reaction includes at least an utterance by the object, and the program implements in a computer a user utterance information acquisition function that acquires utterance information, which is information indicating the content of the utterance made by the user, and a reaction control function that causes the object to react in accordance with the utterance information, and the reaction control function controls the content of the reaction by the object based on the start or end of the user's utterance.

2. The natural language reaction control program of claim 1, wherein the reaction control function causes the object to perform an additional reaction when a user utters something while the object is speaking or preparing to speak, and the additional reaction is a non-verbal reaction.

3. The natural language reaction control program of claim 1, wherein the reaction control function, when a user utters something while the object is speaking or preparing to speak, determines whether the content of the user utterance is a backchannel response, and if the content of the user utterance is a backchannel response, continues the reaction by the object without interruption, and if the content of the user utterance is not a backchannel response, interrupts the reaction by the object.

4. The natural language reaction control program of claim 3, wherein the reaction control function, if the content of the user utterance is not a backchannel response, interrupts the reaction by the object, and then determines whether the user has stopped speaking, and if the user has stopped speaking, controls the reaction by the object to encourage the user to speak.

5. The natural language reaction control program according to claim 1, wherein the reaction control function causes the object to utter a filler during the time until the object starts to react in response to the utterance information.

6. The natural language reaction control program of claim 1, further comprising: a speech feature analysis function in the computer that analyzes the characteristics of how a user speaks; and a reaction control function that controls the reaction of the object so that the object starts reacting after a predetermined time has elapsed since the user's speech stopped; and that changes the value of the predetermined time according to the analysis result of the speech feature analysis function.

7. A natural language reaction control system that controls a reaction in natural language by an object to a user's utterance, wherein the reaction includes at least an utterance by the object, the natural language reaction control system comprising a communication network, a server, and a user terminal, and including: a user utterance information acquisition means that acquires utterance information that is information indicating the content of an utterance made by the user; and a reaction control means that causes the object to react in accordance with the utterance information, wherein the reaction control means controls the content of the reaction by the object based on the start or end of the user's utterance.

8. A computer-implemented natural language reaction control method for controlling a reaction in natural language by an object to a user's utterance, wherein the reaction includes at least an utterance by the object, and the method includes: a user utterance information acquisition process for acquiring utterance information, which is information indicating the content of the utterance made by the user; and a reaction control process for causing the object to react in accordance with the utterance information, wherein the reaction control process controls the content of the reaction by the object based on the start or end of the user's utterance.

Images