Dialogue system and dialogue processing method

Abstract

A dialogue system for a vehicle may include: an input processor configured to receive a dialogue between occupants of the vehicle including a driver and at least one passenger, detect vehicle operation information, identify at least one passenger based on the dialogue between the occupants or the vehicle operation information, generate passenger number information estimating a change in the number of passengers in the vehicle when the vehicle arrives at a stop point based on the dialogue between the occupants, and obtain a pre-speech message based on the passenger number information; and a result processor configured to output the pre-speech based on the pre-speech message.

Description

Technical Field

[0001] Embodiments of the present disclosure generally relate to a dialogue system and a processing method, and more particularly, to a dialogue system configured to recognize a user's intention through a dialogue with the user to provide information or services required by the user, and a dialogue processing method. Background Art

[0002] Many conventional audio-video navigation (AVN) devices for vehicles have relatively small screens and buttons similar to mobile devices. Small screen buttons may cause inconvenience to users when providing visual information or receiving user input.

[0003] In particular, while driving, if a user takes his or her hands off the steering wheel or eyes off the road to operate a device or view information being displayed, there may be a serious risk of danger.

[0004] Therefore, when a dialogue system is implemented in a vehicle, the dialogue system should be able to recognize the user's intention by conversing with the user and provide the information or services required by the user in a more convenient and safer way. Summary of the invention

[0005] Therefore, one aspect of the present disclosure provides a dialogue system, a vehicle having the dialogue system, and a dialogue processing method, which are capable of providing services that meet the user's true intentions or services required by the user by accurately identifying the user's intentions based on various information such as dialogue with the user, vehicle status information, and user information.

[0006] Another aspect of the present disclosure provides a dialogue system, a vehicle having the dialogue system, and a dialogue processing method capable of identifying passengers based on a dialogue between occupants in a vehicle while driving and capable of managing changes in the number of passengers in the vehicle.

[0007] Additional aspects of the disclosure will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention.

[0008] According to an embodiment of the present disclosure, a dialogue system for a vehicle may include: an input processor configured to receive a dialogue between occupants of the vehicle including a driver and at least one passenger, detect vehicle operation information, identify at least one passenger based on the dialogue between the occupants or the vehicle operation information, generate passenger number information estimating a change in the number of passengers in the vehicle when the vehicle arrives at a stop point based on the dialogue between the occupants, and obtain a pre-speech message based on the passenger number information; and a result processor configured to output the pre-speech based on the pre-speech message.

[0009] The pre-speech message may indicate at least one of: a probability that each of the at least one passenger leaves the vehicle at the stop point, a probability that each of the at least one passenger gets on the vehicle again after leaving the vehicle at the stop point, and a probability that the potential passenger gets on the vehicle at the stop point.

[0010] The input processor may include a voice input processor configured to determine whether at least one passenger has boarded the vehicle based on one or more voice features of the voice of at least one passenger contained in a conversation between occupants; and a context information processor configured to determine whether the at least one passenger has boarded the vehicle based on vehicle operation information.

[0011] When it is determined that at least one passenger is boarding the vehicle, the input processor can obtain a pre-speech message corresponding to the at least one passenger boarding the vehicle, receive the speech of the at least one passenger related to the pre-speech message, and identify the at least one passenger by applying a natural language understanding algorithm to the speech of the at least one passenger, and when it is determined that at least one passenger is not boarding the vehicle, the input processor can obtain a pre-speech message corresponding to the at least one passenger not boarding the vehicle, receive the speech of the driver related to the pre-speech message, and verify the presence of the at least one passenger by applying the natural language understanding algorithm to the speech of the driver.

[0012] The input processor can determine the likelihood of each of at least one passenger leaving the vehicle at a stop point and the likelihood of each of at least one passenger getting on the vehicle again after leaving the vehicle at the stop point by applying a natural language understanding algorithm to conversations between occupants, and generate passenger quantity information based on the determined likelihood of at least one passenger leaving the vehicle at the stop point and the likelihood of at least one passenger getting on the vehicle again after leaving the vehicle at the stop point.

[0013] The input processor may receive a call session in the vehicle, determine a likelihood of a potential passenger boarding the vehicle at a stop by applying a natural language understanding algorithm to the received call session, and generate passenger quantity information based on the likelihood of the potential passenger boarding the vehicle at the stop.

[0014] After the vehicle leaves the stop point, the input processor can determine the change in the number of passengers in the vehicle based on the conversation between the occupants and the vehicle operation information, compare the estimated change in the number of passengers in the vehicle based on the passenger number information with the determined change in the number of passengers, and obtain a pre-speech message based on the comparison result.

[0015] After the vehicle leaves the stop, the input processor can obtain a pre-speech message to determine a change in the number of passengers in the vehicle, receive an utterance of at least one passenger related to the pre-speech message, and determine the change in the number of passengers in the vehicle by applying a natural language understanding algorithm to the utterance of the at least one passenger.

[0016] The dialogue system may further include a storage device configured to store travel-related information of the vehicle and passenger information of each of the at least one passenger when the vehicle stops traveling.

[0017] The passenger information may include at least one of passenger identification information, voice feature information, seat location information, boarding vehicle time information, boarding vehicle location information, departure time information, or information related to a location of departure from the vehicle.

[0018] The input processor can receive conversations between occupants and vehicle operation information, determine whether at least one passenger has boarded the vehicle based on the conversations between the occupants and the vehicle operation information, determine whether characteristics of each of the at least one passenger correspond to the passenger information, and obtain a pre-speech message by verifying whether a first passenger having characteristics corresponding to the passenger information participated in a previous trip.

[0019] The input processor can receive an utterance of at least one passenger related to a pre-utterance message, verify whether a first passenger participated in a previous trip by applying a natural language understanding algorithm to the passenger's utterance, and generate passenger quantity information based on a conversation between occupants and the passenger information when the first passenger participated in the previous trip.

[0020] In addition, according to an embodiment of the present disclosure, a conversation processing method for a vehicle may include: receiving a conversation between occupants of the vehicle including a driver and at least one passenger; detecting vehicle operation information; identifying at least one passenger based on the conversation between the occupants or the vehicle operation information; generating passenger number information estimating a change in the number of passengers in the vehicle when the vehicle arrives at a stop point based on the conversation between the occupants; obtaining a pre-speech message based on the passenger number information; and outputting a pre-speech based on the pre-speech message.

[0021] The pre-speech message may indicate at least one of: a probability that each of the at least one passenger leaves the vehicle at the stop point, a probability that each of the at least one passenger gets on the vehicle again after leaving the vehicle at the stop point, and a probability that the potential passenger gets on the vehicle at the stop point.

[0022] The conversation processing method may also include: determining whether at least one passenger has boarded the vehicle based on one or more voice features of the voice of at least one passenger contained in the conversation between occupants; and determining whether at least one passenger has boarded the vehicle based on vehicle operation information.

[0023] The conversation processing method may also include: when it is determined that at least one passenger is boarding the vehicle, obtaining a pre-speech message corresponding to the at least one passenger boarding the vehicle; receiving the speech of at least one passenger related to the pre-speech message; and identifying at least one passenger by applying a natural language understanding algorithm to the speech of the at least one passenger; and when it is determined that at least one passenger has not boarded the vehicle, obtaining a pre-speech message corresponding to the at least one passenger not boarding the vehicle; receiving the speech of the driver related to the pre-speech message; and verifying the presence of at least one passenger by applying a natural language understanding algorithm to the speech of the driver.

[0024] The conversation processing method may also include: determining the possibility of each of at least one passenger leaving the vehicle at a stop point and the possibility of each of at least one passenger getting on the vehicle again after leaving the vehicle at the stop point by applying a natural language understanding algorithm to the conversation between passengers; and generating passenger quantity information based on the determined possibility of at least one passenger leaving the vehicle at a stop point and the possibility of at least one passenger getting on the vehicle again after leaving the vehicle at the stop point.

[0025] The conversation processing method may also include: receiving a call conversation in a vehicle; determining the likelihood of a potential passenger boarding the vehicle at a stop point by applying a natural language understanding algorithm to the received call conversation; and generating passenger quantity information based on the likelihood of the potential passenger boarding the vehicle at the stop point.

[0026] The conversation processing method may also include: after the vehicle leaves a stop point, determining a change in the number of passengers in the vehicle based on conversations between occupants and vehicle operation information; comparing an estimated change in the number of passengers in the vehicle based on the passenger number information with the determined change in the number of passengers; and obtaining a pre-speech message based on the comparison result.

[0027] The conversation processing method may also include: after the vehicle leaves a stop point, obtaining a pre-speech message to determine a change in the number of passengers in the vehicle; receiving an utterance of at least one passenger related to the pre-speech message; and determining a change in the number of passengers in the vehicle by applying a natural language understanding algorithm to the utterance of at least one passenger.

[0028] The conversation processing method may further include storing travel related information of the vehicle and passenger information of each of the at least one passenger when the vehicle stops traveling.

[0029] The travel-related information may include at least one of a departure point, a stop point, and a destination of the travel.

[0030] The passenger information may include at least one of passenger identification information, voice feature information, seat location information, boarding vehicle time information, boarding vehicle location information, departure time information, or information related to a location of departure from the vehicle.

[0031] The conversation processing method may also include: receiving conversations between passengers and vehicle operation information; determining whether at least one passenger has boarded the vehicle based on the conversations between passengers and the vehicle operation information; determining whether characteristics of each of the at least one passenger correspond to the passenger information; and obtaining a pre-speech message by verifying whether a first passenger having characteristics corresponding to the passenger information participated in a previous trip.

[0032] The conversation processing method may also include: receiving the speech of at least one passenger related to the pre-speech message; verifying whether the first passenger participated in a previous trip by applying a natural language understanding algorithm to the passenger's speech; and generating passenger quantity information based on the conversation between the occupants and the passenger information when the first passenger participated in the previous trip. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] These and / or other aspects of the present disclosure will become apparent and more easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0034] Figure 1 A control block diagram showing a dialog system according to an embodiment of the present disclosure;

[0035] Figure 2A To show a view of the interior of a vehicle;

[0036] Figure 2B To show that from Figure 2A A view of the interior of the vehicle from a different angle;

[0037] Figures 3 to 6 A diagram showing an example of a dialogue generated between a dialogue system and a driver;

[0038] Figure 7 and Figure 8 To illustrate a diagram of a dialog system configured to estimate a change in the number of passengers and output a pre-utterance;

[0039] Fig. 9 and Fig.10 A control block diagram schematically showing the connections between the dialog system and vehicle components;

[0040] Fig.11 and Fig.12 A control block diagram schematically showing connections between components of a dialog system and vehicle components;

[0041] Fig.13A control block diagram for illustrating a vehicle-individual mode of setting a dialogue system in a vehicle;

[0042] Fig.14 and Fig.15 A control block diagram showing a vehicle gateway approach, in which a dialog system is provided in a remote server and a vehicle serves as a gateway to connect a user to the dialog system;

[0043] Fig.16 A control block diagram showing a case where a vehicle can perform input processing and output processing in a vehicle gateway mode;

[0044] Fig.17 A control block diagram showing a hybrid method in which both the remote dialogue system server and the vehicle perform dialogue processing;

[0045] Fig.18 and Fig.19 A control block diagram showing a mobile gateway method in which a mobile device connected to a vehicle connects a user to a remote dialogue system server;

[0046] Fig. 20 A control block diagram for illustrating a mobile individual mode of setting a dialog system in a mobile device;

[0047] Fig.21 , Fig.22A and Fig. 22B A control block diagram showing in detail the configuration of an input processor in the configuration of a dialog system;

[0048] Fig.23A , Fig. 23B and Fig.23C A diagram showing an example of information stored in a context understanding table;

[0049] Fig.24 A control block diagram for illustrating a dialog system suitable for a case where the dialog system first outputs an utterance before receiving a user input;

[0050] Fig.25A , Fig.25B , Fig.25C and Fig.25D A view showing an example of information stored in a pre-utterance condition table;

[0051] Fig.26 A control block diagram illustrating in detail the configuration of the dialog manager;

[0052] Fig. 27 is a diagram showing an example of information stored in a relationship action DB;

[0053] Fig.28 is a view showing an example of information stored in the action execution condition DB;

[0054] Fig.29 is a view showing an example of information stored in an action parameter DB;

[0055] Fig.30 is a table showing an example of information stored in the fuzzy resolution information DB;

[0056] Fig.31A and Fig.31B A table showing various examples of vehicle control performed as a result of the fuzzy resolver resolving ambiguity by referring to the fuzzy resolution information DB and extracting actions;

[0057] Fig.32 A control block diagram showing in detail the configuration of the result processor;

[0058] Figures 33 to 45 A diagram showing a specific example in which, when a user inputs an utterance related to route guidance, a dialog system processes the input, manages the dialog, and outputs a result;

[0059] Fig.46 A flowchart showing a method for processing user input in a dialog processing method according to an embodiment;

[0060] Fig.47 A flowchart showing a method for managing a dialog using an output of an input processor in a dialog processing method according to an embodiment;

[0061] Fig.48 A flowchart showing a result processing method for generating a response corresponding to a result of dialog management in a dialog processing method according to an embodiment;

[0062] Figures 49 to 51 A flowchart showing a case where a dialogue system outputs a pre-utterance before inputting an utterance in a dialogue processing method according to an embodiment;

[0063] Fig.52 A flowchart showing a method for processing a repeated task when a dialog system outputs a pre-utterance before a user inputs an utterance in a dialog processing method according to an embodiment;

[0064] Fig.53 A flowchart showing a method for determining that a passenger is boarding a vehicle and outputting a pre-speech in a dialogue processing method according to an embodiment;

[0065] Fig.54 A flowchart showing a method for estimating a change in the number of passengers and outputting a pre-utterance in a dialogue processing method according to an embodiment; and

[0066] Fig.55The flowchart is a method for determining that a passenger who participated in a previous trip has boarded a vehicle and outputting a pre-utterance in a dialogue processing method according to an embodiment.

[0067] It should be understood that the above drawings are not necessarily drawn to scale, and they present slightly simplified representations of various preferred features illustrating the basic principles of the present disclosure. The specific design features of the present disclosure, including, for example, specific dimensions, orientations, locations, and shapes will be determined in part by the specific intended application and use environment.

[0068] Description of Reference Numerals

[0069] 100: Dialogue System

[0070] 110: Input processor

[0071] 120: Dialogue Processor

[0072] 130: Result Processor

[0073] 200: Vehicles

[0074] 210: Voice input device

[0075] 220: Information input device other than voice

[0076] 230: Dialogue output device

[0077] 280: Communication device DETAILED DESCRIPTION

[0078] Hereinafter, the embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. It will be appreciated by those skilled in the art that the described embodiments may be modified in various ways without departing from the spirit or scope of the present disclosure. In the following description, the same reference numerals refer to the same elements throughout the specification.

[0079] Well-known functions or structures are not described in detail because they would obscure one or more exemplary embodiments with unnecessary detail. Terms such as "unit", "module", "component" and "block" can be implemented as hardware or software. Depending on the embodiment, multiple "units", "modules", "components" and "blocks" can be implemented as a single component or a single "unit", "module", "component" and "block" can include multiple components.

[0080] It should be understood that when an element is referred to as being “connected to” another element, it may be directly or indirectly connected to the other element, wherein the indirect connection includes “connection via a wireless communication network”.

[0081] Furthermore, when a component “includes” or “comprises” an element, unless there is a specific description contrary thereto, the component may further include other elements, not excluding other elements.

[0082] As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0083] The reference numerals are used to facilitate the description, but are not intended to indicate the order of each step. Unless the context clearly indicates otherwise, each step can be implemented in an order different from the order shown.

[0084] It should be understood that the term "automobile" or "vehicle" or other similar terms used herein include general motor vehicles, such as passenger cars including sports utility vehicles (SUVs), buses, trucks, various commercial vehicles, watercraft including various ships, aircraft, etc., and include hybrid vehicles, electric vehicles, plug-in hybrid electric vehicles, hydrogen-powered vehicles and other alternative fuel vehicles (e.g., fuels from resources other than petroleum). As mentioned herein, a hybrid vehicle is a vehicle with two or more power sources, such as a gasoline-powered vehicle and an electric vehicle.

[0085] In addition, it should be understood that one or more of the following methods or aspects thereof can be performed by at least one controller. The term "controller" can refer to a hardware device including a memory and a processor. The memory is configured to store program instructions, and the processor is specifically programmed to execute the program instructions to perform one or more processes further described below. As described herein, a control unit can control the operation of a unit, module, component, device, etc. In addition, it should be understood that the following method can be performed by a device including a controller in combination with one or more other components, as will be understood by those of ordinary skill in the art.

[0086] In addition, the controller of the present disclosure may be implemented as a non-transitory computer-readable medium containing executable program instructions executed by a processor. Examples of computer-readable media include, but are not limited to, ROM, RAM, compact disk (CD)-ROM, magnetic tape, floppy disk, flash drive, smart card, and optical data storage device. The computer-readable recording medium may also be distributed throughout a computer network so that program instructions are stored and executed in a distributed manner, such as through a telematics server or a controller area network (CAN).

[0087] Reference will now be made in detail to the embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings.

[0088] According to an embodiment, the dialogue system may be configured to recognize the user's intention by using the user's voice and another input other than the voice, and is configured to provide a service appropriate or required for the user's intention. The dialogue system may perform a dialogue with the user by outputting a system discourse, which is a means configured to provide a service or clearly recognize the user's intention.

[0089] As an example, the dialogue system recognizes the content of the dialogue between the occupants in the vehicle by using voice recognition, identifies the passengers based on the content of the dialogue between the occupants in the vehicle, and determines the possibility of each passenger leaving the vehicle and the possibility of each passenger getting on the vehicle again after leaving. The dialogue system can conduct a dialogue with the passengers by outputting system utterances as a means for notifying the driver of the possibility of each passenger leaving the vehicle and the possibility of each passenger getting on the vehicle again after leaving the vehicle.

[0090] At this time, the dialogue system may conduct a dialogue with the user by outputting a system utterance in response to a user's request, or may conduct a dialogue with the user by outputting a pre-utterance without a user's request.

[0091] The pre-speech described below means a speech output without a user request, and the speech may include a speech output when an immediate response is required when the vehicle arrives at a stop or destination. In addition, the pre-speech may include a speech output when information transmission is required despite no user request by acquiring and analyzing user information.

[0092] The pre-utterance may include an utterance output when information transmission is required by receiving various information from external devices such as a vehicle, a user terminal, and an external server.

[0093] Meanwhile, pre-discourse is not limited to system discourse that is output despite no user request. Pre-discourse may include situations where discourse needs to be output within a period of time or when a specific condition occurs because the user's request is not immediate. For example, when the output of discourse is postponed because the user is on a call despite a user request to output discourse after a period of time or despite an upcoming requested time, the dialogue system selects an appropriate time and outputs discourse at the selected time. In the following, for ease of description, if it is not necessary to distinguish between pre-discourse or system discourse output in response to a user request, they are collectively referred to as discourse.

[0094] The user or occupant in the vehicle described below includes all objects boarding the vehicle. For example, the user or occupant in the vehicle (the person riding in the vehicle) includes not only the driver but also the passengers (i.e., fellow passengers), and specifically, the user or occupant in the vehicle is a general term for the person sitting in the driver's seat, the passenger seat, and the passengers in the back seat. According to an embodiment, the service provided to the user may include all types of operations according to the needs of the user or the intention of the user, wherein all types of operations may include providing information, controlling the vehicle, performing audio / video / navigation functions, and providing content from an external server.

[0095] According to one embodiment, the dialog system provides a dialog processing technology dedicated to a vehicle environment so as to accurately recognize the user's intention in a special environment, namely, a vehicle.

[0096] The gateway for connecting the dialog system with the user can be a vehicle or a mobile device connected to the vehicle. As described below, the dialog system can be set in a remote server in the vehicle or outside the vehicle to send or receive data through communication with the vehicle or a mobile device connected to the vehicle.

[0097] Some components of the dialogue system may be arranged in the vehicle, and some components may be arranged in the remote server. Therefore, the vehicle and the remote server may perform a part of the operation of the dialogue system.

[0098] Figure 1 The figure is a control block diagram showing a dialog system according to an embodiment of the present disclosure.

[0099] refer to Figure 1 The dialogue system 100 may include: an input processor 110, which processes user input including user voice and input other than user voice or input including vehicle-related information or user-related information; a dialogue manager 120, which uses the processing result of the input processor 110 to recognize the user's intention and the vehicle state, and determines an action corresponding to the user's intention or the vehicle state; a result processor 130, which provides a specific service or outputs a system discourse for continuing the dialogue according to the output result of the dialogue manager 120; and a storage device 140, which stores various information for operations described later.

[0100] The input processor 110 may receive two types of input, such as user voice and input other than voice. Input other than voice may include input other than voice of the user input by recognizing the user's gesture or the operation of the input device, vehicle state information indicating the vehicle state, driving environment information related to the driving information of the vehicle, and user information indicating the state of the user. In addition, in addition to the above information, any information related to the user and the vehicle may be input to the input processor 110 as long as the information is used to recognize the user's intention or to provide services to the user or the vehicle. Users may include drivers and passengers.

[0101] The input processor 110 converts the user's voice into a text-type utterance by recognizing the user's voice, and recognizes the user's intention by applying a natural language understanding algorithm to the user utterance.

[0102] The input processor 110 can determine whether the passenger has boarded the vehicle by recognizing the passenger's voice as input and the input other than the user's voice, wherein the input other than the user's voice is input through the information input device other than the voice. The dialogue system 100 can perform a pre-speech to request the identification information of the passenger to identify each passenger determined to board the vehicle. The dialogue system 100 can identify each passenger by receiving the speech of each passenger. The identification information of the passenger can represent each passenger determined to board the vehicle based on the identification information.

[0103] The input processor 110 collects information related to the vehicle state or the driving environment of the vehicle other than the user's voice, and then uses the collected information to understand the context.

[0104] The input processor 110 sends the user intention obtained through the natural language understanding technology and the context-related information to the dialogue manager 120 .

[0105] The dialog manager 120 determines an action corresponding to the user's intention or the current context based on information related to the user's intention and context transmitted from the input processor 110 , and manages parameters required to perform the corresponding action.

[0106] According to an embodiment, an action may represent various actions for providing a specific service, and the type of action may be predetermined. Providing a service may correspond to performing an action as needed.

[0107] For example, the domain / action inference rule DB 141 (refer to Fig.22A) and the actions corresponding to the user's utterance, that is, the actions the user wants, can be extracted according to the stored inference rules. Actions related to events occurring in the vehicle can be predefined and then stored in the relational action DB 146b (refer to Fig.24 ).

[0108] There is no limitation on the type of action. If the dialogue system 100 allows an action to be performed via the vehicle 200 or the mobile device 400, and the action is predefined and its inference rule or relationship with other actions / events is stored, the action can become the above-mentioned action.

[0109] The dialog manager 120 sends information related to the determined action to the result processor 130 .

[0110] The result processor 130 generates and outputs the dialogue response and command required to execute the sent action. The dialogue response can be output in text, image or audio type. When the command is output, services such as vehicle control and external content provision corresponding to the output command can be executed.

[0111] For example, the result processor 130 may generate and output a dialogue response and command for identifying a passenger. As an example, the result processor 130 may output an utterance requesting identification information of a passenger to identify a passenger determined to board a vehicle by the input processor 110 .

[0112] In addition, the result processor 130 can generate and output a dialogue response and command for estimating the change in the number of passengers. As an example, the result processor 130 can output a pre-speech with content related to the possibility of each passenger leaving the vehicle at a stop point and the possibility of each passenger getting on the vehicle again after leaving the vehicle.

[0113] It should be understood that the input processor 110 and the result processor 130 can be implemented as separate processors, or alternatively, as a single processor. It should also be understood that the input processor 110 and the result processor 130 can be implemented as one or more hardware processors, one or more software modules, or a combination thereof. It should also be understood that the operation of the input processor 110 and the result processor 130 can be controlled by the vehicle controller 240.

[0114] The storage device 140 stores various information used for dialogue processing and service provision. For example, the storage device 140 may pre-store information related to domains, actions, voice behaviors, and entity names for natural language understanding, and a context understanding table for understanding context from input information. In addition, the storage device 140 may pre-store data detected by sensors provided in the vehicle, information related to the user, and information required for the action.

[0115] For example, when the travel of the vehicle is terminated, the storage device 140 may store travel-related information about the travel of the vehicle and passenger information about passengers who boarded the vehicle during the travel. Specifically, the storage device 140 may store travel-related information about the travel of the vehicle, such as the departure point, stop point, and destination of the travel, and passenger information about passengers, such as personal identification information, voice feature information, seat position information, boarding vehicle time information, departure time information, boarding vehicle position information, and information about the position of leaving the vehicle. A description of the information stored in the storage device 140 will be described later.

[0116] As described above, the dialogue system 100 provides a dialogue processing technology dedicated to the vehicle environment. All or some components of the dialogue system 100 may be included in the vehicle. The dialogue system 100 may be set in a remote server, and the vehicle may act as a gateway between the dialogue system 100 and the user. In either case, the dialogue system 100 may be connected to the user via the vehicle or a mobile device connected to the vehicle.

[0117] Figure 2A A view showing the interior of a vehicle, and Figure 2B To show that when Figure 2A A view of the vehicle interior from a different angle.

[0118] refer to Figure 2A A display 231 configured to display screens required for vehicle control including audio functions, video functions, navigation functions, and call functions, and an input button 221 configured to receive a user's control command may be disposed in a center fascia 203 corresponding to a center portion of a dashboard 201 inside the vehicle 200.

[0119] For user's operational convenience, input buttons may be provided in the steering wheel 207, and a jog shuttle 225 serving as the input buttons may be provided in the center console area 202 provided between the driver's seat 254a and the passenger seat 254b.

[0120] Meanwhile, the seat 254 is not limited to the driver's seat 254a and the passenger seat 254b. Figure 2B The vehicle 1 may be provided with rear seats 254c and 254d as required.

[0121] At this time, when the passenger gets on the front passenger seat 254b and the back seats 254c and 254d, and when a conversation is conducted between the occupants in the vehicle, the conversation system can determine whether the passenger gets on the vehicle through voice recognition, and determine the seat position of the passenger in the vehicle by estimating the position where the voice is generated. In addition, the conversation system can determine whether the passenger is located in the corresponding seat through various sensors provided on the side of the front passenger seat 254b and the back seats 254c and 254d.

[0122] According to one embodiment, the dialogue system may output a pre-speech to request identification information of the passenger to identify each passenger determined to board the vehicle, and identify each passenger by receiving the speech of each passenger. As described above, identification of the passenger may mean distinguishing passengers based on identification information of the passenger determined to board the vehicle.

[0123] According to one embodiment, the dialogue system may store passenger information including the seat position of each passenger in the vehicle. A detailed description thereof will be described later.

[0124] A module including the display 231, the input button 221, and a processor controlling various functions may correspond to an audio video navigation (AVN) terminal or a head unit.

[0125] The display 231 may be implemented by any of various display devices, for example, a liquid crystal display (LCD), a light emitting diode (LED), a plasma display panel (PDP), an organic light emitting diode (OLED), and a cathode ray tube (CRT).

[0126] 2, the input button 221 may be provided as a hard key type on an area adjacent to the display 231. Alternatively, when the display 231 is implemented by a touch screen, the display 231 may perform the function of the input button 221.

[0127] The vehicle 200 may receive a user control command as voice via the voice input device 210. The voice input device 210 may include a microphone configured to receive sound and then convert the sound into an electrical signal.

[0128] As shown in FIG. 2 , for effective voice input, the voice input device 210 can be mounted to the top plate 205, but the embodiment of the vehicle 200 is not limited thereto. Therefore, the voice input device 210 can be mounted on the dashboard 201 or the steering wheel 207. In addition, the voice input device 210 can be respectively arranged in the co-pilot seat 254b and the back seats 254c and 254d, for inputting the voice of the passengers sitting on the co-pilot seat 254b and the back seats 254c and 254d. In addition, the voice input device 210 can be mounted to any position as long as the position is suitable for receiving the user's voice.

[0129] Inside the vehicle 200, a speaker 232 may be provided, which is configured to have a conversation with the user or to output a sound required to provide a service desired by the user. For example, the speaker 232 may be provided inside the driver's seat door 253a and the passenger seat door 253b.

[0130] The speaker 232 may output voice for navigation route guidance, sound or voice contained in audio and video content, voice for providing information or services desired by the user, and system voice generated as a response to the user's voice.

[0131] According to an embodiment, the dialogue system 100 provides services suitable for the user's lifestyle by using a dialogue processing technology suitable for a vehicle environment, and the dialogue system 100 can implement new services using technologies such as connected cars, the Internet of Things (IoT), and artificial intelligence (AI).

[0132] When a dialog processing technology suitable for a vehicle environment is applied, such as the dialog system 100 according to the embodiment, key contexts can be easily recognized and responded to during the driver's direct driving of the vehicle. Services can be provided by applying weights to parameters that affect driving, such as gasoline shortage and drowsy driving, or information such as travel time and destination information can be easily obtained based on the conditions under which the vehicle moves to the destination in most cases, which is required for the service.

[0133] In addition, intelligent services configured to provide functions can be easily implemented by recognizing the driver's intention. This is because real-time information and actions are prioritized in the case of direct driving by the driver. For example, when the driver searches for a gas station while driving, it can be interpreted as the driver's intention to go to the gas station. However, when the driver searches for a gas station in a place other than the vehicle, in addition to the driver's intention to go to the gas station, it can be interpreted as another intention such as searching for location information query, phone number query, and price query.

[0134] In addition, although the vehicle has a limited space, various situations may occur therein. For example, the driver may utilize the dialogue system 100 in various situations, such as driving a vehicle with an unfamiliar interface such as a rental car, using a designated driver service, vehicle management situations such as a car wash, situations with a baby in the vehicle, and situations of visiting a special destination.

[0135] In addition, various services and dialogue situations may occur in each stage constituting vehicle driving and the stages before and after driving (e.g., vehicle inspection stage, start preparation stage, driving stage, and parking stage). Specifically, the driver can utilize the dialogue system 100 in various situations, such as a situation where the driver does not know how to deal with a problem, a situation where the vehicle is associated with various external devices, a situation where driving habits are checked, such as gas mileage, and a situation where safety support functions are used, such as smart cruise control, a navigation operation situation, a drowsy driving situation, a situation where the same route is driven every day, and a situation where parking is checked at a place.

[0136] Figures 3 to 6 is a diagram showing an example of a dialogue generated between a dialogue system and a driver.

[0137] refer to Figure 3 Although the driver does not input words for inquiring about the current remaining gasoline amount or for requesting guidance from a gas station, the dialogue system 100 can identify the current remaining gasoline by itself and when the identified remaining gasoline is less than a predetermined value, the dialogue system 100 can first output words providing information related to the current remaining gasoline (S1: 43 km can be traveled with the remaining gasoline).

[0138] In response to the utterance, the driver may input an utterance requesting to receive route guidance to a nearby gas station (U1: Let me know the nearby gas station), and the dialogue system 100 may output an utterance providing information about the gas station closest to the current location (S2: The nearby gas stations are A-Oil Seong-rim gas station, B-oil Jang-dae gas station, and C-oil Pacific gas station).

[0139] The driver may additionally input an utterance asking about gasoline prices (U2: Where is the cheapest?), and the dialogue system 100 may output an utterance providing information related to the prices of the fuel types (the lowest gasoline price is at B oil Jang-dae gas station, 1,294 won per liter, and the lowest diesel price is at A oil Seong-rim gas station, 985 won per liter).

[0140] The driver may input an utterance requesting guidance to B oil Jang-dae gas station (U3), and the dialogue system 100 may output an utterance instructing to start guidance to the gas station selected by the driver (S4: start route to B oil Jang-dae gas station).

[0141] That is, the dialogue system 100 can determine that the currently required service is a gasoline guidance service based on the state information of the vehicle received via the input processor 110, and output a pre-speech to provide the required service. In addition, the driver can be guided to the nearest gas station that sells the fuel type of the current vehicle at the lowest price through a dialogue with the dialogue system 100. According to one embodiment, it is assumed that the "pre-speech" refers to the speech that is first output from the dialogue system 100 before the user speaks it.

[0142] At the same time, when Figure 3 In the example shown, when selecting a gas station, the dialogue system 100 can omit some questions and directly provide information, thereby reducing the steps and time of the dialogue.

[0143] For example, the dialogue system 100 may pre-identify that the fuel type of the current vehicle is gasoline, and the driver's criterion for selecting a gas station is price. Information related to the fuel type of the vehicle may be obtained from the vehicle, and the driver's criterion for selecting a gas station may be pre-input from the driver, or the driver's criterion for selecting a gas station may be obtained by learning the driver's dialogue history or gas station selection history. This information may be pre-stored in the storage device 140.

[0144] In this case, if Figure 4 As shown, the dialogue system 100 can actively output utterances providing information related to fuel prices, in particular, the price of gasoline as the fuel type of the current vehicle (S2+S3=S3'), without inputting the driver's utterance (U2) requesting information about fuel prices, i.e., omitting U2.

[0145] The driver can omit the speech (U2) for requesting information about fuel prices, and can form a response of the dialogue system 100 so that the speech (S2) for guiding the nearest gas station and the speech (S3) for guiding the fuel price are integrated as a single response to reduce the steps and time of the dialogue.

[0146] In addition, the dialogue system 100 can recognize on its own that the driver's intention is to search for a gas station based on the fact that the driver inquires about the current remaining amount of gasoline.

[0147] In this case, if Figure 5As shown, although the driver did not input an utterance (U1) asking about the nearest gas station, that is, U1 was omitted, the dialogue system 100 can actively output an utterance (S2+S3=S3") providing information related to fuel prices.

[0148] In a state where the nearest gas station from the current location and the gas station offering the lowest fuel price are the same gas station, the utterance (S3") providing information related to the fuel price may include a question for asking whether to be guided to the corresponding gas station. Therefore, the user can request route guidance to the corresponding gas station by simply inputting an utterance (U3': yes) agreeing to the question of the dialogue system 100, without inputting a specific utterance for requesting guidance to a certain gas station.

[0149] As described above, the dialogue system 100 can identify the user's true intention based on the pre-acquired information by considering the content that the user has not uttered and actively provide information corresponding to the intention. Therefore, the dialogue steps and time for providing the service desired by the user can be reduced.

[0150] refer to Figure 6 , when it is determined based on the conversation between occupants in the vehicle and the vehicle operation information that a passenger has boarded the vehicle, the dialogue system 100 may first output a speech asking for identification information of the passenger (S11: Who are you? Tell me your name).

[0151] In response to the utterance, the passenger may input an utterance including identification information of the passenger (U11: I am OO), and the dialogue system 100 may recognize the identity of the passenger through the utterance of the passenger.

[0152] That is, the dialogue system 100 can determine whether a passenger has boarded the vehicle based on the dialogue between passengers and the vehicle operation information received through the input processor 110, and output a pre-speech to identify the passenger. In addition, the passenger can provide identification information to the dialogue system 100 through a dialogue with the dialogue system 100.

[0153] Figure 7 and Figure 8 To illustrate a diagram of a dialog system, the dialog system is configured to estimate a change in the number of passengers and output a pre-utterance.

[0154] refer to Figure 7 , after recognizing the identities of the passengers, the dialogue system 100 in the vehicle 200 may estimate a change in the number of passengers based on a dialogue between the passengers in the vehicle and output a pre-utterance indicating a result of estimating the change in the number of passengers.

[0155] Specifically, by applying the natural language understanding algorithm to the conversation between the passengers in the vehicle, the dialogue system 100 in the vehicle 200 can obtain the possibility of each passenger leaving the vehicle at the stop point 700 and the possibility of boarding the vehicle again after leaving the vehicle. In addition, by applying the natural language understanding algorithm to the call conversation in the vehicle, the dialogue system 100 in the vehicle 200 can obtain the possibility of passengers (i.e., potential passengers) who intend to board the vehicle at the stop point 700. A detailed description thereof will be described.

[0156] Before arriving at the stop point 700, the dialogue system 100 in the vehicle 200 may output pre-speech related to the result of estimating the change in the number of passengers, such as "A will leave at the stop point", "B will leave the vehicle at the stop point and then get on the vehicle again", "C will not leave at the stop point", and "D will get on the vehicle at the stop point", based on the possibility of each passenger leaving the vehicle and the possibility of each passenger getting on the vehicle again after leaving.

[0157] refer to Figure 8 , after leaving the stop point 700, the dialogue system 100 in the vehicle 200 may output a pre-utterance related to a comparison result between an estimated result of a change in the number of passengers estimated before arriving at the stop point 700 and a result of a change in the number of passengers after leaving the stop point 700.

[0158] Specifically, after leaving the stop point 700 , the dialogue system 100 in the vehicle 200 may compare the estimation result of the change in the number of passengers acquired before arriving at the stop point 700 and the change result of the number of passengers after leaving the stop point 700 .

[0159] The comparison between the estimated result of the change in the number of passengers obtained before arriving at the stop point 700 and the result of the change in the number of passengers obtained after leaving the stop point 700 can be performed based on the conversation between the occupants in the vehicle after leaving the stop point 700 and the vehicle operation information by determining whether the passengers board the vehicle after leaving the stop point 700.

[0160] In addition, after leaving the stop point 700, the dialogue system 100 in the vehicle 200 may output a preliminary utterance for verifying whether the estimated result of the change in the number of passengers is correct, so as to verify the result of the change in the number of passengers. The dialogue system 100 in the vehicle 200 may compare the estimated result of the change in the number of passengers obtained before arriving at the stop point 700 and the result of the change in the number of passengers after leaving the stop point by using the occupant's response utterance to the preliminary utterance for verifying whether the estimated result of the change in the number of passengers is correct.

[0161] After leaving the stop point 700, the dialogue system 100 in the vehicle 200 can output a pre-discourse about the comparison result "the current number of passengers is different from the estimated result of the change in the number of passengers" based on a comparison between the estimated result of the change in the number of passengers obtained before arriving at the stop point 700 and the result of the change in the number of passengers after leaving the stop point 700. Fig. 9 and Fig.10 A control block diagram schematically illustrates the connections between the dialog system and vehicle components.

[0162] refer to Fig. 9 , the user's voice input to the dialogue system 100 may be input via a voice input device 210 provided in the vehicle 200. As shown in FIG. 2 , the voice input device 210 may include a microphone provided inside the vehicle 200.

[0163] The input other than voice among the user inputs may be input through the information input device other than voice 220. The information input device other than voice 220 may include input buttons 221 and 223 and a rotary shuttle 225 for receiving commands through user's operation.

[0164] The information input device 220 other than voice may include a camera for photographing the user. Through the image imaged by the camera, the user's gesture, expression, or sight direction used as a means of command input may be recognized. Alternatively, the user's state (sleepy state, etc.) may be recognized through the image imaged by the camera.

[0165] In addition, the information input device 220 other than voice may include window adjustment buttons, seat adjustment buttons and air conditioning adjustment buttons arranged on the side of the co-pilot seat 254b and the rear seats 254c and 254d to determine the passengers boarding the vehicle and the seat positions of the passengers.

[0166] The vehicle-related information may be input into the dialogue system 100 via the vehicle controller 240. The vehicle-related information may include vehicle state information or surrounding environment information acquired by various sensors provided in the vehicle 200, and information originally stored in the vehicle 200, such as the fuel type of the vehicle.

[0167] The dialogue system 100 can recognize the user's intention and context using the user's voice input via the voice input device 210, input other than the user's voice via the information input device other than voice 220, and various inputs via the vehicle controller 240. The dialogue system 100 outputs a response to perform an action corresponding to the user's intention.

[0168] The dialogue output device 230 is a device configured to provide output to the speaker in a visual, auditory or tactile manner. The dialogue output device 230 may include a display 231 and a speaker 232 provided in the vehicle 200. The display 231 and the speaker 232 may output a response to the user's speech, information about the user's question or the user's request in a visual or auditory manner. In addition, vibration may be output by installing a vibrator in the steering wheel 207.

[0169] Furthermore, according to the response output from the dialogue system 100 , the vehicle controller 240 may control the vehicle 200 to perform an action corresponding to the user's intention or the current situation.

[0170] At the same time, in addition to the information obtained by the sensors installed in the vehicle 200, the vehicle 200 can also collect information obtained from the external content server 300 or the external device via the communication device 280, for example, driving environment information and user information such as traffic conditions, weather, temperature, passenger information and driver personal information, and then the vehicle 200 can send this information to the dialogue system 100.

[0171] like Fig.10 As shown, information obtained by sensors installed in the vehicle 200, such as remaining fuel, rainfall, rainfall speed, surrounding obstacle information, speed, engine temperature, tire pressure, and current location, can be input into the dialogue system 100 via the internal signal controller 241.

[0172] In addition, information acquired by sensors provided in the vehicle, such as window adjustment button operation information, seat adjustment button operation information, and air-conditioning adjustment button operation information, may be input to the dialogue system 100 through the interior signal controller 241 .

[0173] Driving environment information acquired from the outside via vehicle-to-everything (V2X) communication may be input to the dialogue system 100 via the external signal controller 242. V2X may mean that vehicles exchange and share various useful information, such as traffic conditions, by communicating with road infrastructure and other vehicles while driving.

[0174] V2X communication may include vehicle-to-infrastructure (V2I) communication, vehicle-to-vehicle (V2V) communication, and vehicle-to-mobile device (V2N) communication. Thus, by using V2X communication, information such as traffic information about the front or the approach of another vehicle or the risk of collision with another vehicle may be transmitted and received through communication directly performed between vehicles or communication with infrastructure installed on the road, so that the driver may be notified of the information.

[0175] Therefore, the driving environment information input to the dialogue system 100 via the external signal controller 242 may include traffic information about the front, approach information of adjacent vehicles, collision warning with another vehicle, real-time traffic conditions, unexpected conditions, and traffic flow control status.

[0176] Although not shown in the drawings, a signal obtained via V2X may also be input to the vehicle 200 via the communication device 280 .

[0177] The vehicle controller 240 may include: a memory in which a program for performing the above-mentioned operations and operations described later is stored; and a processor for executing the stored program. At least one memory and one processor may be provided, and when multiple memories and processors are provided, they may be integrated on one chip or physically separated.

[0178] In addition, the internal signal controller 241 and the external signal controller 242 may be implemented by the same processor and memory, or by separate processors and memories.

[0179] Fig.11 and Fig.12 A control block diagram schematically illustrates the connections between the dialog system and vehicle components.

[0180] refer to Fig.11 , the user voice sent from the voice input device 210 can be input to the voice input processor 111 set in the input processor 110, and the input other than the user voice sent by the information input device 220 other than voice can be input to the context information processor 112 set in the input processor 110.

[0181] In addition, information input via the internal signal controller 241 or the external signal controller 242 is input to the context information processor 112 provided in the input processor 110 .

[0182] The context information input to the context information processor 112 may include vehicle state information, driving environment information, and user information input from the information input device 220 other than voice and the vehicle controller 240. The context information processor 112 may recognize the context based on the input context information. The dialogue system 100 may accurately recognize the user's intention or effectively find out the service required by the user by recognizing the context.

[0183] For example, the voice input processor 111 may determine that a passenger is boarding the vehicle based on a conversation between passengers in the vehicle identified by the voice input device 210. In addition, the voice input processor 111 may determine the possibility of each passenger leaving the vehicle at a stop point and the possibility of each passenger boarding the vehicle again after leaving at a stop point based on a conversation between passengers in the vehicle identified by the voice input device 210. In addition, the voice input processor 111 may estimate passengers who intend to board the vehicle based on a call conversation in the vehicle identified by the voice input device 210. The context information processor 112 may recognize the operation of the information input device 220 other than voice, and determine whether a passenger is boarding the vehicle based on the recognition result. The response output from the result processor 130 may be input to the dialogue output device 230 or the vehicle controller 240 to allow the vehicle 200 to provide the service required by the user. In addition, the response may be sent to the external content server 300 to request the required service.

[0184] The vehicle state information, driving environment information, and user information transmitted from the vehicle controller 240 may be stored in the storage device 140 .

[0185] refer to Fig.12 , the storage device 140 may include a long-term memory 143 and a short-term memory 144. Data stored in the storage device 140 may be classified into a short-term memory and a long-term memory according to the importance and persistence of the data and the intention of the designer.

[0186] The short-term memory 144 can store previously executed dialogues. The previous dialogue can be a dialogue executed within a reference time from the current time. Alternatively, the dialogue can be stored continuously until the volume of the speech content between the user and the dialogue system 100 reaches a reference value.

[0187] For example, when it is meal time, the vehicle 200 may output a word asking whether to guide to a restaurant via the speaker 232. Whether it is meal time may be identified based on whether the current time is within the predetermined meal time range. When the user says the content of "Let me know restaurants near Gangnam Station" or the content of "Let me know restaurants" and when the current position of the vehicle 200 is near Gangnam Station, the dialogue system 100 may search for restaurants near Gangnam Station through the external content server 300, and then provide the user with information related to the searched restaurants near Gangnam Station. As an example of providing information, the dialogue system 100 may display a list of restaurants on the display 231, and when the user says "first", dialogue content related to a request for a restaurant and a selection of a restaurant may be stored in the short-term memory 144.

[0188] Alternatively, not only the entire conversation content is stored, but also the specific information contained in the conversation content can be stored. For example, the first restaurant in the restaurant list can be stored in the short-term memory 144 or the long-term memory 143 as the restaurant selected by the user.

[0189] When the user asks the dialogue system 100 "How's the weather?" after a conversation about restaurants near Gangnam Station, the dialogue system 100 can assume from the conversation stored in the short-term memory 144 that the user's location of interest is Gangnam Station, and then output a response "It's raining at Gangnam Station."

[0190] Next, when the user says "recommend a restaurant menu", the dialogue system 100 may assume from the dialogue stored in the short-term memory that "the restaurant" means a restaurant near Gangnam Station, and acquire information related to the recommended menu of the corresponding restaurant through the service provided from the external content server 300. Therefore, the dialogue system 100 may output a response "noodles are the best menu in the restaurant".

[0191] The long-term storage 143 may store data according to the persistence of the data. For example, the long-term storage 143 may determine that the persistence of data (such as telephone numbers of relatives and friends and information of locations of interest (POIs), such as home or company, and user preferences for certain parameters) can be guaranteed, and then store the data in the long-term storage. On the contrary, when it is determined that the persistence of the data is not guaranteed, the data may be stored in the short-term storage 144.

[0192] For example, when the travel of the vehicle is terminated, the long-term memory 143 may store travel-related information about the travel of the vehicle and passenger information of passengers boarding the vehicle during the travel. Specifically, the long-term memory 143 may store travel-related information about the travel of the vehicle, such as the departure point, stop point, and destination of the travel, and passenger information about passengers, such as personal identification information, voice feature information, seat position information, boarding vehicle time information, departure time information, boarding vehicle position information, and information related to the location of leaving the vehicle. For example, the current location of the user may be temporary data and thus stored in the short-term memory 144, and the user's preference for restaurants may be persistent data that may be available later and thus stored in the long-term memory 143.

[0193] When the user says "Is there any restaurant here?", the dialogue system 100 can identify the user's current location from the short-term memory 144 and indicate from the long-term memory 143 that the user likes Chinese restaurants. Therefore, the dialogue system 100 can recommend a list of the user's favorite Chinese restaurants around the current location by using external content.

[0194] In addition, the dialog system 100 may use the data stored in the long-term memory 143 and the short-term memory 144 to proactively provide services and information to the user.

[0195] For example, information related to the user's residence may be stored in the long-term storage 143. The dialog system 100 may acquire information related to the user's residence from the external content server 300 and then provide information indicating that "water outage is expected this Friday due to cleaning of the apartment."

[0196] In addition, when the same passengers as those in the previous trip ride in the next trip, the dialogue system 100 can estimate the possibility of each passenger leaving the vehicle and the possibility of boarding the vehicle again after leaving based on the stored passenger information and the dialogue between the occupants in the vehicle.

[0197] Information related to the vehicle battery status may be stored in the short-term memory 144. The dialog system 100 may analyze the vehicle battery status stored in the short-term memory 144 and then provide information indicating “The battery is in a bad condition. Fix it before winter.”

[0198] Fig.13 A control block diagram showing a vehicle-individual mode in which a dialogue system is provided in a vehicle.

[0199] like Fig.13 As shown, according to the vehicle-only mode, the dialogue system 100 having the input processor 110 , the dialogue manager 120 , the result processor 130 , and the storage device 140 may be included in the vehicle 200 .

[0200] When the dialogue system 100 is included in the vehicle 200, the vehicle 200 can process the dialogue with the user by itself and provide the service required by the user. However, information required for the dialogue processing and service provision may be obtained from the external content server 300.

[0201] Vehicle status information or driving environment information detected by the vehicle detector 260 , such as remaining fuel amount, rainfall amount, rainfall speed, surrounding obstacle information, speed, engine temperature, tire pressure, and current location, can be input into the dialogue system 100 via the vehicle controller 240 .

[0202] According to the response output from the dialogue system 100 , the vehicle controller 240 may control an air conditioning device 251 , a window 252 , a door 253 , a seat 254 , or an AVN 255 provided in the vehicle 200 .

[0203] For example, when the dialogue system 100 determines that the user's intention or the service required by the user is to lower the temperature inside the vehicle 200 and then generates and outputs a corresponding command, the vehicle controller 240 can lower the temperature inside the vehicle 200 by controlling the air conditioner 251.

[0204] For another example, when the dialogue system 100 determines that the user's intention or the service required by the user is to raise the driver's seat window 252a and generates and outputs a corresponding command, the vehicle controller 240 can raise the driver's seat window 252a by controlling the window 252.

[0205] For another example, when the dialogue system 100 determines that the user's intention or the service required by the user is to guide a route to a certain destination and generates and outputs a corresponding command, the vehicle controller 240 can perform route guidance by controlling the AVN 255. If necessary, the communication device 280 can obtain map data and POI information from the external content server 300 and then use the information to provide services.

[0206] Fig.14 and Fig.15 A control block diagram showing a vehicle gateway approach, in which a dialog system is provided in a remote server and a vehicle serves as a gateway to connect a user to the dialog system.

[0207] like Fig.14 As shown, according to the vehicle gateway method, the remote dialogue system server 1 can be provided outside the vehicle 200, and the dialogue system client 270 connected to the remote dialogue system server 1 via the communication device 280 can be set in the vehicle 200. The communication device 280 serves as a gateway for connecting the vehicle 200 and the remote dialogue system server 1.

[0208] The dialog system client 270 may serve as an interface connected to an input / output device and perform collection as well as transmission and reception of data.

[0209] When the voice input device 210 and the information input device 220 other than voice provided in the vehicle 200 receive user input and transmit the user input to the dialogue system client 270 , the dialogue system client 270 may transmit the input data to the remote dialogue system server 1 via the communication device 280 .

[0210] The vehicle controller 240 may also transmit the data detected by the vehicle detector 260 to the dialogue system client 270 , and the dialogue system client 270 may transmit the data detected by the vehicle detector 260 to the remote dialogue system server 1 via the communication device 280 .

[0211] Since the above-mentioned dialogue system 100 is provided in the remote dialogue system server 1 , input data processing, dialogue processing based on the result of the input data processing, and result processing based on the result of the dialogue processing can all be executed by the remote dialogue system server 1 .

[0212] In addition, the remote dialogue system server 1 can acquire information or content required for input data processing, dialogue management, or result processing from the external content server 300 .

[0213] According to the response transmitted from the remote dialogue system server 1 , the vehicle 200 can acquire information or content of a service desired by the user from the external content server 300 .

[0214] refer to Fig.15 The communication device 280 may include at least one communication module configured to communicate with an external device. For example, the communication device 280 may include at least one of a short-range communication module 281, a wired communication module 282, and a wireless communication module 283.

[0215] The short-range communication module 281 may include various short-range communication modules (e.g., a Bluetooth module, an infrared communication module, a radio frequency identification (RFID) communication module, a wireless local area network (WLAN) communication module, an NFC communication module, and a ZigBee communication module), which are configured to send and receive signals using the short-range wireless communication module.

[0216] The wired communication module 282 may include various wired communication modules, such as a local area network (LAN) module, a wide area network (WAN) module, or a value-added network (VAN) module, and various cable communication modules, such as a universal serial bus (USB), a high-definition multimedia interface (HDMI), a digital video interface (DVI), a recommended standard 232 (RS-232), power line communication, or a plain old telephone service (POTS).

[0217] The wireless communication module 283 may include a wireless communication module that supports various wireless communication methods, such as a Wifi module, a wireless broadband module, a global system for mobile communications (GSM), code division multiple access (CDMA), wideband code division multiple access (WCDMA), universal mobile telecommunications system (UMTS), time division multiple access (TDMA), long term evolution (LTE), 4G and 5G.

[0218] In addition, the communication device 280 may further include an internal communication module (not shown) for communication between electronic devices in the vehicle 200. The communication protocol of the vehicle 200 may use a controller area network (CAN), a local interconnect network (LIN), FlexRay, and Ethernet.

[0219] The dialogue system 100 can transmit and receive data to and from the external content server 300 or the remote dialogue system server 1 via the wireless communication module 283. The dialogue system 100 can perform V2X communication using the wireless communication module 283. In addition, by using the short-range communication module 281 or the wired communication module 282, the dialogue system 100 can transmit and receive data to and from a mobile device connected to the vehicle 200.

[0220] Fig.16 This is a control block diagram showing a case where a vehicle can perform input processing and output processing in a vehicle gateway mode.

[0221] As described above, the dialog system client 270 of the vehicle 200 may only collect and send and receive data, but when Fig.16 As shown, when the input processor 271, the result processor 273 and the storage device 274 are included in the dialogue system client 270, the dialogue system client 270 can also process data input from the user or the vehicle or perform processing related to determining the provision of services required by the user. That is, the operations of the input processor 110 and the result processor 130 can be performed not only by the remote dialogue system server 1 but also by the vehicle 200.

[0222] In this case, the dialog system client 270 may perform all or some operations of the input processor 110. The dialog system client 270 may perform all or some operations of the result processor 130.

[0223] The sharing of tasks between the remote dialogue system server 1 and the dialogue system client 270 may be determined in consideration of the volume of data to be processed and the data processing speed.

[0224] Fig.17 This is a control block diagram showing a hybrid method in which both the remote dialogue system server and the vehicle execute dialogue processing.

[0225] like Fig.17 As shown, according to the hybrid method, since the input processor 110, the dialogue manager 120, the result processor 130 and the storage device 140 are set in the remote dialogue system server 1, the remote dialogue system server 1 can perform dialogue processing, and since the terminal dialogue system 290 provided with the input processor 291, the dialogue manager 292, the result processor 293 and the storage device 294 is set in the vehicle 200, the vehicle 200 can also perform dialogue processing.

[0226] However, there may be a difference in capacity or performance between the processor and memory provided in the vehicle 200 and the processor or memory provided in the remote dialogue system server 1. Therefore, when the terminal dialogue system 290 is able to output a result by processing all input data and managing a dialogue, the terminal dialogue system 290 may perform the entire process. Otherwise, the remote dialogue system server 1 may be requested to process.

[0227] Before executing the dialogue process, the terminal dialogue system 290 may determine whether the dialogue process can be executed based on the data type, and the terminal dialogue system 290 may directly execute the process or request the remote dialogue system server 1 for the process based on the result of the determination.

[0228] When an event that the terminal dialogue system 290 cannot execute occurs during dialogue processing, the terminal dialogue system 290 can request processing from the remote dialogue system server 1 and send the result of its own processing to the remote dialogue system server 1.

[0229] For example, when high-performance computing power or long-term data processing is required, the remote dialogue system server 1 can perform dialogue processing, and when real-time processing is required, the terminal dialogue system 290 can perform dialogue processing. For example, when a situation occurs that requires immediate processing and therefore requires processing of data before synchronization, it can be set so that the terminal dialogue system 290 processes the data first.

[0230] In addition, when there is an unregistered speaker in the vehicle and thus user confirmation is required, the remote dialogue system server 1 can process the dialogue. That is, when it is necessary to identify the passenger because a new passenger is in the vehicle, the remote dialogue system server 1 can process the dialogue.

[0231] Furthermore, when the terminal dialogue system 290 cannot complete the dialogue process by itself in a state where it cannot connect to the remote dialogue system server 1 via the communication device 280, it can notify the user via the dialogue output device 230 that the dialogue process cannot be executed.

[0232] The data stored in the terminal dialogue system 290 and the data stored in the remote dialogue system server 1 may be determined according to the data type or data capacity. For example, in the case of data with a risk of privacy violation due to personal identification, the data may be stored in the storage device 294 of the terminal dialogue system 290. In addition, a large amount of data may be stored in the storage device 140 of the remote dialogue system server 1, and a small amount of data may be stored in the storage device 294 of the terminal dialogue system 290. Alternatively, a small amount of data may be stored in the storage device 140 of the remote dialogue system server 1 and the storage device 294 of the terminal dialogue system 290.

[0233] Fig.18 and Fig.19 A control block diagram showing a mobile gateway method in which a mobile device connected to a vehicle connects a user to a remote dialogue system server.

[0234] like Fig.18 As shown, according to the mobile gateway method, the mobile device 400 can receive vehicle status information and driving environment information from the vehicle 200, and send the user input and the vehicle status information to the remote dialogue system server 1. That is, the mobile device 400 can act as a gateway to connect the user to the remote dialogue system server 1 or to connect the vehicle 200 to the remote dialogue system server 1.

[0235] The mobile device 400 may be represented as an electronic device that is portable and capable of sending and receiving data to and from an external server and a vehicle by communicating with the external server and the vehicle, wherein the mobile device 400 may include a smart phone, a smart watch, smart glasses, a PDA, and a tablet computer.

[0236] The mobile device 400 may include a voice input device 410 for receiving a user's voice, an input device 420 other than the voice input device 410 for receiving input other than the user's voice, an output device 430 for outputting a response in a visual, auditory or tactile manner, a communication device 480 for sending and receiving data to and from the remote dialogue system server 1 and the vehicle 200 through communication, and a dialogue system client 470 that collects input data from the vehicle 200 and the user and sends the data to the remote dialogue system server 1 via the communication device 480.

[0237] The voice input device 410 may include a microphone that receives sound, converts the sound into an electric signal, and outputs the electric signal.

[0238] The input device 420 other than the voice input device 410 may include an input button, a touch screen, or a camera provided in the mobile device 400 .

[0239] The output device 430 may include a display, a speaker, or a vibrator provided in the mobile device 400 .

[0240] The voice input device 410, the input device 420 and the output device 430 other than the voice input device 410 provided in the mobile device 400 can be used as the input and output interface of the user. In addition, the voice input device 210, the information input device 220 other than the voice, and the dialogue output device 230 provided in the vehicle 200 can be used as the input and output interface of the user.

[0241] When the vehicle 200 transmits data detected by the vehicle detector 260 and a user input to the mobile device 400 , the dialogue system client 470 of the mobile device 400 may transmit the data and the user input to the remote dialogue system server 1 .

[0242] The dialogue system client 470 can transmit the response or command transmitted from the remote dialogue system server 1 to the vehicle 200. When the dialogue system client 470 uses the dialogue output device 230 provided in the vehicle 200 as an input and output interface for the user, the speech of the dialogue system 100 or the response to the speech of the user can be output via the dialogue output device 230. When the dialogue system client 470 uses the output device 430 provided in the mobile device 400, the speech of the dialogue system 100 or the response to the speech of the user can be output via the output device 430.

[0243] A command for vehicle control may be transmitted to the vehicle 200 , and the vehicle controller 240 may perform control corresponding to the transmitted command, thereby providing a service required by a user.

[0244] The dialog system client 470 may collect input data and transmit the input data to the remote dialog system server 1. The dialog system client 470 may also execute all or some functions of the input processor 110 and the result processor 130 of the dialog system 100.

[0245] refer to Fig.19 The communication device 480 of the mobile device 400 may include at least one communication module configured to communicate with an external device. For example, the communication device 480 may include at least one of a short-range communication module 481, a wired communication module 482, and a wireless communication module 483.

[0246] The short-range communication module 481 may include various short-range communication modules (e.g., a Bluetooth module, an infrared communication module, a radio frequency identification (RFID) communication module, a wireless local area network (WLAN) communication module, an NFC communication module, and a ZigBee communication module), which is configured to send and receive signals using the short-range wireless communication module.

[0247] The wired communication module 482 may include various wired communication modules, such as a local area network (LAN) module, a wide area network (WAN) module, or a value-added network (VAN) module, and various cable communication modules, such as a universal serial bus (USB), a high-definition multimedia interface (HDMI), a digital video interface (DVI), a recommended standard 232 (RS-232), power line communication, or a plain old telephone service (POTS).

[0248] The wireless communication module 483 may include a wireless communication module that supports various wireless communication methods, such as a Wifi module, a wireless broadband module, a global system for mobile communications (GSM), code division multiple access (CDMA), wideband code division multiple access (WCDMA), universal mobile telecommunications system (UMTS), time division multiple access (TDMA), long term evolution (LTE), 4G and 5G.

[0249] For example, the mobile device 400 may be connected to the vehicle 200 via the short-range communication module 481 or the wired communication module 482 , and the mobile device 400 may be connected to the remote dialogue system server 1 or the external content server 300 via the wireless communication module 483 .

[0250] Fig. 20 A control block diagram showing the setting of the mobile individual mode of the dialogue system in the mobile device.

[0251] like Fig. 20 As shown, the dialogue system 100 may be provided in the mobile device 400 according to the mobile standalone mode.

[0252] Therefore, the mobile device 400 can process the dialogue with the user and provide the service required by the user by itself without connecting to the remote dialogue system server 1 for dialogue processing. However, the mobile device 400 can obtain part of the information used for dialogue processing and service provision from the external content server 300.

[0253] According to any of the above-described methods, the components forming the dialogue system 100 may be physically separated from each other, or some components may be omitted. For example, even when the dialogue system 100 is provided in the remote dialogue system server 1, some components forming the dialogue system 100 may be provided in a separate server or vehicle. The operator or administrator of the separate server may be the same as or different from the operator or administrator of the remote dialogue system server 1. For example, a speech recognizer or a natural language understanding part described later may be provided in a separate server, and the dialogue system 100 may receive the result of speech recognition or the result of natural language understanding about the user's speech from the separate server. Alternatively, the storage device 140 may be provided in a separate server.

[0254] A detailed configuration and a description of a detailed operation of each component of the dialogue system 100 will be described in detail. According to the embodiment described later, for convenience of explanation, it is assumed that the dialogue system 100 is set in the vehicle 200. Specific components of the dialogue system 100 described later may be classified according to their operations, and there may be no limitation on whether the components are implemented by the same processor and memory and the physical locations of the processor and memory.

[0255] Fig.21 , Fig.22A and Fig. 22B This is a control block diagram showing in detail the configuration of an input processor in the configuration of the dialog system.

[0256] refer to Fig.21 The input processor 110 may include a voice input processor 111 for processing voice input and a context information processor 112 for processing context information.

[0257] The user voice input through the voice input device 210 may be transmitted to the voice input processor 111 , and input other than the user voice input through the information other than voice input device 220 may be transmitted to the context information processor 112 .

[0258] The vehicle controller 240 may transmit vehicle state information, driving environment information, and user information to the context information processor 112. The driving environment information and user information may be provided from an external content server 300 or a mobile device 400 connected to the vehicle 200.

[0259] Inputs other than voice may be included in the context information. That is, the context information may include vehicle state information, driving environment information, user information, passenger boarding information, stop point arrival information, and stop point departure information. Passenger boarding information may correspond to context information indicating whether a passenger has boarded a vehicle, stop point arrival information may correspond to context information indicating whether a vehicle has arrived at a stop point, and stop point departure information may correspond to context information indicating whether a vehicle has left a stop point after arriving at the stop point.

[0260] The dialogue system 100 may acquire the passenger boarding vehicle information based on the information on the passenger acquired by recognizing the dialogue between the occupants in the vehicle through the voice input device 210 and the information on the passenger acquired through the information input device 220 other than voice.

[0261] In addition, the dialogue system 100 can determine whether the vehicle arrives at or leaves the stop point based on vehicle state information such as the current position and vehicle speed detected by the vehicle detector 260, and the dialogue system 100 can acquire the stop point arrival information and the stop point departure information.

[0262] The vehicle state information may include information indicating a vehicle state and acquired by a sensor provided in the vehicle 200 , and information related to the vehicle and stored in the vehicle, such as a fuel type of the vehicle.

[0263] The driving environment information may be information acquired by sensors disposed in the vehicle 200. The driving environment information may include image information acquired by a front camera, a rear camera, or a stereo camera, obstacle information acquired by a sensor (e.g., a radar, a laser radar, an ultrasonic sensor), and information related to rainfall and rain speed acquired by a rain sensor.

[0264] The driving environment information may also include traffic status information, traffic light information, and adjacent vehicle approach or adjacent vehicle collision risk information obtained via V2X.

[0265] The user information may include information related to the user's status measured by a camera or a biometric reader provided in the vehicle, information related to the user directly input by the user using an input device provided in the vehicle, information related to the user stored in the external content server 300, and information stored in the mobile device 400 connected to the vehicle.

[0266] The voice input processor 111 may include: a voice recognizer 111a, which outputs text-type utterances by recognizing the input user's voice; a natural language understanding part 111b, which recognizes the user's intention contained in the utterance by applying natural language understanding technology to the user's utterance; and a dialogue input manager 111c, which sends the results of natural language understanding and context information to the dialogue manager 120.

[0267] The voice recognizer 111 a may include a voice recognition engine, and the voice recognition engine may recognize a voice spoken by a user and generate a recognition result by applying a voice recognition algorithm to an input voice.

[0268] Since the input speech is converted into a more useful form for speech recognition, the speech recognizer 111a can detect the actual speech portion included in the speech by detecting the start and end points from the speech signal. This is called end point detection (EPD).

[0269] The speech recognizer 111a may obtain a feature vector of the input speech from the detected portion by applying a feature vector extraction technique, such as Cepstrum, Linear Prediction Coefficient (LPC), Mel-Frequency Cepstrum Coefficient (MFCC), or filter bank energy.

[0270] The speech recognizer 111a can obtain the recognition result by comparing the extracted feature vector with the trained reference pattern. At this time, the speech recognizer 111a can use an acoustic model that models and compares the signal characteristics of the speech, and a language model that models the language order relationship of the words or syllables corresponding to the recognition vocabulary. To this end, the storage device 140 can store the acoustic model and the language model DB.

[0271] The acoustic model can be classified into a direct comparison method of setting a recognition object as a feature vector model and comparing the feature vector model with a feature vector of a speech signal, and a statistical method of statistically processing the feature vector of the recognition object.

[0272] The direct comparison method is to set a unit such as a word or phoneme as a recognition object to a feature vector model, and compare the received speech with the feature vector model to determine the similarity between them. A representative example of the direct comparison method is vector quantization. Vector quantization is to map the feature vector of the received speech signal to a codebook as a reference model to encode the mapping result into a representative value, and compare the representative values ​​with each other.

[0273] The statistical model method is to configure the unit of the recognition object into a state sequence and use the relationship between the state sequences. Each state sequence can be configured with multiple nodes. The method using the relationship between state sequences can be classified into dynamic time warping (DTW), hidden Markov model (HMM) and methods using neural networks.

[0274] DTW is a method that takes into account the dynamic characteristics of speech that the length of the signal changes over time even if the same person speaks the same pronunciation, and compensates for the difference on the time axis by comparing with a reference model. HMM is a recognition method that assumes speech to be a Markov process with a state transition probability and an observation probability of a node (output symbol) in each state, and then estimates the state transition probability and observation probability of the node based on the learning data and generates the probability of the received speech from the estimated model calculation.

[0275] At the same time, by applying the sequential relationship between the units constituting the language to the units acquired through speech recognition, the language model that models the language sequential relationship of words, syllables, etc. can reduce acoustic ambiguity and recognition errors. Language models can include statistical language models and models based on finite state automata (FSA). Statistical language models use chain probabilities of words, such as Unigram, Bigram, and Trigram.

[0276] The speech recognizer 111a can use any of the above methods to perform speech recognition. For example, the speech recognizer 111a can use an acoustic model using HMM, or an N-best search method combining an acoustic model with a speech model. The N-best search method can improve recognition performance by selecting N or fewer recognition result candidates using an acoustic model and a language model, and then re-evaluating the ranking of the recognition result candidates.

[0277] The speech recognizer 111a can calculate a confidence value to ensure the reliability of the recognition result. The confidence value can be a standard for indicating how reliable the speech recognition result is. For example, the confidence value can be defined as a relative value of the probability of emitting the corresponding phoneme or word from different phonemes or words for the phoneme or word as the recognition result. Therefore, the confidence value can be expressed as a value between 0 and 1 or between 1 and 100.

[0278] When the confidence value is greater than a predetermined threshold, the speech recognizer 111a may output a recognition result to allow an operation corresponding to the recognition result to be performed. When the confidence value is equal to or less than the threshold, the speech recognizer 111a may reject the recognition result.

[0279] In addition, the voice recognizer 111a can distinguish the voice input of the passenger through the voice input device 210. Specifically, the voice recognizer 111a can distinguish the voice of each passenger by comparing the non-verbal voice features and the verbal features of the passenger voice input through the voice input device 210. The non-verbal voice features may include the height, intensity, breathing and speed of the passenger voice. The verbal features may include the dialect, slang and stress of the passenger voice.

[0280] In addition, the voice recognizer 111 a may determine whether a new passenger is boarding the vehicle by distinguishing the passenger's voice input through the voice input device 210 .

[0281] The utterance in text form as a recognition result of the speech recognizer 111 a may be input to the natural language understanding part 111 b .

[0282] The natural language understanding part 111b can recognize the intention of the user's speech included in the speech language by applying the natural language understanding technology. Therefore, the user can input the control command through natural dialogue, and the dialogue system 100 can also cause the input of the control command through dialogue and provide the service required by the user.

[0283] The natural language understanding part 111b can perform morpheme analysis on the speech in the form of text. A morpheme is the smallest unit of meaning and represents the smallest semantic element that cannot be further subdivided. Therefore, morpheme analysis is the first step of natural language understanding and converts an input string into a morpheme string.

[0284] The natural language understanding part 111b can obtain domains from the utterance based on the morpheme analysis result. The domains can be used to identify the subject of the user's utterance language, and domains indicating various subjects (such as route guidance, weather search, traffic search, schedule management, fuel management and air conditioning control, passengers boarding the vehicle, and changes in the number of passengers) can be stored as a database.

[0285] The natural language understanding part 111b can recognize entity names from the utterance. The entity names can be proper nouns, such as names of people, places, names of organizations, time, date, and currency, and the entity name recognition can be configured to recognize entity names in sentences and determine the type of the recognized entity names. The natural language understanding part 111b can use entity name recognition to obtain important keywords from sentences and recognize the meaning of sentences.

[0286] The natural language understanding part 111b can analyze the voice behavior contained in the speech. The voice behavior analysis can be configured to identify the intention of the user's speech, for example, whether the user asks a question, whether the user makes a request, whether the user responds, or whether the user simply expresses emotion.

[0287] The natural language understanding part 111b extracts an action corresponding to the intention of the user's utterance. The natural language understanding part 111b can recognize the intention of the user's utterance based on information such as domain, entity name, and voice behavior, and extract an action corresponding to the utterance. The action can be defined by an object and an operator.

[0288] The natural language understanding part 111b can obtain parameters related to the action execution. The parameters related to the action execution can be valid parameters directly required for the action execution, or invalid parameters used to extract valid parameters.

[0289] For example, when the user's utterance is "Let's go to Seoul Station", the natural language understanding part 111b can acquire "navigation" as a domain corresponding to the utterance, and "route guidance" as an action, where the voice behavior corresponds to "request".

[0290] The entity name "Seoul Station" may correspond to [parameter: destination] related to the action execution, but a specific exit number or GPS information of the station may be required to actually guide the route via the navigation system. In this case, [parameter: destination: Seoul Station] extracted by the natural language understanding part 111b may be a candidate parameter for searching for "Seoul Station" actually desired by the user among multiple Seoul Station POIs.

[0291] The natural language understanding section 111b may acquire a tool configured to express a relationship between words or sentences, for example, a parse tree.

[0292] The morpheme analysis result, domain information, action information, voice behavior information, extracted parameter information, entity name information, and parse tree as the processing result of the natural language understanding part 111b can be sent to the dialog input manager 111c.

[0293] In addition, the morpheme analysis results, domain information, action information, voice behavior information, extracted parameter information, entity name information and parse tree as the processing results of the natural language understanding part 111b can be sent to the dialogue input manager 111c through the passenger determiner 111d.

[0294] The passenger determiner 111d estimates the change in the number of passengers in the vehicle based on the output of the natural language understanding part 111b. Specifically, the passenger determiner 111d can estimate the possibility of each passenger leaving the vehicle and the possibility of each passenger boarding the vehicle again after leaving based on the conversations between the occupants in the vehicle, and the passenger determiner 111d can estimate the number of potential passengers based on the call conversations in the vehicle.

[0295] The passenger determiner 111d may generate passenger number information based on the estimation result of the change in the number of passengers.

[0296] The passenger quantity information may include the probability of each passenger leaving the vehicle at the stop point, the probability of each passenger boarding the vehicle again after leaving the vehicle at the stop point, and the probability of a potential passenger boarding the vehicle at the stop point.

[0297] The context information processor 112 may include: a context information collector 112a, which collects information from an information input device 220 other than voice and a vehicle controller 240; a context information collection manager 112b, which manages the collection of context information; and a context understanding part 112c, which understands the context based on the results of natural language understanding and the collected context information.

[0298] The input processor 110 may include: a memory in which a program for performing the above-mentioned operations and operations described later is stored; and a processor for executing the stored program. At least one memory and one processor may be provided, and when multiple memories and processors are provided, they may be integrated on one chip or physically separated.

[0299] The speech input processor 111 and the context information processor 112 included in the input processor 110 may be implemented by the same processor and memory or by separate processors and memories.

[0300] In the following, reference will be made to Fig.22A and Fig. 22BThe method by which the components of the input processor 110 process input data using information stored in the storage device 140 is described in detail.

[0301] refer to Fig.22A , the natural language understanding part 111b can use the domain / action inference rule DB 141 to perform domain extraction, entity recognition, voice behavior analysis, and action extraction.

[0302] In the domain / action inference rule DB 141, domain extraction rules, voice behavior analysis rules, entity name conversion rules, and action extraction rules may be stored.

[0303] Other information such as user input other than voice, vehicle state information, driving environment information, and user information may be input to the context information collector 112 a and then stored in the context information DB 142 , the long-term memory 143 , or the short-term memory 144 .

[0304] For example, raw data detected by the vehicle detector 260 may be classified by sensor type and sensor value and then stored in the context information DB 142 .

[0305] In the short-term memory 144 and the long-term memory 143 , data meaningful to the user may be stored, which may include the current user status, the user's preferences and orientation, or data used to determine the user's preferences and orientation.

[0306] As described above, information that ensures persistence and is therefore usable for a long time can be stored in the long-term storage 143, and can include information related to the user's phone book, schedule, preferences, education, personality, work, and family members. In addition, the long-term storage 143 can store driving-related information related to driving and passenger information of passengers boarding the vehicle while driving.

[0307] Information that cannot ensure persistence or has uncertainty and is therefore available in a short period of time can be stored in the short-term memory 144, and can include current and previous locations, today's schedule, previous conversation content, conversation participants, environment, domain, and driver status. Depending on the type of data, data can be stored repeatedly in at least two storage devices among the context information DB 142, the short-term memory 144, and the long-term memory 143.

[0308] In addition, among the information stored in the short-term memory 144 , data determined to ensure persistence may be sent to the long-term memory 143 .

[0309] Information to be stored in the long-term memory 143 may be acquired using information stored in the short-term memory 144 or the context information DB 142. For example, the user's preference may be acquired by analyzing the destination information or the conversation content stored for a specific duration, and the acquired user's preference may be stored in the long-term memory 143.

[0310] Obtaining information to be stored in the long-term memory 143 by using information stored in the short-term memory 144 or the context information DB 142 may be performed in the dialogue system 100 or in another external system.

[0311] In the former case, it can be executed in the memory manager 135 of the result processor 130. In this case, the data used when obtaining meaningful information or persistent information (e.g., user's preference or orientation) from the data stored in the short-term memory 144 or the context information DB 142 can be stored in the long-term memory 143 in a log file type.

[0312] The memory manager 135 may acquire persistent data by analyzing data stored for more than a certain duration and re-input the data into the long-term memory 143. In the long-term memory 143, a location where persistent data is stored may be different from a location where data stored in a log file type is stored.

[0313] The memory manager 135 may determine persistent data among the data stored in the short-term memory 144 , and move and store the determined data in the long-term memory 143 .

[0314] like Fig. 22B As shown, when information to be stored in the long-term memory 143 is obtained using information stored in the short-term memory 144 or the context information DB 142 in an additional external system, a data management system 800 provided with a communicator 810, a storage device 820, and a controller 830 may be used.

[0315] The communicator 810 may receive data stored in the context information DB 142 or the short-term memory 144. All stored data may be sent to the communicator 810, or data used when obtaining meaningful information or persistent information (such as user's preference or orientation) may be selected and then sent. The received data may be stored in the storage device 820.

[0316] The controller 830 may acquire persistent data by analyzing the stored data, and then transmit the acquired data to the dialogue system 100 via the communicator 810. The transmitted data may be stored in the long-term memory 143 of the dialogue system 100.

[0317] In addition, the dialogue input manager 111c can obtain context information related to the action execution by sending the output result of the natural language understanding part 111b to the context understanding part 112c.

[0318] The context understanding part 112 c may determine which context information is related to the action execution corresponding to the intention of the user's utterance by referring to the context information stored by action in the context understanding table 145 .

[0319] Fig.23A and Fig. 23B is a view showing an example of information stored in a context understanding table.

[0320] refer to Fig.23A For example, the context information related to the action execution and the type of context information can be stored in the context understanding table 145 according to each action.

[0321] For example, when the action is route guidance, the current location may be required as context information, and the type of context information may be GPS information. When the action is vehicle status check, the travel distance may be required as context information, and the type of context information may be an integer. When the action is gas station recommendation, the remaining fuel amount and distance to empty (DTE) may be required as context information, and the type of context information may be an integer.

[0322] When context information related to the action execution corresponding to the intention of the user's speech is pre-stored in the context information DB 142, the long-term memory 143 or the short-term memory 144, the context understanding part 112c can obtain the corresponding information from the context information DB 142, the long-term memory 143 or the short-term memory 144, and send the corresponding information to the dialogue input manager 111c.

[0323] When context information related to the action execution corresponding to the intention of the user utterance is not stored in the context information DB 142, the long-term memory 143, or the short-term memory 144, the context understanding part 112c may request the context information collection manager 112b for the required information. The context information collection manager 112b may allow the context information collector 112a to collect the required information.

[0324] The context information collector 112a may collect data periodically or only when a specific event occurs. In addition, the context information collector 112a may collect data periodically and then additionally collect data when a specific event occurs. In addition, the context information collector 112a may collect data when a data collection request is received from the context information collection manager 112b.

[0325] The context information collector 112a may collect the required information and then store the information in the context information DB 142 or the short-term memory 144. The context information collector 112a may send a confirmation signal to the context information collection manager 112b.

[0326] The context information collection manager 112b may send a confirmation signal to the context understanding part 112c, and the context understanding part 112c may obtain the required information from the long-term memory 143 or the short-term memory 144 and then send the required information to the dialogue input manager 111c.

[0327] Specifically, when the action corresponding to the intention of the user's utterance is route guidance, the context understanding part 112c may search the context understanding table 145 and recognize that the context information related to the route guidance is the current location.

[0328] When the current position is pre-stored in the short-term memory 144, the context understanding part 112c can acquire the current position and send the current position to the dialogue input manager 111c.

[0329] When the current location is not stored in the short-term memory 144 , the context understanding portion 112 c may request the current location from the context information collection manager 112 b , and the context information collection manager 112 b may allow the context information collector 112 a to obtain the current location from the vehicle controller 240 .

[0330] The context information collector 112a may obtain the current location and then store the current location in the short-term memory 144. The context information collector 112a may send a confirmation signal to the context information collection manager 112b. The context information collection manager 112b may send a confirmation signal to the context understanding part 112c, and the context understanding part 112c may obtain the current location information from the short-term memory 144 and then send the information to the dialogue input manager 111c.

[0331] The dialogue input manager 111c may send the output of the natural language understanding part 111b and the output of the context understanding part 112c to the dialogue manager 120, and the dialogue input manager 111c may try to prevent duplicate input from being input to the dialogue manager 120. At this time, the output of the natural language understanding part 111b and the output of the context understanding part 112c may be combined into one output and then sent to the dialogue manager 120 or may be sent to the dialogue manager 120 independently.

[0332] When the context information collection manager 112b determines that a certain event occurs because the data collected by the context information collector 112a satisfies a predetermined condition, the context information collection manager 112b may send an action trigger signal to the context understanding part 112c. The context understanding part 112c may search the context understanding table 145 for context information related to the corresponding event, and when the searched context information is not stored in the context understanding table 145, the context understanding part 112c may send a context information request signal to the context information collection manager 112b again.

[0333] For example, when the context information collection manager 112b determines that a passenger has boarded the vehicle because the vehicle operation information of the information input device 220 other than voice collected by the context information collector 112a satisfies a predetermined condition, the context information collection manager 112b may send an action trigger signal to the context understanding part 112c. The context understanding part 112c may search the context understanding table 145 for context information related to the corresponding event, and when the searched context information is not stored in the context understanding table 145, the context understanding part 112c may send a context information request signal to the context information collection manager 112b again. Fig. 23B As shown, the context information related to the event and the type of context information may be stored in the context understanding table 145 according to each event.

[0334] For example, when the generated event is an engine temperature warning, the engine temperature in integer form may be stored as context information related to the event. When the generated event is driver drowsy driving detection, the driver drowsy driving state in integer form may be stored as context information related to the event. When the generated event is insufficient tire pressure, the tire pressure in integer form may be stored as context information related to the event. When the generated event is a fuel warning, the distance to empty (DTE) in integer form may be stored as context information related to the event. When the generated event is a sensor error, the sensor name in text form may be stored as context information related to the event.

[0335] In addition, if Fig.23CAs shown, context information related to the event and the type of context information can be stored in the context understanding table 145 according to each event. When the event generated is a window adjustment button operation, the window adjustment information in text form can be stored as the context information. When the event generated is a seat adjustment button operation, the seat adjustment information in text form can be stored as the context information. When the event generated is an air conditioning adjustment button operation, the air conditioning adjustment information in text form can be stored as the context information. In addition, an event related to a passenger boarding the vehicle can occur, and at this time, the passenger boarding the vehicle information in text form can be stored as the context information.

[0336] The context information collection manager 112b can collect the required context information via the context information collector 112a and send a confirmation signal to the context understanding part 112c. The context understanding part 112c can obtain the required context information from the context information DB 142, the long-term storage 143 or the short-term storage 144, and then send the context information together with the action information to the dialogue input manager 111c.

[0337] The dialog input manager 111 c may input the output of the context understanding part 112 c to the dialog manager 120 .

[0338] Hereinafter, a case where the dialogue system 100 outputs a preliminary utterance by itself before an utterance of a user is input will be described.

[0339] Fig.24 A control block diagram showing a dialog system suitable for a case where the dialog system first outputs an utterance before receiving a user input, and Fig.25A , Fig.25B and Fig.25C , Fig.25D is a view showing an example of information stored in the pre-utterance condition table.

[0340] refer to Fig.24 The input processor 110 of the dialog system 100 may further include: a pre-discourse determiner 151 that determines whether it is a pre-discourse context; and a repetitive task processor 152. The storage device 140 may further include a pre-discourse condition table 145a storing pre-discourse conditions and a task processing DB 145b.

[0341] The data stored in the context information DB 142, the long-term memory 143, and the short-term memory 144 may be transmitted to the pre-utterance determiner 151. The pre-utterance determiner 151 may analyze the transmitted data and determine whether the transmitted data satisfies the pre-utterance condition stored in the pre-utterance condition table 145a.

[0342] In addition, the voice input processor 111 and the context information processor 112 of the input processor 110 may generate context information indicating whether a passenger has boarded the vehicle, and transmit the generated context information to the pre-utterance determiner 151 .

[0343] The passenger determiner 111 d of the input processor 110 may generate passenger number information based on the estimation result of the change in the number of passengers, and transmit the generated passenger number information to the pre-utterance determiner 151 .

[0344] In addition, when it is determined that the vehicle has left the stop point, the passenger determiner 111d can compare the estimated result of the change in the number of passengers obtained before arriving at the stop point with the result of the change in the number of passengers after leaving the stop point based on the passenger number information, and send the comparison result to the pre-speech determiner 151.

[0345] The pre-speech determiner 151 may analyze the passenger quantity information and the comparison result, and determine whether the transmitted data satisfies the pre-speech condition stored in the pre-speech condition table 145a. Fig.25A For example, in the pre-speech condition table 145a, the pre-speech condition related to the context information of each context information and the pre-speech message output when the corresponding pre-speech condition is met can be stored.

[0346] When the context information transmitted from the context information DB 142 satisfies the pre-utterance condition, the pre-utterance determiner 151 may determine it as a pre-utterance context and generate a pre-utterance trigger signal.

[0347] The pre-discourse determiner 151 may send a pre-discourse trigger signal to the context understanding part 112c together with the pre-discourse message corresponding to the corresponding pre-discourse context. In addition, the pre-discourse determiner 151 may send information related to the corresponding pre-discourse context. The information related to the corresponding pre-discourse context may include a pre-discourse condition corresponding to the corresponding pre-discourse context or an action corresponding to the pre-discourse context described later.

[0348] For example, when the context information is related to the tire pressure and the tire pressure is equal to or less than a predetermined reference value, the pre-speech condition may be satisfied. When the pre-speech condition of the tire pressure is satisfied, the pre-speech determiner 151 may determine that the pre-speech context is caused by insufficient tire pressure and generate a pre-speech trigger signal.

[0349] The pre-speech determiner 151 may send a pre-speech trigger signal together with a pre-speech message to the context understanding part 112c. For example, in a pre-speech context caused by insufficient tire pressure, a pre-speech message indicating low tire pressure (such as "tire pressure is too low") may be sent to the context understanding part 112c.

[0350] In addition, when the context information is related to the engine temperature and the engine temperature is equal to or higher than a predetermined reference value, the pre-speech condition may be satisfied. When the pre-speech condition of the engine temperature is satisfied, the pre-speech determiner 151 may determine that the pre-speech context is caused by an abnormality in the engine temperature and generate a pre-speech trigger signal.

[0351] The pre-speech determiner 151 may send a pre-speech trigger signal together with a pre-speech message to the context understanding part 112c. For example, in a pre-speech context caused by an abnormality in engine temperature, a pre-speech message indicating engine overheating (such as "engine temperature is too high") may be sent to the context understanding part 112c.

[0352] In addition, the pre-speech condition can be met when the context information is related to the remaining amount of gasoline and the remaining amount of gasoline is equal to or less than a predetermined reference value. When the user sets a destination using the vehicle's navigation service, a predetermined reference value can be set based on the distance from the current location to the destination. When the destination is not set, a default value can be used as a reference value. For example, a value less than a reference value for indicating a low fuel warning light can be set as a reference value for a pre-speech condition related to the insufficient remaining amount of gasoline. When the pre-speech condition for the remaining amount of gasoline is met, the pre-speech determiner 151 can determine that the pre-speech context is caused by the insufficient remaining amount of gasoline and generate a pre-speech trigger signal.

[0353] The pre-speech determiner 151 may send the pre-speech trigger signal together with the pre-speech message to the context understanding part 112c. For example, in the pre-speech context caused by insufficient remaining gasoline, a pre-speech message indicating insufficient remaining gasoline (such as "the remaining gasoline is not enough to reach the destination") may be sent to the context understanding part 112c.

[0354] However, Fig.25A The pre-discourse conditions and pre-discourse messages shown in are merely examples that can be applied to the dialogue system 100. In the above example, the case where the pre-discourse message corresponding to the pre-discourse context is the content of notifying the current situation has been described. However, the dialogue system 100 may also first suggest a specific function or service required to execute the pre-discourse context.

[0355] refer to Fig.25B When the pre-speech context is caused by insufficient tire pressure or abnormal engine temperature, a corresponding pre-speech message with the following content can be stored, which is an active suggestion for a repair shop appointment service, such as "Do you want to make an appointment with a repair shop?"

[0356] In addition, when the pre-speech context is caused by insufficient remaining gasoline, a pre-speech message corresponding to this may be stored, the content of which is an active suggestion of a gas station guidance service, such as "Do you want to be guided to a gas station?".

[0357] In addition, when the pre-speech context is caused by the internal temperature of the vehicle and when the internal temperature of the vehicle exceeds a predetermined reference range, the pre-speech condition may be satisfied. When the pre-speech condition of the internal temperature of the vehicle is satisfied, the context understanding part 112c may determine that the pre-speech context is caused by an abnormality in the internal temperature of the vehicle and generate a pre-speech trigger signal.

[0358] In the case where the pre-speech context is caused by abnormal temperature inside the vehicle, a pre-speech message corresponding to this may be stored, the content of which is to actively suggest an interior temperature control function, such as “Do you want to operate the air conditioner?”.

[0359] In addition, when the context information is related to the microphone input and the microphone input value is equal to or less than a predetermined reference value, the pre-speech condition can be met. When the pre-speech condition of the microphone input is met, the context understanding part 112c can determine that the pre-speech condition is a pre-speech context for changing emotions, and generate a pre-speech trigger signal. Therefore, a pre-speech message corresponding to the following content can be stored, and the content is an active suggestion of a multimedia playback service, such as "Do you want to play music?"

[0360] In addition, when the context information is related to the opening and closing of the window and whether it is raining, and when the window is open and it is raining, the pre-speech condition can be met. When the window is open and it is raining, the context understanding part 112c can determine that the pre-speech context is caused by the opening of the window and generate a pre-speech trigger signal.

[0361] In the case where the pre-speech context is caused by the opening of a window, a corresponding pre-speech message with the following content may be stored, the content being an active suggestion for a window closing function, such as “Do you want to close the window?”.

[0362] exist Fig.25A and Fig.25B In the above example, the case where the pre-discourse message corresponding to the pre-discourse context is pre-stored in the pre-discourse condition table 145a has been described. However, the example of the dialogue system 100 is not limited thereto, and thus an action corresponding to the pre-discourse context may be pre-stored.

[0363] As described above, when the user's utterance is input, the natural language understanding part 111b can refer to the domain / action inference rule DB 141 to obtain the action corresponding to the user's utterance. Fig.25C As shown, when the dialogue system 100 outputs a pre-utterance, an action corresponding to the pre-utterance context may be pre-stored in each pre-utterance context.

[0364] For example, when the pre-speech context is caused by abnormalities in tire pressure and engine temperature, "repair shop guidance" may be stored as the corresponding action, and when the pre-speech context is caused by insufficient remaining gasoline, "gas station guidance" may be stored as the corresponding action.

[0365] In addition, when the pre-speech context is caused by the abnormal temperature inside the vehicle, "air conditioning operation" can be stored as a corresponding action, and when the pre-speech context is used to change emotions, "multimedia playback" can be stored as a corresponding action. When the pre-speech context is caused by the opening of the window, "opening and closing of the window" can be stored as a corresponding action.

[0366] As described above, when the action corresponding to the pre-utterance context is pre-stored, the pre-utterance trigger signal may be sent to the context understanding part 112c together with the action corresponding to the pre-utterance context, and the dialogue input manager 111c may input the pre-utterance trigger signal together with the action corresponding to the pre-utterance context to the dialogue manager 120. In this case, the same operation as the case of inputting the user utterance may be performed in the dialogue manager 120.

[0367] For another example, in the pre-discourse condition table 145a, the pre-discourse contexts may be stored in a manner that matches the pre-discourse contexts with the virtual user utterances corresponding to each pre-discourse context, or the pre-discourse determiner 151 may generate virtual user utterances corresponding to the pre-discourse contexts. The pre-discourse determiner 151 may send the user utterances stored in the pre-discourse condition table 145a or generated by the pre-discourse determiner 151 to the natural language understanding part 111b in a text type. For example, when the pre-discourse context is caused by an abnormality in tire pressure, virtual user utterances such as "check tire pressure" or "guidance to the repair shop" may be stored or generated. In addition, when the pre-discourse context is caused by an abnormality in the temperature inside the vehicle, virtual user utterances such as "turn on the air conditioner" may be stored or generated.

[0368] In addition, according to the mobile gateway mode in which the mobile device 400 acts as a gateway between the vehicle and the dialogue system 100, the dialogue system client 470 of the mobile device 400 can perform some operations of the pre-utterance determiner 151. In this case, the dialogue system client 470 can generate a virtual user utterance corresponding to the pre-utterance context and send the virtual user utterance to the natural language understanding part 111b.

[0369] The natural language understanding part 111b may acquire the domain and action corresponding to the transmitted virtual user utterance, and transmit the domain and action to the dialogue input manager 111c. The action extracted by the natural language understanding part 111b may become an action corresponding to the pre-utterance context. The process performed after the action corresponding to the pre-utterance context is transmitted to the dialogue manager 120 may be performed in the same manner as in the case where the user first utters.

[0370] The above context information, pre-speech conditions, pre-speech messages and actions are merely examples of embodiments applied to the dialogue system 100, but the embodiments of the dialogue system 100 are not limited thereto. In addition, various context information, pre-speech conditions, pre-speech messages and actions may be stored.

[0371] When the pre-utterance determiner 151 transmits information related to the pre-utterance trigger signal and the pre-utterance context to the context understanding part 112 c , the context understanding part 112 c may transmit the information related to the pre-utterance context to the repetitive task processor 152 .

[0372] The repetitive task processor 152 may determine (based on information stored in the task processing DB 145 b ) whether a task related to the currently occurring pre-utterance context has been processed, or whether the task is a repetitive task.

[0373] In the task processing DB 145b, information related to tasks that have been processed or are currently being processed may be stored. For example, the conversation history (including the conversation content and each conversation time), the vehicle status during the conversation time, and whether the task is completed, etc. may be stored. In addition, the task process and processing results such as route guidance using a navigation function may be stored regardless of the conversation.

[0374] Specifically, when the pre-speech context is caused by insufficient remaining gasoline, the repetitive task processor 152 can determine whether the gas station guidance task is currently processed based on the information stored in the task processing DB 145b. When the dialogue of gas station guidance is currently being executed or the gas station guidance action is currently being executed, the repetitive task processor 152 can determine that the task related to the current pre-speech context is a repetitive task and terminate the pre-speech context.

[0375] Furthermore, when an utterance for gas station guidance has been previously output, and when there is a dialogue history in which the user rejects the gas station guidance, the repetitive task processor 152 may determine that the task associated with the current pre-utterance context is a repetitive task and terminate the pre-utterance context.

[0376] In addition, when a gas station guidance task using a navigation function is currently being processed, the repetitive task processor 152 may also determine that a task related to the current pre-discourse context is a repetitive task and terminate the pre-discourse context regardless of the conversation history of the gas station guidance. The repetitive task processor 152 may identify that a gas station guidance task using a navigation function is currently being processed based on information stored in the task processing DB 145b.

[0377] In addition, when the reference time period has not passed since the dialogue related to the guidance of the remaining amount of gasoline was performed, it can be assumed that the user drives to the gas station by himself, even if the gas station guidance has not been performed yet. Therefore, the repetitive task processor 152 can determine that the task related to the current pre-discourse context is a repetitive task and terminate the pre-discourse context.

[0378] In addition, in a state where the pre-discourse context is used to indicate a schedule based on information stored in the long-term memory 143 (such as the user's birthday or the birthday of a family member), when there is a conversation history that previously led to the same schedule and a reference time period has not yet passed since the corresponding conversation was performed, the repetitive task processor 152 can determine that the task related to the current pre-discourse context is a repetitive task and terminate the pre-discourse context.

[0379] That is, the repetitive task processor 152 can determine whether a pre-utterance has been previously output and the user's intention regarding the pre-utterance context based on the conversation history stored in the task processing DB 145b. The repetitive task processor 152 can determine whether it is a repetitive task based on the stored conversation time, the user's intention, the vehicle state, or the completion of the task.

[0380] In the repetitive task processor 152, a policy configured to determine whether it is a repetitive task based on the information stored in the task processing DB 145b, that is, whether to terminate the pre-discourse context, may be stored. The repetitive task processor 152 may determine whether the task related to the current pre-discourse context is a repetitive task according to the stored policy, and when it is determined to be a repetitive task, the repetitive task processor 152 may terminate the pre-discourse context.

[0381] In the above-described example, the case where the dialogue system 100 includes the pre-utterance determiner 151 , the repetitive task processor 152 , the pre-utterance condition table 145 a , and the task processing DB 145 b has been described.

[0382] However, the example of the dialogue system 100 is not limited thereto, so Fig.22A and Fig. 22B The components shown in can also perform the operations of the above-mentioned components.

[0383] For example, the context understanding part 112c may perform an operation corresponding to the pre-utterance determiner 151 that determines whether the pre-utterance condition is satisfied, and an operation corresponding to the repetitive task processor 152 that processes a repetitive task.

[0384] The information stored in the pre-utterance condition table 145 a may be stored in the context understanding table 145 , and the information stored in the task processing DB 145 b may be stored in the dialogue and action state DB 147 described later.

[0385] refer to Fig.25D For example, in the pre-speech condition table 145a, the pre-speech condition related to each piece of context information and the pre-speech message output when the corresponding pre-speech condition is met may be stored.

[0386] That is, the dialogue system 100 may obtain the pre-speech message stored in the pre-speech condition table based on the context information and the pre-speech condition.

[0387] When it is determined that the passenger has boarded the vehicle in a state where the context information is related to whether the passenger has boarded the vehicle, the pre-speech condition may be satisfied. When the pre-speech condition is satisfied because it is determined that the passenger has boarded the vehicle, the pre-speech determiner 151 may determine it as a pre-speech context caused by the passenger boarding the vehicle, and may generate a pre-speech trigger signal.

[0388] The pre-speech determiner 151 may send a pre-speech trigger signal to the dialog input manager 111c together with the stored pre-speech message. For example, in the case of a pre-speech context related to a passenger boarding a vehicle, a pre-speech message such as "Who are you? Tell me your name" requesting identification information of the passenger may be sent to the dialog input manager 111c.

[0389] In addition, for context information related to whether a passenger has boarded the vehicle, when it is determined that the passenger has not boarded the vehicle, the pre-speech condition may be satisfied. When the pre-speech condition is satisfied because it is determined that the passenger has not boarded the vehicle, the pre-speech determiner 151 may determine it as a pre-speech context caused by the passenger not boarding the vehicle, and may generate a pre-speech trigger signal.

[0390] The pre-speech determiner 151 may send a pre-speech trigger signal together with the stored pre-speech message to the dialogue input manager 111c. For example, in a pre-speech context related to a passenger not boarding the vehicle, a pre-speech message for verifying whether the passenger exists (such as "Are there any other passengers boarding the vehicle?") may be sent to the dialogue input manager 111c.

[0391] In addition, when the dialogue system 100 estimates the possibility of a potential passenger boarding the vehicle in a state where the context information is related to whether the potential passenger will board the vehicle, the pre-speech condition may be satisfied. When the pre-speech condition is satisfied due to the estimation of the possibility of a potential passenger boarding the vehicle, the pre-speech determiner 151 may determine it as a pre-speech context caused by the potential passenger boarding the vehicle, and may generate a pre-speech trigger signal.

[0392] The pre-speech determiner 151 may send a pre-speech trigger signal to the dialog input manager 111c together with the stored pre-speech message. For example, in the case of pre-speech that a potential passenger will board a vehicle, a pre-speech message verifying whether the passenger exists (such as "Who is on the way to board the vehicle? Tell me his / her name") may be sent to the dialog input manager 111c.

[0393] In addition, when the change in the number of passengers is estimated in a state where the context information is related to before arriving at the stop point, the pre-speech condition can be met. When the pre-speech condition is met due to the estimated change in the number of passengers, the pre-speech determiner 151 can determine that it is due to the pre-speech context before arriving at the stop point, and a pre-speech trigger signal can be generated.

[0394] The pre-speech determiner 151 may send a pre-speech trigger signal together with the stored pre-speech message to the dialogue input manager 111c. For example, in a pre-speech context related to before arriving at a stop, pre-speech related to an estimated result of a change in the number of passengers, such as "A will leave at the stop," "B will get on the vehicle again after leaving the vehicle at the stop," "C will not leave at the stop," and "D will get on the vehicle at the stop" may be sent to the dialogue input manager 111c.

[0395] In addition, when the change in the number of passengers is estimated in a state where the context information is related to leaving the stop point, the pre-speech condition can be met. When the pre-speech condition is met due to the estimated change in the number of passengers, the pre-speech determiner 151 can determine that it is due to the pre-speech context after leaving the stop point, and a pre-speech trigger signal can be generated.

[0396] The pre-speech determiner 151 may send a pre-speech trigger signal to the dialog input manager 111c together with the stored pre-speech message. For example, in the pre-speech context after leaving the stop point, a pre-speech to verify whether the estimated result of the change in the number of passengers is correct, such as "A left?", "Did B get on the vehicle again?", "Is C still there?", and "Did D get on the vehicle?" may be sent to the dialog input manager 111c to verify the result of the change in the number of passengers after leaving the stop point.

[0397] In addition, when the estimation result of the change in the number of passengers is compared with the result of the change in the number of passengers after leaving the stop point in a state where the context information is related to the state after leaving the stop point, the pre-speech condition may be satisfied. When the pre-speech condition is satisfied due to the comparison of the estimation result of the change in the number of passengers with the result of the change in the number of passengers after leaving the stop point, the pre-speech determiner 151 may determine that it is due to the pre-speech context after leaving the stop point, and may generate a pre-speech trigger signal.

[0398] The pre-discourse determiner 151 may send a pre-discourse trigger signal to the dialogue input manager 111c together with the stored pre-discourse message. For example, in a pre-discourse context related to after leaving a stop point, a pre-discourse indicating that the estimated result of the change in the number of passengers is different from the result of the change in the number of passengers after leaving the stop point, such as "the current number of passengers is different from the estimated result of the change in the number of passengers" may be sent to the dialogue input manager 111c, or a pre-discourse indicating that the estimated result of the change in the number of passengers is the same as the result of the change in the number of passengers after leaving the stop point, such as "the current number of passengers is the same as the estimated result of the change in the number of passengers" may be sent to the dialogue input manager 111c.

[0399] In addition, when the characteristics of the passenger boarding the vehicle are determined to be the same as the stored passenger information in a state where the context information is related to whether the passenger has boarded the vehicle, the pre-speech condition may be satisfied. When the pre-speech condition is satisfied due to the estimated change in the number of passengers, the pre-speech determiner 151 may determine that it is a pre-speech context due to the passenger boarding the vehicle, and may generate a pre-speech trigger signal.

[0400] The pre-speech determiner 151 may send a pre-speech trigger signal to the dialog input manager 111c together with the stored pre-speech message. For example, in a pre-speech context caused by a passenger boarding a vehicle, a pre-speech message (such as "You are 00") for verifying whether the current passenger was in the previous trip may be sent to the dialog input manager 111c.

[0401] As described above, the dialogue system 100 may determine whether a passenger has boarded the vehicle by using the pre-utterance condition table 145 a and output a pre-utterance for identifying the passenger.

[0402] Specifically, the dialogue system 100 may determine that a passenger has boarded the vehicle based on at least one of a dialogue between occupants in the vehicle and vehicle operation information. For example, the dialogue system 100 may determine that a passenger has boarded the vehicle based on a dialogue between occupants in the vehicle input through the voice input processor 111. The occupants in the vehicle may include a driver and at least one passenger, and the vehicle operation information may include operation information of the information input device 220 other than voice.

[0403] The determination that a passenger has boarded the vehicle, performed by the dialogue system 100 , may be performed within a certain period of time from when the vehicle 200 starts traveling or within a certain period of time from when the vehicle 200 stops traveling.

[0404] The voice input processor 111 can distinguish the voice of each passenger based on the conversation between the occupants in the vehicle input through the voice input device 210 provided in the vehicle 200 and the voice input device 410 provided in the mobile device 400. Specifically, the voice input processor 111 can distinguish the voice of each passenger by comparing the voice of each passenger based on the non-verbal voice features and verbal features of the passenger voice input through the voice input device 210. The non-verbal voice features may include the height, intensity, breathing, and speed of the passenger voice. The verbal voice features may include the dialect, slang, and stress of the passenger voice.

[0405] Therefore, the voice input processor 111 can determine whether each passenger boards the vehicle by distinguishing the voice of each passenger input through the voice input device 210 and the voice input device 410 provided in the mobile device 400 based on one or more voice features.

[0406] The speech feature may include at least one of a non-verbal feature and a verbal feature.

[0407] The conversation between the occupants in the vehicle, which is input through the voice input processor 111 to determine whether the passenger has boarded the vehicle, may represent not speech for transmitting an intention to the vehicle 200, but a conversation between the occupants in the vehicle including the driver.

[0408] The context information processor 112 of the dialogue system 100 can determine whether a passenger has boarded the vehicle based on the vehicle operation information. That is, the dialogue system 100 can determine whether a passenger has boarded the vehicle based on the vehicle operation information, so as to determine whether there is a passenger who has not boarded the vehicle determined by the voice input processor 111 because he has not participated in the dialogue.

[0409] Specifically, the context information processor 112 may detect the operation of the information input device 220 other than voice performed by the passenger. The information input device 220 other than voice may include window adjustment buttons, seat adjustment buttons, and air conditioning adjustment buttons respectively provided on the passenger seat 254b and the rear seats 254c and 254d to determine the passenger boarding the vehicle and the seat position of the passenger. When the operation of the information input device 220 other than voice is detected, the context information processor 112 may acquire vehicle operation information based on the operation of the information input device 220 other than voice.

[0410] The vehicle operation information may include at least one of window adjustment button operation information, seat adjustment button operation information, or air conditioning adjustment button operation information related to the passenger seat 254 b and the rear seats 254 c and 254 d .

[0411] The contextual information processor 112 may determine that a passenger has boarded the vehicle based on the vehicle operation information.

[0412] That is, the input processor 110 may collect passenger boarding vehicle information indicating a context in which a passenger boarded the vehicle or a context in which a passenger did not board the vehicle through at least one of the voice input processor 111 and the context information processor 112 .

[0413] When the dialogue system 100 determines that a passenger has boarded the vehicle within a period of time from when the vehicle 200 starts to travel or within a period of time from when the vehicle 200 stops traveling, the dialogue system 100 may output a preliminary speech for requesting identification information. Specifically, when it is determined that a passenger has boarded the vehicle, the dialogue system 100 may output a preliminary speech for requesting identification information.

[0414] For example, when it is determined that a passenger has boarded the vehicle, the dialogue system 100 may output a pre-utterance requesting identification information of the passenger, such as “Who are you? Tell me your name”.

[0415] The pre-speech determiner 151 of the input processor 110 may determine whether to output a pre-speech based on a pre-speech condition, the pre-speech condition being related to determining whether a passenger has boarded the vehicle based on context information related to whether a passenger has boarded the vehicle. In addition, when the context information related to whether a passenger has boarded the vehicle satisfies the pre-speech condition corresponding to the determination of whether a passenger has boarded the vehicle, the pre-speech determiner 151 may determine it as a pre-speech context and generate a pre-speech trigger signal.

[0416] The pre-speech determiner 151 may obtain a pre-speech message corresponding to a pre-speech context related to a passenger boarding a vehicle, such as “Who are you? Tell me your name.” When the pre-speech determiner 151 sends a pre-speech trigger signal and a pre-speech message to the dialogue input manager 111c, the dialogue input manager 111c may send the pre-speech message to the dialogue manager 120. At this time, a pre-speech trigger signal or a signal indicating a pre-speech context may be sent together with the pre-speech message.

[0417] The dialog manager 120 may generate a dialog task for outputting the transmitted pre-utterance message, and transmit the pre-utterance message to the result processor 130. The result processor 130 may output the input pre-utterance message via the speaker 232.

[0418] The dialogue system 100 can identify the passenger by receiving the passenger's speech. Specifically, the dialogue system 100 can identify the passenger by receiving the passenger's speech about the pre-speech message.

[0419] For example, the passenger may speak "I am 00" in response to a pre-speech message requesting the passenger's identification information. That is, the passenger may speak a message including his / her name in response to the pre-speech message.

[0420] When the passenger's speech is input, the voice input processor 111 recognizes the input passenger's speech. The passenger's speech may be input through the voice input device 210 provided in the vehicle 200 or the voice input device 410 provided in the mobile device 400.

[0421] The speech recognizer 111a may recognize an input user's utterance and output the utterance in text form. The natural language understanding section 111b may apply a natural language understanding technique to the utterance in text form and output a result of the natural language understanding.

[0422] The natural language understanding process may include performing morpheme analysis on the utterance in text form and recognizing the name based on the result of the morpheme analysis.

[0423] In addition, the natural language understanding section 111b can increase the recognition rate of the name using the driver's phone book stored in the long-term memory 143. Specifically, the natural language understanding section 111b can increase the recognition rate of the name by comparing the name contained in the passenger's utterance with the name contained in the phone book.

[0424] The passenger determiner 111d may verify the passenger's name based on the output of the natural language understanding part 111b in order to identify the passenger.

[0425] Therefore, based on the passenger's utterances, the dialogue system 100 can identify the identity of the passenger who uttered the message.

[0426] In addition, the dialogue system 100 can determine the seat position of the passenger in the vehicle by estimating the position of the speech based on the direction and size of the passenger's speech. The dialogue system 100 can determine the seat position of the passenger in the vehicle based on the vehicle operation information related to the window adjustment buttons, seat adjustment buttons, and air conditioning adjustment buttons provided on the front passenger seat 254b and the rear seats 254c and 254d. Therefore, the dialogue system 100 can generate seat position information for each passenger by mapping the passenger with the seat position in the vehicle.

[0427] When the seat position of the passenger is changed while driving, the dialogue system 100 can estimate the change of the seat position of the passenger based on the passenger's speech and vehicle operation information, and determine the changed seat position of the passenger. In this case, the dialogue system 100 can generate seat position information by applying the changed passenger seat position.

[0428] Passenger information about the identified passenger may be stored in the storage device 140 in real time, wherein the passenger information may include personal identification information, one or more voice features of the passenger's voice, and seat location information.

[0429] When it is determined that the passenger has not boarded the vehicle, the dialogue system 100 may output a pre-speech to verify whether the passenger exists. Specifically, when the dialogue system 100 determines that the passenger has not boarded the vehicle within a period of time since the vehicle 200 started to travel or within a period of time since the vehicle 200 stopped traveling, the dialogue system 100 may output a pre-speech to verify whether the passenger exists.

[0430] For example, when the dialogue system 100 determines that a passenger has not boarded the vehicle within a period of time since the vehicle 200 started driving or within a period of time since the vehicle 200 stopped driving, the dialogue system 100 can output a pre-utterance to verify whether the passenger exists, "Are there any other passengers boarding the vehicle?".

[0431] The pre-speech determiner 151 can determine whether to output the pre-speech based on the context information related to whether the passenger has boarded the vehicle and the pre-speech condition related to the passenger not boarding the vehicle. In addition, when the context information related to whether the passenger has boarded the vehicle satisfies the pre-speech condition related to the passenger not boarding the vehicle, the pre-speech determiner 151 can determine it as the pre-speech context and generate a pre-speech trigger signal.

[0432] The pre-speech determiner 151 may obtain a pre-speech message corresponding to the pre-speech context in which the passenger has not boarded the vehicle, such as “Are there any other passengers boarding the vehicle?” When the pre-speech determiner 151 sends the pre-speech trigger signal and the pre-speech message to the dialogue input manager 111c, the dialogue input manager 111c may send the pre-speech message to the dialogue manager 120. At this time, the pre-speech trigger signal or the signal indicating the pre-speech context may be sent together with the pre-speech message.

[0433] The dialog manager 120 may generate a dialog task for outputting the transmitted pre-utterance message, and transmit the pre-utterance message to the result processor 130. The result processor 130 may output the input pre-utterance message via the speaker 232.

[0434] The dialogue system 100 can verify whether the passenger exists by receiving the driver's speech. Specifically, the dialogue system 100 can verify whether the passenger exists by receiving the driver's speech about the pre-speech message.

[0435] For example, in response to a pre-speech message verifying whether a passenger exists, the driver can say "no" or "yes." That is, the driver can say a message including a response indicating the presence of a passenger in response to the pre-speech message.

[0436] When the driver's speech is input, the voice input processor 111 recognizes the input driver's speech. The driver's speech may be input through the voice input device 210 provided in the vehicle 200 or the voice input device 410 provided in the mobile device 400.

[0437] The speech recognizer 111a may recognize an input user's utterance and output the utterance in text form. The natural language understanding section 111b may apply a natural language understanding technique to the utterance in text form and output a result of the natural language understanding.

[0438] The natural language understanding part 111b can recognize entity names from the utterance. The entity names can be proper nouns, such as names of people, places, names of organizations, time, date, and currency, and the entity name recognition can be configured to recognize entity names in sentences and determine the type of the recognized entity names. The natural language understanding part 111b can use entity name recognition to obtain important keywords from sentences and recognize the meaning of sentences.

[0439] Specifically, the natural language understanding process may include performing morpheme analysis on a text-form utterance and recognizing an entity name based on a result of the morpheme analysis.

[0440] The output of the natural language understanding part 111 b as a result of natural language understanding may include an entity name and a result of morpheme analysis corresponding to the utterance of the passenger.

[0441] The passenger determiner 111d may recognize the presence of a passenger based on the output of the natural language understanding part 111b.

[0442] Therefore, the voice input processor 111 can verify the presence of the passenger based on the driver's words.

[0443] The dialogue system 100 can estimate the possibility of each passenger leaving the vehicle at the stop point and the possibility of each passenger getting on the vehicle again after leaving the stop point by identifying the passengers, and guide them.

[0444] In addition, the dialogue system 100 can perform recognition of a passenger based on a conversation between occupants in the vehicle and vehicle operation information within a period of time from when the vehicle 200 starts traveling or within a period of time from when the vehicle 200 stops traveling. Therefore, the dialogue system 100 can continuously recognize new passengers.

[0445] In addition, when the dialogue system 100 erroneously recognizes a passenger who has already boarded the vehicle as a new passenger and outputs a pre-utterance, the passenger may notify that he / she is an existing passenger by uttering a message including his / her name.

[0446] The dialogue system 100 can estimate the change in the number of passengers by using the pre-utterance condition table 145a and output a pre-utterance regarding the estimation result of the change in the number of passengers.

[0447] Specifically, as described above, the dialogue system 100 can identify the passenger. The dialogue system 100 can determine the passenger's boarding of the vehicle based on at least one of the dialogue between occupants in the vehicle and the vehicle operation information, and identify the passenger by using the pre-utterance.

[0448] The dialogue system 100 can generate passenger quantity information based on the dialogues between the passengers in the vehicle. Specifically, the dialogue system 100 can determine the possibility of each passenger leaving the vehicle at a specific stop point and the possibility of each passenger getting on the vehicle again after leaving at a specific stop point by continuously receiving the dialogues between the passengers in the vehicle.

[0449] For example, the voice input processor 111 of the dialogue system 100 may continuously receive a dialogue between occupants in the vehicle through the voice input device 210 provided in the vehicle 200 or the voice input device 410 provided in the mobile device 400 .

[0450] The speech recognizer 111a may recognize an input user's utterance and output the utterance in text form. The natural language understanding section 111b may apply a natural language understanding technique to the utterance in text form and output a result of the natural language understanding.

[0451] The natural language understanding part 111b can recognize entity names from the utterance. The entity names can be proper nouns, such as names of people, places, names of organizations, time, date, and currency, and the entity name recognition can be configured to recognize entity names in sentences and determine the type of the recognized entity names. The natural language understanding part 111b can use entity name recognition to obtain important keywords from sentences and recognize the meaning of sentences.

[0452] Specifically, the natural language understanding process may include performing morpheme analysis on a text-form utterance and recognizing an entity name based on a result of the morpheme analysis.

[0453] The output of the natural language understanding part 111 b as a result of natural language understanding may include an entity name and a result of morpheme analysis corresponding to the utterance of the passenger.

[0454] The passenger determiner 111d can estimate the change in the number of passengers based on the output of the natural language understanding part 111b. Specifically, the passenger determiner 111d can estimate the change in the number of passengers at a specific stop by analyzing the utterances of the passengers.

[0455] For example, when a passenger issues a message indicating that he / she will leave at a specific stop, such as "I'm leaving at Seoul Station," the speech recognizer 111a can output the passenger's speech in text form, and the natural language understanding part 111b can apply natural language understanding technology to the speech in text form and output the result of natural language understanding.

[0456] Specifically, the natural language understanding section 111b may output morpheme analysis results such as "I", "soon", "Seoul Station" corresponding to the entity name, and "leave".

[0457] The passenger determiner 111d may estimate that a specific passenger will leave at a specific stop point based on the entity name and morpheme analysis results of the natural language understanding part 111b.

[0458] Specifically, when a certain passenger sends a message indicating that he / she will leave at a specific stop, such as "I will leave at Seoul Station soon", the passenger determiner 111d can estimate that the specific passenger will leave at Seoul Station in the near future. That is, the passenger determiner 111d can estimate the possibility of the specific passenger leaving at the specific stop and the time point.

[0459] In addition, when a passenger utters a message indicating that he / she will leave at a specific stop and then board the vehicle again, such as "I am about to leave at Seoul Station and board the vehicle again," the natural language understanding part 111b can output morpheme analysis results such as "I," "leave," "Seoul Station," "again," and "board the vehicle" corresponding to the entity names.

[0460] The passenger determiner 111d can estimate that a specific passenger will leave at a specific stop point and board the vehicle again based on the entity name and morpheme analysis result of the natural language understanding part 111b.

[0461] Specifically, when a certain passenger sends a message indicating that he / she will leave and board the vehicle again at a specific stop point, such as "I will leave and board the vehicle again at Seoul Station", the passenger determiner 111d can estimate that the specific passenger will leave and board the vehicle again at Seoul Station. That is, the passenger determiner 111d can estimate the possibility that the specific passenger will board the vehicle again after leaving.

[0462] Additionally, the dialog system 100 can estimate the number of potential passengers by receiving a call session in the vehicle to determine the likelihood of the called party boarding the vehicle.

[0463] Specifically, when a passenger in a vehicle utters a message indicating that a specific potential passenger will board the vehicle at a specific stop point, such as "We will meet at Seoul Station soon," the natural language understanding part 111b can output morpheme analysis results such as "soon," "Seoul Station" corresponding to the entity name, and "meet."

[0464] The passenger determiner 111d may estimate that a potential passenger will board the vehicle at a specific stop point based on the entity name and morpheme analysis results of the natural language understanding part 111b.

[0465] Specifically, when an occupant in the vehicle speaks a message indicating that a specific potential passenger will board the vehicle at a specific stop, such as "We will meet soon at Seoul Station", the passenger determiner 111d can estimate that the specific potential passenger will board the vehicle at Seoul Station. That is, the passenger determiner 111d can estimate the possibility that the potential passenger will board the vehicle.

[0466] When estimating the possibility of a potential passenger boarding the vehicle, the dialogue system 100 may output a pre-utterance for verifying the possibility of the potential passenger boarding the vehicle.

[0467] For example, when estimating the possibility of a potential passenger boarding the vehicle, the dialogue system 100 may output a pre-utterance for verifying the possibility of the potential passenger boarding the vehicle, such as “Who is boarding the vehicle midway? Tell me his / her name”.

[0468] The pre-speech determiner 151 can determine whether to output the pre-speech based on the context information related to whether the potential passenger will board the vehicle and the pre-speech condition related to the estimation of the possibility of boarding the vehicle. In addition, when the context information related to whether the potential passenger will board the vehicle satisfies the pre-speech condition related to the estimation of the possibility of the potential passenger boarding the vehicle, the pre-speech determiner 151 can determine it as the pre-speech context and generate a pre-speech trigger signal.

[0469] The pre-speech determiner 151 may obtain a pre-speech message corresponding to a pre-speech context in which a potential passenger will board the vehicle, such as "Who is boarding the vehicle midway? Tell me his / her name". When the pre-speech determiner 151 sends a pre-speech trigger signal and a pre-speech message to the dialogue input manager 111c, the dialogue input manager 111c may send the pre-speech message to the dialogue manager 120. At this time, a pre-speech trigger signal or a signal indicating a pre-speech context may be sent together with the pre-speech message.

[0470] The dialog manager 120 may generate a dialog task for outputting the transmitted pre-utterance message, and transmit the pre-utterance message to the result processor 130. The result processor 130 may output the input pre-utterance message via the speaker 232.

[0471] The dialogue system 100 can verify the possibility of a potential passenger boarding the vehicle by receiving the occupant's speech in the vehicle. Specifically, the dialogue system 100 can verify whether a potential passenger exists by receiving the occupant's speech about the pre-speech message.

[0472] The passenger determiner 111d can estimate the change in the number of passengers in the vehicle based on the output of the natural language understanding part 111b. Specifically, the passenger determiner 111d can estimate the number of potential passengers based on the call conversation, and the passenger determiner 111d can also estimate the possibility of each passenger leaving the vehicle and the possibility of each passenger getting on the vehicle again after leaving based on the conversation between the occupants in the vehicle.

[0473] The passenger determiner 111d may generate passenger number information based on the estimation result of the change in the number of passengers.

[0474] That is, the passenger determiner 111d may generate passenger quantity information based on the probability of each passenger leaving the vehicle at the stop point, the probability of each passenger boarding the vehicle again after leaving the stop point, and the probability of potential passengers boarding the vehicle at the stop point.

[0475] Before arriving at the stop point, the dialogue system 100 may output preliminary utterances related to the estimation result of the change in the number of passengers based on the passenger number information.

[0476] For example, when it is determined that before the vehicle 200 arrives at a stop point, the dialogue system 100 can output pre-speech related to the estimated result of the change in the number of passengers, such as "A will leave at the stop point", "B will leave the vehicle at the stop point and then board the vehicle again", "C will not leave at the stop point" and "D will board the vehicle at the stop point".

[0477] That is, before arriving at the stop point, the dialogue system 100 can output pre-speech related to the possibility of each passenger leaving the vehicle at the stop point, the possibility of each passenger boarding the vehicle again at the stop point, and the possibility of boarding the vehicle at the stop point contained in the passenger quantity information.

[0478] However, the dialogue system 100 may output preliminary utterances related to the estimation result of the change in the number of passengers based on the passenger number information not only before arriving at the stop point but also after arriving at the stop point.

[0479] In addition, content related to the probability of each passenger leaving the vehicle at a stop point, the probability of each passenger boarding the vehicle again at a stop point, and the probability of boarding the vehicle at a stop point may include messages about the number of passengers leaving the vehicle, such as "See you next time", and messages about passengers boarding the vehicle again after leaving the vehicle, such as "Have a nice trip and come back safely."

[0480] The dialogue system 100 may determine whether the vehicle is just before arriving at the stop point or whether the vehicle is just after arriving at the stop point based on vehicle state information such as the vehicle position and vehicle speed detected by the vehicle detector 260 .

[0481] Specifically, when the gear is placed in the P gear, the dialogue system 100 may determine that the vehicle 200 arrives at the stop point, and when the speed is equal to or less than 10 kph, the dialogue system 100 may determine that the vehicle 200 is just before arriving at the stop point.

[0482] The pre-speech determiner 151 can determine whether to output the pre-speech based on the context information related to the arrival at the stop point and the pre-speech condition related to the estimation of the change in the number of passengers. The pre-speech determiner 151 can determine whether to satisfy the pre-speech condition related to the estimation of the change in the number of passengers based on the passenger number information sent from the passenger determiner 111d. In addition, when the context information related to the arrival at the stop point satisfies the pre-speech condition related to the estimation of the change in the number of passengers, the pre-speech determiner 151 can determine it as the pre-speech context and generate a pre-speech trigger signal.

[0483] The pre-discourse determiner 151 may obtain a pre-discourse message corresponding to a pre-discourse context of estimating a change in the number of passengers based on the passenger number information, such as "A will leave at the stop point", "B will get on the vehicle again after leaving the vehicle at the stop point", "C will not leave at the stop point", and "D will get on the vehicle at the stop point". When the pre-discourse determiner 151 sends the pre-discourse trigger signal and the pre-discourse message to the dialogue input manager 111c, the dialogue input manager 111c may send the pre-discourse message to the dialogue manager 120. At this time, the pre-discourse trigger signal or the signal indicating the pre-discourse context may be sent together with the pre-discourse message.

[0484] The dialog manager 120 may generate a dialog task for outputting the transmitted pre-utterance message, and transmit the pre-utterance message to the result processor 130. The result processor 130 may output the input pre-utterance message via the speaker 232.

[0485] The dialogue system 100 may compare the estimation result of the change in the number of passengers with the result of the change in the number of passengers after departure from the stop point based on the passenger number information.

[0486] The dialogue system 100 may determine whether the vehicle leaves the stop point based on vehicle state information such as the vehicle position and vehicle speed detected by the vehicle detector 260 .

[0487] Specifically, the dialogue system 100 may determine that the vehicle leaves the stop point based on facts such as the parking brake being released, the ignition being on, or the brake pedal being on.

[0488] The dialogue system 100 may detect passengers through the voice input processor 111 and the context information processor 112 and identify the passengers through the passenger determiner 111D to determine the result of the change in the number of passengers after leaving the stop point.

[0489] Therefore, when determining that the vehicle 200 leaves the stop point, the passenger determiner 111d of the dialogue system 100 may compare the estimation result of the change in the number of passengers based on the passenger number information with the result of the change in the number of passengers after leaving the stop point.

[0490] In addition, the dialogue system 100 may output a preliminary utterance to compare the estimated result of the change in the number of passengers with the result of the change in the number of passengers after leaving the stop point.

[0491] For example, the dialogue system 100 may output a pre-utterance to verify whether the estimation result of the change in the number of passengers is correct, so as to verify the result of the change in the number of passengers after leaving the stop point. Specifically, the dialogue system 100 may output a pre-utterance to determine whether a passenger who is determined to leave at the stop point leaves at the stop point, such as "A leaves?", and a pre-utterance to determine whether a passenger who gets on the vehicle again after being determined to leave the vehicle at the stop point gets on the vehicle again at the stop point, such as "Did B get on the vehicle again?".

[0492] In addition, the dialogue system 100 can output pre-words to determine whether a passenger who is determined not to leave at the stop point has not left at the stop point, such as "Is C still here?", and pre-words to determine whether a potential passenger who is determined to board the vehicle at the stop point has boarded the vehicle at the stop point, such as "Has D boarded the vehicle?".

[0493] The pre-speech determiner 151 can determine whether to output the pre-speech based on the context information after leaving the stop point and the pre-speech condition related to whether to estimate the change in the number of passengers. The pre-speech determiner 151 can determine whether the pre-speech condition related to the estimation of the change in the number of passengers is satisfied based on the passenger number information sent from the passenger determiner 111d. In addition, when the context information related to leaving the stop point satisfies the pre-speech condition related to the estimation of the change in the number of passengers, the pre-speech determiner 151 can determine it as the pre-speech context and generate a pre-speech trigger signal.

[0494] The pre-speech determiner 151 may obtain a pre-speech message corresponding to the change in the number of passengers. When the pre-speech determiner 151 sends the pre-speech trigger signal and the pre-speech message to the dialogue input manager 111c, the dialogue input manager 111c may send the pre-speech message to the dialogue manager 120. At this time, the pre-speech trigger signal or a signal indicating that it is a pre-speech context may be sent together with the pre-speech message.

[0495] The dialog manager 120 may generate a dialog task for outputting the transmitted pre-utterance message, and transmit the pre-utterance message to the result processor 130. The result processor 130 may output the input pre-utterance message via the speaker 232.

[0496] The dialogue system 100 may verify the result of the change in the number of passengers by receiving the occupant's speech in the vehicle. Specifically, the passenger determiner 111d of the dialogue system 100 may verify the result of the change in the number of passengers by receiving the occupant's speech corresponding to the pre-speech message.

[0497] For example, the occupant may utter “Yes, he / she left” or “No, he / she boarded the vehicle” in response to a pre-speech message such as “Did A leave?” that inquires whether the passenger who is to be determined to leave at the stop point has left at the stop point. That is, the occupant in the vehicle may utter a message indicating the result of the change in the number of passengers in response to the pre-speech message that inquires whether the passenger who is to be determined to leave at the stop point has left at the stop point.

[0498] When the occupant's speech in the vehicle is input, the voice input processor 111 recognizes the input occupant's speech in the vehicle. The occupant's speech may be input through the voice input device 210 provided in the vehicle 200 or the voice input device 410 provided in the mobile device 400.

[0499] The speech recognizer 111a may recognize an input user's utterance and output the utterance in text form. The natural language understanding section 111b may apply a natural language understanding technique to the utterance in text form and output a result of the natural language understanding.

[0500] Specifically, the natural language understanding process may include performing morpheme analysis on the utterance in the text form, and recognizing the result of the change in the number of passengers based on the result of the morpheme analysis.

[0501] Therefore, the passenger determiner 111d of the dialogue system 100 can verify the result of the change in the number of passengers based on the dialogue between the occupants in the vehicle.

[0502] The dialogue system 100 may output the preliminary utterance related to the comparison result between the estimation result of the change in the number of passengers and the result of the change in the number of passengers after leaving from the stop point.

[0503] For example, the dialogue system 100 may output a pre-speech message indicating that the estimated result of the change in the number of passengers is different from the result of the change in the number of passengers after leaving the stop point, such as “The current number of passengers is different from the estimated result of the change in the number of passengers”, and a pre-speech message indicating that the estimated result of the change in the number of passengers is the same as the result of the change in the number of passengers after leaving the stop point, such as “The current number of passengers is the same as the estimated result of the change in the number of passengers”.

[0504] Specifically, the pre-speech determiner 151 can determine whether to output the pre-speech based on the context information related to after leaving the stop point, according to the pre-speech condition related to whether to compare the estimated result of the change in the number of passengers with the result of the change in the number of passengers after leaving the stop point. The pre-speech determiner 151 can determine whether the pre-speech condition is satisfied based on the comparison result between the estimated result of the change in the number of passengers sent from the passenger determiner 111d and the result of the change in the number of passengers after leaving the stop point. In addition, when the context information after leaving the stop point satisfies the pre-speech condition related to the comparison between the estimated result of the change in the number of passengers and the result of the change in the number of passengers after leaving the stop point, the pre-speech determiner 151 can be determined as a pre-speech context and generate a pre-speech trigger signal.

[0505] The pre-speech determiner 151 may acquire a pre-speech message indicating a comparison result based on a comparison result between an estimation result of a change in the number of passengers and a result of a change in the number of passengers after leaving a stop point. When the pre-speech determiner 151 sends a pre-speech trigger signal and a pre-speech message to the dialogue input manager 111c, the dialogue input manager 111c may send the pre-speech message to the dialogue manager 120. At this time, a pre-speech trigger signal or a signal indicating a pre-speech context may be sent together with the pre-speech message.

[0506] The dialog manager 120 may generate a dialog task for outputting the transmitted pre-utterance message, and transmit the pre-utterance message to the result processor 130. The result processor 130 may output the input pre-utterance message via the speaker 232.

[0507] Therefore, the driver can verify whether each passenger leaves or boards the vehicle based on the comparison result between the estimation result of the change in the number of passengers and the result of the change in the number of passengers after leaving from the stop point, and focus on vehicle management such as driving and parking without paying attention to whether the passengers leave or board the vehicle.

[0508] In addition, it is possible to prevent a situation where a passenger remains at a stop point because he / she fails to board the vehicle again or a situation where a passenger fails to leave when the vehicle arrives at the stop point.

[0509] When the driving of the vehicle is terminated, the dialogue system 100 may store the driving related information and the passenger information about each passenger.

[0510] For example, when the driving of the vehicle is terminated, the storage device 140 of the dialogue system 100 may store information related to the driving of the vehicle and passenger information of each passenger who boarded the vehicle while driving.

[0511] Specifically, the storage device 140 of the dialogue system 100 can store travel-related information about the vehicle's travel, such as the starting point, stop point and destination of the travel, and passenger information about passengers, such as personal identification information, voice feature information, seat position information, boarding vehicle time information, departure time information, boarding vehicle location information and information related to the location of leaving the vehicle.

[0512] That is, the storage device 140 of the dialogue system 100 may store travel-related information related to the travel of the vehicle, such as a departure point, a stop point, and a destination of the travel, by collecting GPS values ​​from the vehicle controller 240 .

[0513] In addition, the storage device 140 of the dialogue system 100 can collect passenger identification information, voice feature information, seat location information and passenger quantity information and store passenger information about passengers, such as passenger identification information, voice feature information, seat location information, boarding vehicle time information, departure time information, boarding vehicle location information and information related to the location of leaving the vehicle.

[0514] Furthermore, the dialogue system 100 may determine that a passenger has boarded the vehicle by using the pre-utterance condition table 145 a and output a pre-utterance as to whether the current passenger is the same passenger in the previous travel.

[0515] Specifically, the dialogue system 100 can determine that a passenger has boarded the vehicle based on the dialogue between the passengers in the vehicle and the vehicle operation information. Specifically, the voice input processor 111 of the input processor 110 can determine that a passenger has boarded the vehicle by receiving the dialogue between the passengers in the vehicle, and obtain features such as voice features, seat position, boarding time, and boarding position of each passenger.

[0516] The dialogue system 100 may determine whether the passenger characteristics are the same as the stored passenger characteristics. Specifically, the voice input processor 111 of the dialogue system 100 may compare the characteristics of the passenger acquired from the storage device 140 with the passenger characteristics (such as voice characteristics, seat position, boarding time and boarding position) of the passenger determined to have boarded the vehicle.

[0517] For example, the voice input processor 111 may compare the voice characteristics, seat position, boarding time and boarding position of the passenger contained in the passenger information with the characteristics of the passenger who is determined to have boarded the vehicle.

[0518] When at least two of the passenger's voice characteristics, seat position, boarding time and boarding position contained in the passenger information are the same as the detected passenger characteristics, the voice input processor 111 can determine that the characteristics of the passenger determined to have boarded the vehicle are the same as the passenger information.

[0519] When comparing the passenger's voice features, seat position, boarding time and boarding position contained in the passenger information with the features of the detected passenger, the voice input processor 111 can determine similar voice features, seat position, boarding time and boarding position within a certain range as the same.

[0520] When it is determined that the characteristics of the passenger determined to have boarded the vehicle are identical to the stored passenger information, the dialogue system 100 may output a preliminary utterance to verify whether the passenger participated in a previous trip.

[0521] For example, when it is determined that the characteristics of the passenger boarding the vehicle are the same as the stored passenger information, the dialogue system 100 may output a preliminary utterance to verify whether the passenger is the same as the passenger in the previous travel, such as “Are you 00?”.

[0522] The pre-speech determiner 151 can determine whether to output the pre-speech based on the context information related to whether the passenger has boarded the vehicle and the pre-speech condition related to the characteristics of the passenger determined to have boarded the vehicle being the same as the stored passenger information. In addition, when the context information related to whether the passenger has boarded the vehicle satisfies the pre-speech condition related to the characteristics of the passenger determined to have boarded the vehicle being the same as the stored passenger information, the pre-speech determiner 151 can determine it as a pre-speech context and generate a pre-speech trigger signal.

[0523] The pre-speech determiner 151 may obtain a pre-speech message corresponding to the pre-speech context of the passenger boarding the vehicle, such as “Are you 00?” When the pre-speech determiner 151 sends the pre-speech trigger signal and the pre-speech message to the dialogue input manager 111c, the dialogue input manager 111c may send the pre-speech message to the dialogue manager 120. At this time, the pre-speech trigger signal or the signal indicating the pre-speech context may be sent together with the pre-speech message.

[0524] The dialog manager 120 may generate a dialog task for outputting the transmitted pre-utterance message, and transmit the pre-utterance message to the result processor 130. The result processor 130 may output the input pre-utterance message via the speaker 232.

[0525] The dialogue system 100 can verify whether the passenger participated in the previous travel by receiving the utterance of the passenger in the vehicle. Specifically, the dialogue system 100 can verify whether the passenger participated in the previous travel by receiving the utterance of the passenger corresponding to the pre-utterance message.

[0526] For example, the passenger may say "yes" or "no" in response to a pre-speech message asking whether the passenger participated in the previous trip. That is, the passenger may say a message including a response indicating whether the passenger participated in the previous trip in response to a pre-speech message asking whether the passenger participated in the previous trip.

[0527] When the passenger's speech is input, the voice input processor 111 recognizes the input passenger's speech in the vehicle. The passenger's speech may be input through the voice input device 210 provided in the vehicle 200 or the voice input device 410 provided in the mobile device 400.

[0528] The speech recognizer 111a may recognize an input user's utterance and output the utterance in text form. The natural language understanding section 111b may apply a natural language understanding technique to the utterance in text form and output a result of the natural language understanding.

[0529] Specifically, the natural language understanding process may include performing morpheme analysis on the utterance in text form, and identifying whether the passenger participated in a previous trip based on the result of the morpheme analysis.

[0530] Therefore, the passenger determiner 111d of the dialogue system 100 can verify whether the passenger participated in the previous travel based on the utterance of the occupant in the vehicle.

[0531] When it is determined that the passenger participated in the previous trip, the dialogue system 100 can generate passenger quantity information based on the dialogue between the occupants in the vehicle and the stored passenger information. That is, when generating passenger quantity information based on the dialogue between the occupants in the vehicle, the dialogue system 100 can additionally consider the stored passenger information.

[0532] For example, the passenger determiner 111d of the dialogue system 100 can estimate the change in the number of passengers at the stop point based on the output of the natural language understanding part 111b. Specifically, the passenger determiner 111d can estimate the possibility of each passenger leaving the vehicle and the possibility of each passenger boarding the vehicle again after leaving based on the dialogue between the occupants in the vehicle, and the passenger determiner 111d can estimate the number of potential passengers boarding the vehicle based on the call session in the vehicle.

[0533] When estimating a change in the number of passengers based on conversations between occupants in the vehicle, the passenger determiner 111d may increase the accuracy of the estimation result of the change in the number of passengers by using departure time information and information related to a location of departure from the vehicle among the stored passenger information.

[0534] Specifically, when it is estimated that a passenger will leave at a specific stop point based on a conversation between occupants in the vehicle, the passenger determiner 111d can verify the departure time and departure location of the vehicle in the previous trip by using the departure time information and information related to the departure location in the stored passenger information.

[0535] The passenger determiner 111d may determine whether a specific stop point estimated to be left by a passenger is the same as a left position in a previous trip based on a conversation between occupants in the vehicle.

[0536] When a specific stop point from which passengers estimated to leave based on conversations between occupants in the vehicle is the same as a departure location in previous travel, the passenger determiner 111d may generate passenger information by estimating a change in the number of passengers estimated based on conversations between occupants in the vehicle.

[0537] When the specific stop point where the passenger leaves estimated based on the conversation between occupants in the vehicle is different from the departure position in the previous travel, the passenger determiner 111d can verify whether the departure position is the specific stop point position by speaking a pre-speech to the passenger, and generate passenger information by utilizing the passenger's speech.

[0538] Fig.26 A control block diagram showing the configuration of the dialog manager in detail is shown below. Fig. 27 A diagram showing an example of information stored in a relationship action DB. Fig.28 is a view showing an example of information stored in the action execution condition DB, and Fig.29 is a view showing an example of information stored in the action parameter DB.

[0539] refer to Fig.26 , the dialog manager 120 may include: a dialog flow manager 121 that requests to generate, delete, and update dialogs or actions; a dialog action manager 122 that generates, deletes, and updates dialogs or actions according to the request of the dialog flow manager 121; a fuzzy resolver 123 that clarifies the user's intention by resolving the ambiguity of the context and the ambiguity of the dialog; a parameter manager 124 that manages the parameters required for action execution; an action priority determiner 125 that determines whether an action is executable with respect to multiple candidate actions; and an external information manager 126 that manages an external content list and related information, and manages parameter information for external content queries.

[0540] The dialog manager 120 may include: a memory in which a program for performing the above-mentioned operations and operations described later is stored; and a processor for executing the stored program. At least one memory and one processor may be provided, and when multiple memories and processors are provided, they may be integrated on one chip or physically separated.

[0541] Each component included in the dialog manager 120 may be implemented by the same processor or separate processors.

[0542] Additionally, the dialog manager 120 and the input processor 110 may be implemented by the same processor or by separate processors.

[0543] When a user utterance is input or when a user utterance matching a pre-utterance context is sent to the natural language understanding part 111b, the dialogue input manager 111c may send the result of natural language understanding (the output of the natural language understanding part) and context information (the output of the context understanding part) to the dialogue flow manager 121. In addition, when a pre-utterance context occurs, the dialogue input manager 111c may send a pre-utterance trigger signal.

[0544] The output of the natural language understanding part 111b may include information related to the user's utterance content, such as morpheme analysis results, and information such as domain and action. The output of the context understanding part 112c may include events and context information determined by the context information collection manager 112b.

[0545] The dialog flow manager 121 may search whether a dialog task or an action task corresponding to the input by the dialog input manager 111 c exists in the dialog and action state DB 147 .

[0546] The dialogue and action state DB 147 may be a storage space for managing dialogue states and action states, so the dialogue and action state DB 147 may store currently ongoing dialogues and actions, as well as dialogue states and action states related to preliminary actions to be processed. For example, the dialogue and action state DB 147 may store states related to completed dialogues and actions, stopped dialogues and actions, ongoing dialogues and actions, and dialogues and actions to be processed.

[0547] The dialog and action status DB 147 may store the latest output status regarding whether switching and nesting, switching action index, action change time, and screen / voice / command.

[0548] For example, in the case of extracting domains and actions corresponding to user utterances, when there are dialogues and actions corresponding to the corresponding domains and actions in the most recently stored dialogues, the dialogue and action status DB 147 can determine it as a dialogue task or an action task corresponding to the input from the dialogue input manager 111c.

[0549] When the domain and action corresponding to the user utterance are not extracted, the dialog and action state DB 147 may request the dialog action manager 122 to generate a random task or refer to a recently stored task.

[0550] When a dialog task or an action task corresponding to the input of the input processor 110 does not exist in the dialog and action state DB 147 , the dialog flow manager 121 may request the dialog action manager 122 to generate a new dialog task or action task.

[0551] In addition, when a pre-discourse trigger signal is sent from the input processor 110, although there is a currently executed dialogue task or action task, the dialogue task or action task can be temporarily stopped, and a dialogue task or action task corresponding to the pre-discourse context can be generated first. In addition, the priority can be selected according to the established rules.

[0552] When a pre-speech trigger signal and an action corresponding to the pre-speech trigger signal are input from the dialog input manager 111c, the dialog flow manager 121 can request the dialog action manager 122 to generate a new dialog task or action task in the same way as the case of obtaining the action from the user speech.

[0553] Furthermore, when a pre-discourse trigger signal and a pre-discourse message corresponding to the pre-discourse trigger signal are input from the dialog input manager 111 c , the dialog flow manager 121 may request the dialog action manager 122 to generate a new dialog task or action task for outputting the input pre-discourse message.

[0554] When the dialog flow manager 121 manages the dialog flow, the dialog flow manager 121 may refer to the dialog policy DB 148. The dialog policy DB 148 may store policies for continuing a dialog, wherein the policies may represent policies for selecting, starting, suggesting, stopping, and terminating a dialog.

[0555] In addition, the dialogue strategy DB 148 may store strategies regarding the timing and method of the system outputting a response. The dialogue strategy DB 148 may store strategies for generating a response by linking a plurality of services and strategies for deleting a previous action and replacing an action with another action.

[0556] For example, two policies may be allowed, which may include: a policy that generates responses to two actions immediately, for example, "Is it necessary to perform action B after performing action A?"; and a policy that generates a separate response to one action after generating a response to another action, for example, "Perform action A" → "Do you want to perform action B?".

[0557] The dialog and action state DB 147 may store a strategy for determining priorities among candidate actions. The priority determination strategy will be described later.

[0558] The dialog action manager 122 may designate a storage space to the dialog and action state DB 147 and generate a dialog task and an action task corresponding to an output of the input processor 110 .

[0559] When the domain and action cannot be extracted from the user's utterance, the dialogue action manager 122 can generate a random dialogue state. In this case, as described later, the fuzzy solver 123 can recognize the user's intention based on the content of the user's utterance, environmental conditions, vehicle status, and user information, and determine an action suitable for the user's intention.

[0560] When a dialog task or an action task corresponding to the output of the input processor 110 exists in the dialog and action state DB 147 , the dialog flow manager 121 may request the dialog action manager 122 to refer to the corresponding dialog task or action task.

[0561] The action priority determiner 125 may search the relational action DB 146b to search for an action list related to an action or event included in the output of the input processor 110, and then the action priority determiner 125 may obtain a candidate action. Fig. 27 As shown in , the relational action DB 146b can indicate actions related to each other, the relationships between these actions, actions related to events, and the relationships between events. For example, route guidance, vehicle status check, and gas station recommendation can be classified as relational actions, and the relationships therein can correspond to associations.

[0562] Therefore, when executing route guidance, the vehicle status check and gas station recommendation can be performed together. In this case, "performed together" can include the case where the vehicle status check and gas station recommendation are performed before or after the route guidance, and the case where the vehicle status check and gas station recommendation are performed during the route guidance (for example, added as a stop point).

[0563] The warning light output event may be stored as an event action related to the repair shop guiding action, and the relationship between them may correspond to an association.

[0564] When a warning light output event occurs, a repair shop guidance action may be performed according to the warning light type or whether repair is required.

[0565] When the input processor 110 transmits the action corresponding to the user's utterance together with the event determined by the context information collection manager 112b, the action related to the action corresponding to the user's utterance and the action related to the event may become candidate actions.

[0566] The extracted candidate action list may be transmitted to the dialog action manager 122, and the dialog action manager 122 may update the action status of the dialog and action status DB 147 by adding the candidate action list.

[0567] The action priority determiner 125 may search the action execution condition DB 146 c for a condition for executing each candidate action.

[0568] like Fig.28 As shown, the action execution condition DB 146c may store, according to each action, conditions required for executing the action, and parameters for determining whether the corresponding conditions are satisfied.

[0569] For example, the execution condition for the vehicle status check may be a case where the destination distance is equal to or greater than 100 km, wherein the parameter for determining the condition may correspond to the destination distance. The condition for a gas station recommendation may be a case where the destination distance is greater than the distance to empty (DTE), wherein the parameter for determining the condition may correspond to the destination distance and the distance to empty (DTE).

[0570] The action priority determiner 125 may transmit the execution conditions of the candidate actions to the dialog action manager 122 , and the dialog action manager 122 may add the execution conditions according to each candidate action and update the action status of the dialog and action status DB 147 .

[0571] The action priority determiner 125 can search for parameters required to determine the action execution conditions (hereinafter referred to as condition determination parameters) from the context information DB 142, long-term storage 143, short-term storage 144 or dialogue and action status DB 147, and use the searched parameters to determine whether the candidate action can be executed.

[0572] When parameters for determining action execution conditions are not stored in the context information DB 142 , the long-term memory 143 , the short-term memory 144 , or the dialog and action state DB 147 , the action priority determiner 125 may acquire required parameters from the external content server 300 via the external information manager 126 .

[0573] The action priority determiner 125 may determine whether the candidate action can be executed using parameters for determining action execution conditions. In addition, the action priority determiner 125 may determine the priority of the candidate action based on whether the candidate action can be executed and the priority determination rules stored in the dialogue strategy DB 148.

[0574] The score of each candidate action can be calculated according to the current situation. Candidate actions with higher calculated scores can be given higher priority. For example, actions corresponding to user utterances, safety scores, convenience scores, processing time, processing time points (whether immediate processing is required), user preferences (user acceptance of suggested services or preferences predetermined by users), administrator scores, vehicle status-related scores, and action success rates (dialogue success rates) can be used as parameters for calculating scores, as shown in Equation 1 below. w1, w2, w3, w4, w5, w6, w7, w8, and w9 represent weight values ​​of each parameter.

[0575] [Equation 1]

[0576] Priority score = w1*user speech action + w2*safety score + w3*convenience score + w4*processing time + w5*processing time point + w6*user preference + w7*administrator score + w8*score related to vehicle status + w9*action success rate*possibility of action execution (1: possible, not yet known, 0: impossible)*action completion status (completed: 1, not completed: 0).

[0577] As described above, the action priority determiner 125 can provide the user with the most needed service by searching for actions directly associated with the user's utterance and context information and an action list related thereto, and by determining the priorities among them.

[0578] The action priority determiner 125 may transmit the likelihood and priority of the candidate action execution to the dialog action manager 122 , and the dialog action manager 122 may update the action status of the dialog and action status DB 147 by adding the transmitted information.

[0579] The parameter manager 124 may search the action parameter DB 146 a for parameters for performing each candidate action (hereinafter referred to as action parameters).

[0580] like Fig.29 As shown in , the action parameter DB 146a can store necessary parameters, optional parameters, initial values ​​of parameters, and reference positions for acquiring parameters according to each action. In a state where the initial values ​​of the parameters are stored, when there is no parameter value corresponding to the corresponding parameter in the user's speech and context information output from the input processor 110 and there is no parameter value in the context information DB 142, the action can be performed according to the stored initial value, or whether to perform the action can be confirmed to the user according to the stored initial value.

[0581] For example, the necessary parameters for route guidance may include the current location and the destination, and the optional parameters may include the type of route. The initial values ​​of the optional parameters may be stored as a quick route. The current location and the destination may be acquired by sequentially searching the dialogue and action state DB 147, the context information DB 142, the short-term memory 144, or the long-term memory 143.

[0582] The necessary parameters for vehicle state inspection may include vehicle state information, and the optional parameters may include a portion to be inspected (hereinafter referred to as an "inspection portion"). The entire portion (whole) may be stored as an initial value of the optional parameter. The vehicle state information may be acquired from the context information DB 142.

[0583] The selective parameters of the gas station recommendation may include a favorite gas station, and "A gas station" may be stored as an initial value of the selective parameters. The favorite gas station may be retrieved from the long-term memory 143. The selective parameters may also include the fuel type and fuel price of the vehicle.

[0584] As described above, parameter manager 124 can obtain parameter values ​​of parameters searched in action parameter DB 146a from corresponding reference locations. The reference location for obtaining parameter values ​​can be at least one of context information DB 142, short-term memory 144 or long-term memory 143, dialogue and action state DB 147, and external content server 300.

[0585] The parameter manager 124 may acquire the parameter value from 300 via the external information manager 126. The external information manager 126 may determine where to acquire the information by referring to the external service set DB 146d.

[0586] The external service set DB 146d may store information about external content servers connected to the dialog system 100. For example, the external service set DB 146d may store external service names, descriptions about external services, types of information provided from external services, external service usage methods, and entities providing external services.

[0587] The parameter values ​​acquired by the parameter manager 124 may be transmitted to the dialog action manager 122 , and the dialog action manager 122 may update the dialog and action state DB 147 by adding the initial values ​​to the action states according to the candidate actions.

[0588] The parameter manager 124 may acquire parameter values ​​of all candidate actions, or the parameter manager 124 may acquire parameter values ​​of only candidate actions determined to be executable by the action priority determiner 125 .

[0589] The parameter manager 124 can selectively use parameter values ​​of different types indicating the same information. For example, by using the destination search service of the navigation system, "Seoul Station" indicating the destination and in text form can be converted into "Seoul Station" in the form of POI.

[0590] When there is no ambiguity in the conversation and context, the required information can be acquired and the conversation and action can be managed according to the above-mentioned operations of the action priority determiner 125, the parameter manager 124, and the external information manager 126. When there is ambiguity in the conversation and context, it may be difficult to provide the service required by the user using only the operations of the action priority determiner 125, the parameter manager 124, and the external information manager 126.

[0591] In this case, the ambiguity resolver 123 can handle ambiguity in the conversation or in the context. For example, when the conversation contains anaphora (e.g., that person, that place yesterday, father, mother, grandmother, and daughter-in-law), there may be ambiguity because it is unclear who or which one the anaphora represents. In this case, the ambiguity resolver 123 can resolve the ambiguity by referring to the context information DB 142, the long-term memory 143, or the short-term memory 144, or provide guidance for resolving the ambiguity.

[0592] For example, ambiguous words included in "that place yesterday", "the market near the residence", and "Seoul Station where I went yesterday" may correspond to parameter values ​​of action parameters or parameter values ​​of condition determination parameters. However, in this case, due to the ambiguity of the words, it is not possible to perform an actual action or determine an action execution condition by using the corresponding words.

[0593] The ambiguity resolver 123 may resolve the ambiguity of the parameter value by referring to information stored in the context information DB 142, the long-term memory 143, or the short-term memory 144. The ambiguity resolver 123 may acquire required information from the external content server 300 by using the external information manager 126 as needed.

[0594] For example, the fuzzy solver 123 can search for a place that the user went yesterday by referring to the short-term memory 144, so as to convert "that place yesterday" into information that can be used as the destination of the route guidance action. The fuzzy solver 123 can search for the user's residential address by referring to the long-term memory 143, and obtain location information related to the A market near the user's residential address from the external content server 300. Therefore, the fuzzy solver 123 can convert "the market near the residence" into information that can be used as the destination of the route guidance action.

[0595] When the input processor 110 does not clearly extract an action (object and operator) or when the user's intention is unclear, the fuzzy resolver 123 may recognize the user's intention by referring to the fuzzy resolution information DB 146e and determine an action corresponding to the recognized intention.

[0596] Fig.30 is a table showing an example of information stored in the fuzzy resolution information DB.

[0597] Based on the vehicle state information and the surrounding environment information, the fuzzy parsing information DB 146e can match the speech with the action corresponding to the speech and then store the speech and the action. The speech stored in the fuzzy parsing information DB 146e may be a speech for which the action cannot be extracted through natural language understanding. Fig.30 The utterance content according to the morpheme analysis result is shown as "my hands are freezing" or "my hands are cold".

[0598] The surrounding environment information may include the outdoor temperature of the vehicle and whether it is raining, and the vehicle status information may include the on / off of the air conditioner and heater, the air volume and direction of the air conditioner, and the on / off of the steering wheel heating wire.

[0599] Specifically, when the outdoor temperature exceeds 20 degrees during rainy weather and the air conditioner is turned on, it can be recognized that the hands are freezing because the air conditioner temperature is set to a low level, so "increase the air conditioner temperature by 3 degrees" can be stored as the corresponding vehicle control action.

[0600] In a state where the outdoor temperature exceeds 20 degrees when it is raining, when the air conditioner is turned OFF, it can be recognized that the user feels cold due to the rain, and thus "heater on" can be stored as a vehicle control action corresponding thereto.

[0601] When the outdoor temperature is over 20 degrees without raining, when the air conditioner is turned on (ON) and the wind direction of the air conditioner is upward, it can be recognized that the hands are freezing because the wind of the air conditioner directly affects the hands, so "changing the wind direction of the air conditioner to the lower side" can be stored as the corresponding vehicle control action.

[0602] When the outdoor temperature exceeds 20 degrees without raining, when the air conditioner is turned on (ON), the wind direction of the air conditioner is downward and the air volume is set to exceed the middle level, it can be recognized that the user feels cold due to the excessive air volume of the air conditioner, so "reduce the air volume of the air conditioner" can be stored as the corresponding vehicle control action.

[0603] When the outdoor temperature exceeds 20 degrees without raining, when the air conditioner is turned on (ON), the wind direction of the air conditioner is downward and the wind volume is set to weak, "increase the air conditioner temperature by 3 degrees" can be stored as the corresponding vehicle control action.

[0604] When the heater is turned OFF when the outdoor temperature is below 20 degrees, it can be recognized that the hands are freezing due to the cold weather, and thus “turn on the heater” can be stored as the corresponding vehicle control action.

[0605] When the outdoor temperature is below 20 degrees, when the heater is turned on (ON) and the steering wheel heating wire is turned off, it can be recognized that the hands will be frozen because the hot air will not be transferred to them, and therefore, "steering wheel heating wire on" can be stored as the corresponding vehicle control action.

[0606] When the outdoor temperature is below 20 degrees, when the heater and the steering wheel heating wire are turned on (ON) and the wind direction of the heater is downward, it can be recognized that the hands will be frozen because the wind from the heater will not be transmitted to the hands, and therefore "changing the wind direction of the heater to two-way" can be stored as the corresponding vehicle control action.

[0607] When the outdoor temperature is below 20 degrees, the heater and steering wheel heating wire are turned on (ON), and the wind direction of the heater is upward. When the heater temperature is set to be lower than the maximum temperature, "increasing the temperature of the heater" can be stored as the corresponding vehicle control action.

[0608] When the outdoor temperature is below 20 degrees, the heater and steering wheel heating wire are turned on (ON), the wind direction of the heater is upward, and the heater temperature is set to the highest. When the capacity of the heater is not set to the highest, "increase the air volume of the heater" can be stored as the corresponding vehicle control action.

[0609] When the outdoor temperature is below 20 degrees, the heater and steering wheel heating wire are turned on (ON), the wind direction of the heater is upward, and the heater temperature and the heater air volume are set to the highest. When the seat heating wire is turned off, "turn on the seat heating wire" can be stored as the corresponding vehicle control action.

[0610] When the outdoor temperature is below 20 degrees, the heater and steering wheel heating wire are turned on (ON), the wind direction of the heater is upward, and the heater temperature and the heater air volume are set to the highest. When the seat heating wire is turned on, "notification to wait for a while because the heater is now in full operation (full load) state" can be stored as the corresponding vehicle control action.

[0611] Fig.31A and Fig.31B is a table showing various examples of vehicle control performed as a result of the fuzzy resolver resolving ambiguity and extracting actions by referring to the fuzzy resolution information DB.

[0612] For example, Fig.31A and Fig.31B As shown, in a state where the utterance content according to the morpheme analysis result is that the hands are freezing or the hands are cold, when the surrounding environment is summer, the vehicle state is that the wind direction of the air conditioner is on the upper side (upper side) of the passenger's head, the air conditioner set temperature is 19 degrees, and the air volume of the air conditioner is at a high level, it can be recognized that the hands are freezing due to the air direction of the air conditioner. The air conditioner control action for reducing the air volume intensity while changing the wind direction to the foot side (lower side) can be extracted as an action corresponding to the utterance, and the vehicle can be controlled according to the extracted action.

[0613] In an utterance with the same content, when the surrounding environment is winter, the vehicle state is that the air conditioner wind direction is the passenger's feet, the air conditioner set temperature is 25 degrees, and the air volume of the air conditioner is at a high level, it can be recognized that the hands are freezing because the hot air is not transferred to the hands. The "turn on the steering wheel heating wire" action can be extracted as an action corresponding to the utterance, and the vehicle can be controlled according to the extracted action.

[0614] In a state where the content of the speech according to the result of morpheme analysis is "irritated", when the vehicle speed is 30 km or less and the front and rear gap is less than 30 cm, it can be identified that the irritation is caused by heavy traffic. Therefore, "changing route options in route guidance actions (fast route guidance)", "playing multimedia content such as music", or "turning on the chat function" can be extracted as actions corresponding to the speech, and the vehicle can be controlled according to the extracted actions.

[0615] In a state where the content of the speech according to the result of morpheme analysis is "drowsy", when the vehicle state is in the inside air mode, it can be recognized that the drowsiness is caused by the lack of air circulation. Therefore, "change to outside air mode" can be extracted as an action corresponding to the speech, and the vehicle can be controlled according to the extracted action.

[0616] In an utterance with the same content, when the vehicle state is in the outside air mode and the heater is turned on (ON), it can be recognized that drowsiness is caused by hot air from the heater. "Open the window" can be extracted as an action corresponding to the utterance, and the vehicle can be controlled according to the extracted action.

[0617] In a state where the content of the speech according to the result of morpheme analysis is "sweating" or "hot", when the surrounding environment is winter and the heater is turned on (ON), it can be recognized that the heat is caused by the hot air emitted by the heater. Therefore, "lower the heater temperature" or "reduce the air volume" can be stored as the action corresponding to the speech.

[0618] In an utterance with the same content, when the surrounding environment is winter and the heater is OFF, it can be recognized that the heat is caused by the user's body heat. Therefore, "open the window" or "suggest opening the window" can be extracted as an action corresponding to the utterance, and the vehicle can be controlled according to the extracted action.

[0619] In the utterance with the same content, when the surrounding environment is summer and the air conditioner is turned off (OFF), it can be recognized that the heat is caused by the increase in the internal temperature of the vehicle. Therefore, "turn on the air conditioner" can be extracted as an action corresponding to the utterance, and the vehicle can be controlled according to the extracted action.

[0620] In an utterance with the same content, when the surrounding environment is summer and when the air conditioner is turned on (ON), it can be recognized that the heat is caused by the air conditioner temperature set to high. Therefore, "lower the air conditioner temperature" or "increase the air volume of the air conditioner" can be extracted as an action corresponding to the utterance, and the vehicle can be controlled according to the extracted action.

[0621] In a state where the content of the speech according to the result of morpheme analysis is "cold", when the surrounding environment is summer and the air conditioner is turned on (ON), it can be recognized that the cold is caused by the air conditioner temperature being set too low or the air conditioner being too strong. Therefore, "increase the air conditioner temperature" or "reduce the air volume" can be extracted as an action corresponding to the speech, and the vehicle can be controlled according to the extracted action.

[0622] In the speech with the same content, when the surrounding environment is summer and when the air conditioner is OFF, it can be recognized that the cold is caused by the user's physical condition. "Operate the heater" or "check the user's biorhythm" can be extracted as an action corresponding to the speech, and the vehicle can be controlled according to the extracted action.

[0623] In an utterance with the same content, when the surrounding environment is winter and the heater is turned on (ON), it can be recognized that the cold is caused by the heater temperature being set to low or the wind volume being weak. Therefore, "increase heater temperature" or "increase wind volume" can be extracted as an action corresponding to the utterance, and the vehicle can be controlled according to the extracted action.

[0624] In the utterance with the same content, when the surrounding environment is winter and the heater is turned off (off), it can be recognized that the cold is caused by the non-operation of the heater. "Operate the heater" can be extracted as an action corresponding to the utterance, and the vehicle can be controlled according to the extracted action.

[0625] In a state where the content of the speech according to the result of morpheme analysis is "headache", when the surrounding environment is winter and the heater is turned on (ON), it can be recognized that the headache is caused by lack of air circulation. Therefore, "change to outside air mode" or "open the window" can be extracted as an action corresponding to the speech, and the vehicle can be controlled according to the extracted action.

[0626] In an utterance with the same content, when the surrounding environment is winter and the heater is OFF, it can be recognized that the headache is caused by the cold. "Operate the heater" can be extracted as an action corresponding to the utterance, and the vehicle can be controlled according to the extracted action.

[0627] In the utterance with the same content, when the surrounding environment is summer and the air conditioner is OFF, it can be recognized that the headache is caused by heat. "Operate the air conditioner" can be extracted as an action corresponding to the utterance, and the vehicle can be controlled according to the extracted action.

[0628] In an utterance with the same content, when the surrounding environment is summer and the air conditioner is turned on (ON), it can be recognized that the headache is caused by air conditioning. "Change the wind direction or wind volume of the air conditioner" can be extracted as an action corresponding to the utterance, and the vehicle can be controlled according to the extracted action.

[0629] In a state where the content of the speech according to the result of morpheme analysis is "uncomfortable", when the surrounding environment is winter and it is raining, it can be recognized that the discomfort is caused by high humidity. Therefore, "operating the defogger function" or "operating the dehumidification function" can be extracted as an action corresponding to the speech, and the vehicle can be controlled according to the extracted action.

[0630] In an utterance with the same content, when the surrounding environment is summer and it is not raining, it can be recognized that the discomfort is caused by seasonal characteristics and heat. Therefore, "operate the air conditioner at the lowest temperature" can be extracted as an action corresponding to the utterance, and the vehicle can be controlled according to the extracted action.

[0631] In an utterance with the same content, when the surrounding environment is summer and it is raining, it can be recognized that the discomfort is caused by heat and high humidity. Therefore, "operating the air conditioner in dehumidification mode" can be extracted as an action corresponding to the utterance, and the vehicle can be controlled according to the extracted action.

[0632] According to the operation of the above-mentioned fuzzy resolver 123, although there is ambiguity in the user's speech or situation, the fuzzy resolver 123 can accurately identify the action actually desired by the user, or the action actually required by the user by comprehensively considering the surrounding environment information and vehicle status information of the user's speech, and provide the desired action and the required action.

[0633] Information related to the action determined by the fuzzy resolver 123 may be transmitted to the dialog action manager 122 , and the dialog action manager 122 may update the dialog and action state DB 147 based on the transmitted information.

[0634] As described above, the action priority determiner 125 and the parameter manager 124 may determine the action execution conditions regarding the actions determined by the fuzzy resolver 123 , determine the priorities thereof, and acquire parameter values.

[0635] When all values ​​that can be obtained through the current context and dialog among parameter values ​​for executing each action are obtained, the dialog action manager 122 may send a signal to the dialog flow manager 121 .

[0636] When necessary parameter values ​​for action execution and condition determination do not exist in the dialog and action status DB 147, the external content server 300, the long-term storage 143, the short-term storage 144 and the context information DB 142, and these necessary parameters can only be obtained through the user, the result processor 130 can generate a dialog response for inquiring the user about the parameter values.

[0637] The dialog flow manager 121 may transmit information related to the action corresponding to the first priority action and the dialog state to the result processor 130. In addition, the dialog flow manager 121 may transmit information related to a plurality of candidate actions according to the dialog policy.

[0638] When the dialog system 100 outputs a pre-discourse, that is, when a pre-discourse trigger signal is generated by the input processor 110, the dialog state sent from the result processor 130 may include the pre-discourse trigger signal. However, the pre-discourse trigger signal is not required to be included in the dialog state, but any type of information may be included in the dialog state as long as the pre-discourse context is indicated. When the information indicating the pre-discourse context is included in the dialog state, the result processor 130 may give priority to other types of responses and output the dialog response first, or output the dialog response and other types of responses.

[0639] In the state where the dialogue system 100 outputs pre-discourse, when a pre-discourse message corresponding to a pre-discourse context is input from the dialogue input manager 111c, the pre-discourse message can be sent to the result processor 130 without the above-mentioned ambiguity resolution, parameter management and action priority determination processes.

[0640] In the state where the dialogue system 100 outputs pre-discourse, when an action corresponding to the pre-discourse context is input from the dialogue input manager 111c, the pre-discourse message can be sent to the result processor 130 with or without going through the above-mentioned ambiguity resolution, parameter management and action priority determination processes.

[0641] Fig.32 A control block diagram showing the configuration of the result processor in detail.

[0642] refer to Fig.32The result processor 130 may include: a response generation manager 131, which manages the generation of a response required to execute an action input from the dialogue manager 120; a dialogue response generator 132, which generates a text, image or audio type response according to a request of the response generation manager 131; a command generator 136, which generates a command for vehicle control or providing a service using external content according to a request of the response generation manager 131; a service editor 134, which executes multiple services sequentially or occasionally and collects their results to provide a service desired by a user; an output manager 133, which outputs the generated text type response, image type response or audio type response, outputs the command generated by the command generator 136, or determines the order of the output when there are multiple outputs; and a memory manager 135, which manages the long-term memory 143 and the short-term memory 144 based on the outputs of the response generation manager 131 and the output manager 133.

[0643] The result processor 130 may include: a memory in which a program for performing the above-mentioned operations and operations described later is stored; and a processor for executing the stored program. At least one memory and one processor may be provided, and when multiple memories and processors are provided, they may be integrated on a single chip or physically separated.

[0644] Each component included in the result processor 130 may be implemented by the same processor or separate processors.

[0645] Additionally, the result processor 130, the dialog manager 120, and the input processor 110 may be implemented by the same processor or by separate processors.

[0646] The response outputted by corresponding to the user's speech or context may include a dialogue response, vehicle control, and external content provision. The dialogue response may include an initial dialogue, an inquiry, and an answer including information. The dialogue response may be stored in a response template 149 as a database.

[0647] The response generation manager 131 may request the dialog response generator 132 and the command generator 136 to generate a response required to perform an action determined by the dialog manager 120. To this end, the response generation manager 131 may send information related to the action to be performed to the dialog response generator 132 and the command generator 136, wherein the information related to the action to be performed may include an action name and parameter values. When generating a response, the dialog response generator 132 and the command generator 136 may refer to the current dialog state and the action state.

[0648] The dialogue response generator 132 may acquire a dialogue response template by searching the response template 149, and generate a dialogue response by filling the extracted dialogue response template with parameter values. The generated dialogue response may be transmitted to the response generation manager 131. When parameter values ​​required to generate a dialogue response are not transmitted from the dialogue manager 120 or when an introduction using external content is transmitted, the dialogue response generator 132 may receive parameter values ​​from the external content server 300 or search the long-term storage 143, the short-term storage 144, or the context information DB 142.

[0649] For example, when the action determined by the dialog manager 120 corresponds to route guidance, the dialog response generator 132 may search for the response template 149 and then extract the dialog response template “It takes [duration: -] to get from [current location: -] to [destination: -]. Start guiding?”.

[0650] [Current location] and [Destination] among the parameters to be filled in the dialogue response template may be sent from the dialogue manager 120, and the parameter value of [Duration] may not be sent. In this case, the dialogue response generator 132 may request the duration taken from [Current location] to [Destination] from the external content server 300.

[0651] When the response to the user's speech or context includes vehicle control or external content provision, the command generator 136 may generate a command for executing the vehicle control or external content provision. For example, when the action determined by the dialogue manager 120 is control of an air conditioner, a window, a seat, and an AVN, the command generator 136 may generate a command for executing the control and then send the command to the response generation manager 131.

[0652] When the action determined by the dialog manager 120 requires external content provision, the command generator 136 may generate a command for receiving corresponding content from the external content server 300 and then transmit the command to the response generation manager 131 .

[0653] When multiple commands are provided by the command generator 136 , the service editor 134 may determine a method and order of executing the multiple commands and send the method and order to the response generation manager 131 .

[0654] The response generation manager 131 may transmit a response transmitted from the dialog response generator 132 , the command generator 136 , or the service editor 134 to the output manager 133 .

[0655] The output manager 133 may determine the output timing, output order, and output location of the dialog responses generated by the dialog response generator 132 and the commands generated by the command generator 136 .

[0656] The output manager 133 may output the response by sending the dialogue response generated by the dialogue response generator 132 and the command generated by the command generator 136 to the appropriate output location in an appropriate order at an appropriate timing. The output manager 133 may output a text-to-speech (TTS) response via the speaker 232, and output a text response via the display 231. When outputting a TTS type dialogue response, the output manager 133 may use a TTS module provided in the vehicle 200, or alternatively, the output manager 133 may include the TTS module.

[0657] According to the control target, the command may be transmitted to the vehicle controller 240 or the communication device 280 for communicating with the external content server 300 .

[0658] The response generation manager 131 may also transmit a response transmitted from the dialog response generator 132 , the command generator 136 , or the service editor 134 to the memory manager 135 .

[0659] The output manager 133 may send a response of its own output to the memory manager 135 .

[0660] The memory manager 135 may manage the long-term memory 143 or the short-term memory 144 based on the content sent from the response generation manager 131 and the output manager 133. For example, the memory manager 135 may update the short-term memory 144 by storing the conversation content between the user and the system based on the generated and output conversation response. The memory manager 135 may update the long-term memory 143 by storing information related to the user acquired through the conversation with the user.

[0661] Among the information stored in the short-term memory 144 , persistent information (eg, preference or orientation of a user) or information for acquiring the persistent information may be stored in the long-term memory 143 .

[0662] Based on the vehicle control and external content requests corresponding to the generated and output commands, the user preferences or vehicle control history stored in the long-term memory 143 may be updated.

[0663] Meanwhile, in a state where the dialogue system 100 outputs a pre-utterance before the user inputs an utterance, when an action corresponding to the pre-utterance context is input from the dialogue input manager 111c, the dialogue response generator 132 receiving information related to the action can obtain a dialogue response template by searching for a response template 149, and generate a dialogue response by filling the extracted dialogue response template with parameter values. The generated dialogue response can be sent to the response generation manager 131. The dialogue response can become the pre-utterance of the dialogue system 100.

[0664] The response generation manager 131 may transmit the dialog response transmitted from the dialog response generator 132 to the output manager 133 .

[0665] The output manager 133 may output the dialogue response generated by the dialogue response generator 132 via the speaker 232 .

[0666] When the result processor 130 receives the pre-utterance message itself corresponding to the pre-utterance context from the dialog flow manager 121 , the input pre-utterance message may become a dialog response, and the input pre-utterance message may be sent to the output manager 133 .

[0667] The output manager 133 may output the transmitted pre-utterance message via the speaker 232 .

[0668] When a user utterance is input after the dialogue system 100 outputs a pre-utterance, the same operation as that for processing the user utterance described above may be performed.

[0669] According to the above embodiment, the dialogue system 100 can provide the most suitable service for the user by considering various situations occurring inside the vehicle. Without inputting the user's speech, the dialogue system 100 can also determine the service required by the user based on the context information or driver information collected by itself, and actively provide the service.

[0670] For example, the evaluation criteria for the vehicle status can vary depending on the situation when the vehicle is started, so feedback can be provided proactively. The driving start time can be defined as the time when the vehicle is started, the time point when the electronic parking brake (EPB) is released, or the time point when the navigation destination is set. The vehicle condition evaluation system that calculates the driving availability score can assign weights to each device and change the variable weight applied to each device according to situational factors. When it is determined that there is a problem with the vehicle status, solutions regarding each device, such as repair shop guidance, can be provided.

[0671] By considering the destination when the vehicle starts, it can be determined whether the vehicle is short of fuel. When short of fuel, as feedback of the short of fuel, it can be performed to add the user's favorite gas station as an automatic stop point in the route to the destination, and notify the user of the change of the stop point. In addition, according to the user's response, the gas station added as the automatic stop point can be changed.

[0672] Although the current vehicle status does not indicate a lack of fuel, a refueling station or refueling time may be proactively provided by comprehensively considering the user's next schedule, major movement records, and the remaining fuel amount.

[0673] By acquiring information related to the driver's physical condition and sleep record, the vehicle can be conditionally allowed to start based on the acquired information. For example, when the risk of drowsy driving is identified by identifying the physical condition and sleep record from outside the vehicle, the user can be advised not to drive the vehicle. Alternatively, information related to recommended driving time can be provided based on the physical condition or sleep record.

[0674] When a trigger indicating a risk of drowsy driving occurs repeatedly, the risk of drowsy driving may be detected and a warning may be output according to the degree of risk or feedback such as automatically changing the route, i.e., changing the route to a rest area, may be provided. The trigger indicating the risk of drowsy driving may be obtained by manually measuring the driver's state and the vehicle state (e.g., a case where the heart rate decreases, a case where the front and rear gap is a reference distance or more, a case where the vehicle speed is a reference speed or less) or by active measurement via a conversation (e.g., a case where a question is spoken to the driver and the driver's response speed to the question is measured).

[0675] When the user inputs an utterance indicating an emotion, the dialogue system 100 may not extract a certain domain or action from the user's utterance. However, the dialogue system 100 may recognize the user's intention by using the surrounding environment information, the vehicle state information, and the user state information, and then continue the dialogue. As described above, this embodiment may be performed by resolving the ambiguity of the user's utterance by the fuzzy resolver 123.

[0676] In addition, when a passenger gets on the vehicle, the dialogue system 100 may determine that the passenger gets on the vehicle and output a pre-speech for identifying the passenger. Specifically, when a passenger gets on the vehicle, the dialogue system 100 may determine that the passenger gets on the vehicle through voice input or input other than voice, and ask a question (e.g., who are you? Tell me your name) for identifying the passenger. The dialogue system 100 may receive the passenger's speech related to the question and actively identify the passenger through dialogue.

[0677] In addition, when estimating the change in the number of passengers in the vehicle, the dialogue system 100 may output a pre-utterance related to the estimated result of the change in the number of passengers. Specifically, the dialogue system 100 may receive a conversation between passengers in the vehicle through voice input, and estimate the possibility of each passenger leaving the vehicle and the possibility of each passenger getting on the vehicle again after leaving, so as to output a pre-utterance related to the estimated result of the change in the number of passengers.

[0678] Hereinafter, an example of dialogue processing using the dialogue system 100 will be described in detail.

[0679] Figures 33 to 45 It is a diagram showing a specific example in which, when a user inputs an utterance related to route guidance, the dialog system 100 processes the input, manages the dialog, and outputs a result.

[0680] like Fig.33 As shown, when the user inputs an utterance "Let's go to the Seoul Station we went to yesterday", the voice recognizer 111a may output the user's voice in a text form of the utterance (Let's go to the Seoul Station we went to yesterday).

[0681] The natural language understanding part 111b can perform morpheme analysis by referring to the domain / action inference rule DB 141 and output [domain: navigation], [action: route guidance], [voice behavior; request] and [parameter: NLU: destination: Seoul Station] from the morpheme analysis results (yesterday / NNG, go / VV, Seoul Station / NNP, go / VV), and then input them into the dialogue input manager 111c.

[0682] refer to Fig.34 When the natural language understanding result of the natural language understanding part 111b is sent to the context understanding part 112c and there is additional information in the context understanding part 112c, the dialogue input manager 111c may request the context understanding part 112c to send the additional information.

[0683] The context understanding section 112c may search the context understanding table 145 and extract the fact that the context information related to [domain: navigation] and [action: route guidance] is the current location and the type of the context information is a GPS value.

[0684] The context understanding part 112c may acquire the GPS value of the current location by searching the context information DB 142. When the GPS value of the current location is not stored in the context information DB 142, the context understanding part 112c may request the GPS value of the current location from the context information collection manager 112b.

[0685] The context information collection manager 112b may send a signal to the context information collector 112a so that the context information collector 112a collects the GPS value of the current location. The context information collector 112a may collect the GPS value of the current location from the vehicle controller 240, and then store the GPS value of the current location in the context information DB 142, while sending a GPS value collection confirmation signal to the context information collection manager 112b. When the context information collection manager 112b sends the GPS value collection confirmation signal to the context understanding part 112c, the context understanding part 112c may obtain the GPS value of the current location from the context information DB 142, and then send the GPS value of the current location to the dialogue input manager 111c.

[0686] The dialogue input manager 111c can combine [Domain: Navigation], [Action: Route guidance], [Voice behavior; Request], [Parameter: NLU: Destination: Seoul Station] and [Context information: Current location: Seoul Station (GPS value)] into a natural language understanding result, and then send the combined information to the dialogue manager 120.

[0687] refer to Fig.35 , the dialog flow manager 121 may search the dialog and action state DB 147 and determine whether there is a currently ongoing dialog task or action task. At this time, the dialog flow manager 121 may refer to the dialog strategy DB 148. According to this embodiment, it is assumed that there is no currently ongoing dialog task or action task.

[0688] The dialog flow manager 121 may request the dialog action manager 122 to generate an action task and a dialog task corresponding to the output of the input processor 110. The generation of the action task and the dialog task may mean designating a storage space for storing and managing information related to an action state and a dialog state.

[0689] Therefore, the dialog action manager 122 may designate storage space in the dialog and action status DB 147 to store information related to the action status and the dialog status.

[0690] The dialog action manager 122 may send the action status and the dialog status to the action priority determiner 125 .

[0691] The action priority determiner 125 may search the relational action DB 146b for vehicle status check and gas station recommendation related to route guidance. The route guidance action and the relational action may become candidate actions.

[0692] The action priority determiner 125 may determine the priority of the candidate actions according to pre-stored rules. Priority may be determined before determining the execution conditions of the candidate actions, or alternatively, after determining the execution conditions of the candidate actions, priority may be determined only for the candidate actions that satisfy the execution conditions.

[0693] The candidate action list may be sent to the dialog action manager 122 again, and the dialog action manager 122 may update the action state by adding the searched relationship action.

[0694] refer to Fig.36 , the action priority determiner 125 may search the action execution condition DB 146c for the execution condition of each candidate action and the parameters required to determine the execution condition. The action priority determiner 125 may also determine the priority between the candidate actions.

[0695] For example, the condition for the vehicle status check may be a case where the destination distance is equal to or greater than 100 km, wherein a parameter for determining the condition may correspond to the destination distance.

[0696] The condition for the gas station recommendation may be a case where the destination distance is greater than the distance to empty (DTE), wherein the parameter for determining the condition may correspond to the destination distance and the distance to empty (DTE).

[0697] The dialog action manager 122 may update the action status by adding a condition for executing each candidate action and parameters required to determine the condition to the dialog and action status DB 147 .

[0698] The action priority determiner 125 can search the dialog and action status DB 147, the context information DB 142, the long-term memory 143 or the short-term memory 144 for parameter values ​​required to determine whether the candidate action meets the execution conditions, and obtain the parameter values ​​from the dialog and action status DB 147, the context information DB 142, the long-term memory 143 or the short-term memory 144.

[0699] When the parameter value is contained in the previous conversation content, in the context information related to the conversation content, or in the context information related to the generated event, the action priority determiner 125 may acquire the parameter value from the conversation and action state DB 147 .

[0700] When the action priority determiner 125 fails to acquire the parameter value from the dialog and action state DB 147 , the context information DB 142 , the long-term storage 143 , or the short-term storage 144 , the action priority determiner 125 may request the parameter value from the external information manager 126 .

[0701] For example, the destination distance may be acquired from the external content server 300 providing the navigation service via the external information manager 126, and the DTE may be acquired from the context information DB 142. Meanwhile, in order to search for the destination distance, correct destination information for the navigation service may be required. In this embodiment, the destination input from the user's speech may correspond to "Seoul Station", where "Seoul Station" may include various places having names starting with "Seoul Station" as well as "Seoul Station" having a specific meaning. Therefore, it may be difficult to search for the correct destination distance using only "Seoul Station".

[0702] As needed, parameter values ​​may be acquired from the mobile device 400 connected to the vehicle 200. For example, when user information (e.g., contacts and schedules) not stored in the long-term memory 143 is required as a parameter value, the external information manager 126 may request the required information from the mobile device 400 and then acquire the required parameter value.

[0703] When the parameter value may not be acquired via the storage device 140 , the external content server 300 , and the mobile device 400 , the required parameter value may be acquired by asking the user.

[0704] The action priority determiner 125 may determine the execution conditions of the candidate actions by using the parameter values. Since the destination distance is not searched, the determination of the execution conditions related to the vehicle state check action and the gas station recommendation may be postponed.

[0705] like Fig.37 As shown in , the dialog action manager 122 may update the action status by adding the acquired parameter values ​​and whether the action execution condition is satisfied, determined by using the corresponding parameter values, to the dialog and action status DB 147 .

[0706] The dialog action manager 122 may request the parameter manager 124 for a list of parameters for performing the candidate action.

[0707] The parameter manager 124 may acquire the current location and the destination as necessary parameters for executing the route guidance action from the action parameter DB 146 a , and extract the route type (initial value: express route) as an optional parameter.

[0708] The parameter manager 124 may acquire the check portion (initial value: entire portion) for performing the vehicle state check action as a selective parameter, and extract a favorite gas station (initial value: A-gas station) as a selective parameter for performing the gas station recommendation action.

[0709] The extracted parameter list may be sent to the dialog action manager 122 and used to update the action state.

[0710] The parameter manager 124 may search for a corresponding parameter value in a reference position of each parameter in the dialog and action state DB 147, the context information DB 142, the long-term storage 143, and the short-term storage 144 to obtain parameter values ​​of required parameters and optional parameters corresponding to the candidate action. When a parameter value needs to be provided via an external service, the parameter manager 124 may request the required parameter value from the external content server 300 via the external information manager 126.

[0711] The parameters for determining the execution conditions of the candidate actions and the parameters for executing the candidate actions may overlap. Among the parameter values ​​obtained by the action priority determiner 125 and then stored in the dialogue and action state DB 147, when there are parameters corresponding to the parameters (necessary parameters and optional parameters) for executing the candidate actions, the corresponding parameters can be used.

[0712] refer to Fig.38, the dialog action manager 122 may update the action state by adding the parameter values ​​obtained by the parameter manager 124 .

[0713] As described above, when the destination (Seoul Station) extracted from the user's speech is used as a parameter of the route guidance action, there may be ambiguity. Therefore, the parameters of the route guidance action (destination), the parameters of the vehicle status check action (destination distance), and the parameters of the gas station recommendation (destination distance) may not have been acquired.

[0714] When [parameter: NLU: destination: Seoul Station] is converted into a destination parameter suitable for a route guidance action, the ambiguity resolver 123 can check whether there is ambiguity. As described above, "Seoul Station" may include different kinds of places having names starting with "Seoul Station", as well as "Seoul Station" having a user-specific meaning.

[0715] The fuzzy solver 123 can confirm that there is a modifier of "Seoul Station" in the user's utterance by referring to the morpheme analysis result. The fuzzy solver 123 can search the schedule, mobile location, and contact in the long-term memory 143 or the short-term memory 144 to identify the location of "Seoul Station where we went yesterday".

[0716] For example, the fuzzy solver 123 may confirm that “Seoul Station where we went yesterday” is “Seoul Station Exit 4” from the movement location performed by the user yesterday. After confirming that there is a POI (eg, “Seoul Station Exit 4”), the fuzzy solver 123 may acquire a corresponding value.

[0717] The destination information acquired by the fuzzy resolver 123 may be sent to the dialog action manager 122, and the dialog action manager 122 may update the action state by adding "Seoul Station Exit 4" to the destination parameter of the candidate action.

[0718] The parameter manager 124 may acquire the destination information (Seoul Station Exit 4) from the dialogue and action status DB 147 , and request the destination distance value to the external content server 300 that provides the navigation service via the external information manager 126 .

[0719] refer to Fig.39 , when the external information manager 126 obtains the destination distance value (80 km) from the external content server 300 and then sends the destination distance value to the parameter manager 124, the parameter manager 124 can send the destination distance value to the dialog action manager 122 to allow the action state to be updated.

[0720] The action priority determiner 125 can determine whether the candidate action is executable by referring to the action state and adjust the priority of the candidate action. Since the parameter values ​​of the current position and the destination as necessary parameters are obtained, it can be determined that the route guidance action is executable. Since the destination distance (70km) is less than 100km, it can be determined that the vehicle status check action is not executable. Since the destination distance (80km) is greater than DTE, it can be determined that the gas station recommendation action is executable.

[0721] Since the vehicle state check action is not executable, the vehicle state check action may be excluded from the priority determination. Therefore, the route guidance action may be ranked first, and the gas station recommendation action may be ranked second.

[0722] The dialog action manager 122 may update the action status according to whether the candidate action is executable and the priority of the modification.

[0723] The dialog flow manager 121 may check the dialog state and the action state stored in the dialog and action state DB 147, and may develop a dialog strategy to continue the dialog by referring to the dialog strategy DB 148. For example, the dialog flow manager 121 may select the highest priority action among the executable actions, and the dialog flow manager 121 may request the response generation manager 131 to generate a response for conducting the dialog according to the dialog strategy DB 148.

[0724] The dialog status and action status stored in the dialog and action status DB 147 may be updated to [status: confirmed route guidance start].

[0725] refer to Fig.40 The response generation manager 131 may request generation of a response from the dialog response generator 132 in response to a request from the dialog flow manager 121 .

[0726] The dialogue response generator 132 may generate a TTS response and a text response by searching the response template 149. For example, the dialogue response generator 132 may generate a dialogue response configured to output "It is estimated to take 30 minutes from Uiwang Station to Seoul Station Exit 4. Do you want to start guidance?" in TTS and text forms.

[0727] As another example, the dialog flow manager 121 may send a pre-discourse event to the dialog response generator 132. Therefore, the dialog response generator 132 may generate a dialog configured to output "Who are you?, tell me your name" in TTS and text form. Alternatively, the dialog response generator 132 may generate a dialog configured to output "A will leave at the stop point" in TTS and text form. In other words, the dialog response generator 132 may generate questions for identifying passengers when they board the vehicle, and the dialog response generator 132 may determine the possibility of each passenger leaving the vehicle and the possibility of each passenger boarding the vehicle again after leaving and generate a dialog related thereto.

[0728] The response generation manager 131 may send the TTS response and the text response generated by the dialog response generator 132 to the output manager 133 and the memory manager 135, and the output manager 133 may send the TTS response to the speaker 232 and send the text response to the display 231. At this time, the output manager 133 may send the TTS response to the speaker 232 after passing through a TTS module configured to combine text with speech.

[0729] The memory manager 135 may store the user requested route guidance in the short-term memory 144 or the long-term memory 143 .

[0730] A dialog response configured to ask "It is estimated to take 30 minutes from Uiwang Station to Seoul Station Exit 4. Do you want to start guidance?" may be output via the display 231 and the speaker 232. Fig.41 As shown, when the user says "yes", the user's words can be input into the speech recognizer 111a and then output as [text: yes], and the natural language understanding part 111b can output [domain: -], [action: -], [voice action: -] and [morpheme analysis result: yes / IC].

[0731] The natural language understanding result may be transmitted to the dialog input manager 111 c , and the dialog input manager 111 c may transmit the natural language understanding result to the dialog manager 120 .

[0732] refer to Fig.42 , the dialog flow manager 121 may search the dialog and action state DB 147 and analyze the previous dialog state. The dialog flow manager 121 may request the dialog action manager 122 to update the dialog / action related to the currently executed [route guidance].

[0733] The dialog action manager 122 may update the dialog state and the action state to [state: route guidance started].

[0734] The dialog flow manager 121 may request the result processor 130 to generate a response for starting route guidance.

[0735] refer to Fig.43 , the dialog action manager 122 can update the dialog state to [State: Proceed to next dialog] and update the action state to [State: Execute].

[0736] The dialog flow manager 121 may request the response generation manager 131 to generate a response to the route guidance.

[0737] The dialogue response generator 132 may generate a dialogue response configured to output “start route guidance” as TTS and a text form, and then transmit the dialogue response to the response generation manager 131 .

[0738] The command generator 136 may generate a command for executing route guidance [target: navigation, command: route guidance, destination: Seoul Station Exit 4, departure: Uiwang Station] and then transmit the command to the response generation manager 131 .

[0739] The response generation manager 131 may transmit the generated dialogue response and command to the output manager 133. The output manager 133 may output the dialogue response via the display 231 and the speaker 232. The output manager 133 may transmit the route guidance command to the AVN 230 of the vehicle 200 via the vehicle controller 240, or to the external content server 300 providing a navigation service.

[0740] refer to Fig.44 , the dialog flow manager 121 may select gas station recommendation as the next executable action and request the response generation manager 131 to generate a response configured to ask the user whether to recommend a gas station.

[0741] The dialog status and action status may be updated to [Status: Check related service recommendations].

[0742] The response generation manager 131 may request the dialog response generator 132 to generate a TTS response and a text response, and the dialog response generator 132 may generate a dialog response configured to output "Not enough fuel to reach the destination. Do you want to add A-gas station to the stop point?" in a TTS and text form. The dialog response generator 132 may send the TTS and text to the response generation manager 131.

[0743] The response generation manager 131 may transmit the TTS response and the text response generated by the conversation response generator 132 to the output manager 133 and the memory manager 135 , and the output manager 133 may transmit the TTS response to the speaker 232 and the text response to the display 231 .

[0744] A dialog response configured to ask “There is not enough fuel to reach the destination. Do you want to add a gas station to the stop?” may be output via the display 231 and the speaker 232. Fig.45 As shown, when the user says "No", the user's words can be input into the speech recognizer 111a and then output as [text: No], and the natural language understanding part 111b can output [domain: -], [action: -], [voice behavior: -] and [morpheme analysis result: No / IC].

[0745] The dialog flow manager 121 may request the dialog action manager 122 to update the dialog state and the action state.

[0746] The dialog action manager 122 may update the dialog state to [state: proceed to next dialog] and the action state to [state: cancel].

[0747] The dialog flow manager 121 may request the response generation manager 131 to generate a response indicating that the gas station recommendation service is canceled, and the dialog flow manager 121 may check whether there is a dialog to be continued. When there is no dialog to be continued, the dialog flow manager 121 may update the dialog state to [state: idle] and wait for the user's input.

[0748] The above data processing flow is only an example applied to the dialogue system 100. Therefore, the order in which each component of the dialogue system 100 processes data is not limited to the above example, so multiple components can process data simultaneously, or multiple components can process data in an order different from the above example.

[0749] Hereinafter, according to an embodiment, a dialogue processing method will be described. According to an embodiment, the dialogue processing method can be applied to the above-mentioned dialogue system 100 or a vehicle 200 provided with the dialogue system 100. Therefore, Figures 1 to 45 The description of will apply to the dialog processing method in the same way.

[0750] Fig.46 FIG. 1 is a flowchart showing a method for processing user input in a dialog processing method according to an embodiment of the present invention. The method for processing user input may be executed in the input processor 110 of the dialog system 100 .

[0751] refer to Fig.46When the user's speech is input (Yes in 500 ), the speech recognizer 111 a may recognize the input user's speech ( 510 ). The user's speech may be input to the speech input device 210 provided in the vehicle 200 or the speech input device 410 provided in the mobile device 400 .

[0752] The speech recognizer 111 a may recognize an input user's utterance and output the utterance in a text form.

[0753] The natural language understanding section 111b may apply a natural language understanding technique to the utterance in the form of text (520) and output a result of the natural language understanding.

[0754] Specifically, the natural language understanding process (520) may include performing morpheme analysis on a text-form discourse (521), extracting a domain from the discourse based on the morpheme analysis result (522), identifying entity names (523), analyzing speech acts (524) and extracting actions (525).

[0755] The extraction of the domain, the recognition of the entity name, and the extraction of the action may be performed by referring to the domain / action inference rule DB 141 .

[0756] The output of the natural language understanding part 111b, that is, the result of natural language understanding, may include the result of domain, action, voice behavior and morpheme analysis corresponding to the user's utterance.

[0757] The context information related to the extracted action may be searched (530). The context information related to the extracted action may be stored in the context understanding table 145. The context understanding part 112c may search the context information related to the extracted action in the context understanding table 145, and the context information processor 112 may obtain the information value of the searched context information from the context information DB 142, the long-term memory 143, or the short-term memory 144.

[0758] When additional context information is needed (Yes in 540), that is, when context information cannot be acquired from the context information DB 142, the long-term storage 143, or the short-term storage 144, the context understanding part 112c may request to collect the corresponding context information (550). Input other than speech, such as vehicle state information, surrounding environment information, and driver information may be input separately from the input of the user's speech via the context information collector 112a.

[0759] The information may be entered periodically or only when a specific event occurs. Alternatively, the information may be entered periodically and then additionally when a specific event occurs. In any case, when the information is requested to be collected, the corresponding information may be actively collected.

[0760] Therefore, when context information related to an action has been collected, the corresponding information may be acquired from the context information DB 142, the long-term storage 143, or the short-term storage 144, otherwise, the corresponding information may be collected via the context information collector 112a.

[0761] When the context information collector 112 a receiving the request for collecting context information collects corresponding context information and stores the information in the context information DB 142 , the context understanding part 112 c may acquire the corresponding context information from the context information DB 142 .

[0762] When the context information collection manager 112b determines that a certain event occurs because the data collected by the context information collector 112a satisfies a predetermined condition, the context information collection manager 112b may send an action trigger signal to the context understanding part 112c.

[0763] The context understanding part 112c may search the context understanding table 145 for context information related to the corresponding event, and when the searched context information is not stored in the context understanding table 145, the context understanding part 112c may send a context information request signal to the context information collection manager 112b again.

[0764] When the collection of the required context information is completed, the result of natural language understanding and the context information can be sent to the dialogue manager 120 (560). When an event occurs, information related to the event (which event occurred) and context information related to the event that occurred can also be sent.

[0765] Fig.47 FIG. 1 is a flowchart showing a method for managing a dialog using an output of an input processor in a dialog processing method according to an embodiment. The dialog processing method may be executed by the dialog manager 120 of the dialog system 100 .

[0766] refer to Fig.47 , the dialog flow manager 121 may search for relevant dialog history in the dialog and action state DB 147 (600).

[0767] In this embodiment, the case of extracting domains and actions from the user's speech is described as an example, but there may be a case where it is impossible to extract domains and actions from the user's speech because there is ambiguity in the speech content or context. In this case, the dialogue action manager 122 can generate a random dialogue state, and the fuzzy resolver 123 can recognize the user's intention based on the content of the user's speech, environmental conditions, vehicle status, and user information, and determine an action suitable for the user's intention.

[0768] When there is a relevant dialogue history (Yes in 600), the relevant dialogue history may be referred to (690). When there is no relevant dialogue history (No in 600), a new dialogue task and action task may be generated (610).

[0769] The related action list related to the action extracted from the user's utterance (hereinafter referred to as input action) may be searched in the relational action DB 146b, and a candidate action list may be generated (620). The input action and actions related to the input action may correspond to the candidate action list.

[0770] The action execution condition DB 146c may be searched for an execution condition according to each candidate action (620). The execution condition may indicate a necessary condition for executing the action. Therefore, when the corresponding condition is met, it may be determined that the action is executable, but when the corresponding condition is not met, it may be determined that the action is not executable. In the action execution condition DB 146c, information related to the type of parameter used to determine the action execution condition may also be stored.

[0771] A parameter value for determining the action execution condition may be acquired (640). The parameter for determining the action execution condition may be referred to as a condition determination parameter. The parameter value of the condition determination parameter may be acquired by searching the context information DB 142, the long-term storage 143, the short-term storage 144, or the dialog and action state DB 147. When the parameter value of the condition determination parameter needs to be provided via an external service, the required parameter value may be provided from the external content server 300 via the external information manager 126.

[0772] When it is impossible to obtain the desired parameter value due to ambiguity in the context and the utterance, the desired parameter value may be obtained by resolving the ambiguity using the ambiguity resolver 123 .

[0773] Although the acquired parameters are invalid parameters having difficulty in action execution condition determination, the fuzzy solver 123 can acquire valid parameters from the invalid parameters.

[0774] Based on the acquired condition determination parameters, it can be determined whether each candidate action is executable (650), and the priority of the candidate action can be determined (660). Rules for determining the priority of the candidate actions can be pre-stored. The action priority determiner 125 can determine the priority of the candidate actions by considering only the executable candidate actions after determining whether each candidate action is executable. Alternatively, after determining the priority of the candidate actions regardless of whether each candidate action is executable, the priority of the candidate actions can be modified based on whether each candidate action is executable.

[0775] The action parameter DB 146a may be searched for a parameter list for executing the candidate action (670). The parameters for executing the candidate action may correspond to the action parameters. The action parameters may include required parameters and optional parameters.

[0776] Parameter values ​​for executing the candidate action may be acquired (680). Parameter values ​​of the action parameters may be acquired by searching the context information DB 142, the long-term storage 143, the short-term storage 144, or the dialog and action state DB 147. When the parameter values ​​of the action parameters need to be provided via an external service, the required parameter values ​​may be provided from the external content server 300 via the external information manager 126.

[0777] When a desired parameter value cannot be acquired due to ambiguity in context and discourse, the desired parameter value may be acquired by resolving the ambiguity using the ambiguity resolver 123 .

[0778] Although the acquired parameters are invalid parameters having difficulty in action execution condition determination, the fuzzy solver 123 can acquire valid parameters from the invalid parameters.

[0779] The dialog state and action state managed by the dialog action manager 122 may be executed through the above steps, and the dialog state and action state may be updated whenever the state changes.

[0780] When all available parameter values ​​are obtained, the dialog flow manager 121 may send information about candidate actions and dialog states to the result processor 130. According to the dialog strategy, the dialog flow manager 121 may send information about an action corresponding to the first priority or information about a plurality of candidate actions.

[0781] When the required parameter value can only be acquired by the user because the required parameter value does not exist in the external content server 300, the long-term storage 143, the short-term storage 144, and the context information DB 142, a dialog response for inquiring about the parameter value may be output to the user.

[0782] Fig.48 FIG. 1 is a flowchart showing a result processing method for generating a response corresponding to a result of dialog management in a dialog processing method according to an embodiment of the present invention. The result processing method may be executed by the result processor 130 of the dialog system 100 .

[0783] refer to Fig.48 When a dialog response needs to be generated (Yes in 700), the dialog response generator 132 may search for the response template 149 (710). The dialog response generator 132 may obtain a dialog response template corresponding to the current dialog state and action state, and fill the response template with required parameter values ​​to generate a dialog response (720).

[0784] When a parameter value required for generating a dialog response is not transmitted from the dialog manager 120, or when an introduction using external content is transmitted, the required parameter value may be provided from the external content server 300 or searched in the long-term storage 143, the short-term storage 144, or the context information DB 142. When the required parameter value can only be acquired by the user because the required parameter value does not exist in the external content server 300, the long-term storage 143, the short-term storage 144, and the context information DB 142, a dialog response for inquiring about the parameter value may be generated to the user.

[0785] When command generation is required ( 760 ), command generator 136 may generate commands for vehicle control or external content ( 770 ).

[0786] The generated dialog responses or commands may be input to the output manager 133, and the output manager 133 may determine an output order of the dialog responses and commands or an output order among a plurality of commands (730).

[0787] The memory may be updated based on the generated dialogue response or command (740). The memory manager 135 may update the short-term memory 144 by storing the dialogue content between the user and the system based on the generated dialogue response or command, and update the long-term memory 143 by storing information related to the user acquired through the dialogue with the user. The memory manager 135 may update the user's preference and vehicle control history stored in the long-term memory 143 based on the generated and output vehicle control and external content requests.

[0788] The output manager 133 may output the response by sending the dialogue response and the command to an appropriate output location (750). The TTS response may be output via the speaker 232, and the text response may be output on the display 231. The command may be sent to the vehicle controller 240 according to the control object, or the command may be sent to the external content server 300. In addition, the command may be sent to the communication device 280 configured to communicate with the external content server 300.

[0789] Figures 49 to 51 The flowchart is a flowchart showing a case where a dialogue system outputs a pre-utterance before an utterance is input in a dialogue processing method according to an embodiment.

[0790] refer to Fig.49, the context information collector 112a and the context information collection manager 112b collect context information (810). Specifically, the vehicle controller 240 may input vehicle state information and driving environment information acquired by sensors provided in the vehicle, such as remaining fuel amount, rainfall amount, rainfall speed, surrounding obstacle information, speed, engine temperature, tire pressure, and current position, into the context information processor 112. User information input via the information input device 220 other than voice and information acquired from the external content server 300 or an external device may be input into the context information processor 112. The collected context information may be stored in the context information DB 142, the long-term storage 143, or the short-term storage 144.

[0791] The pre-speech determiner 151 determines the pre-speech condition based on the context information (811). The pre-speech condition may be stored in the pre-speech condition table 145a. FIG. 25A to FIG. 25D As shown, the pre-speech condition related to the context information may be stored in the pre-speech condition table 145 a for each context information.

[0792] When the context information transmitted from the context information DB 142 , the long-term storage 143 , or the short-term storage 144 satisfies the pre-utterance condition ( YES in 812 ), the pre-utterance determiner 151 determines it as the pre-utterance context and generates a pre-utterance trigger signal ( 813 ).

[0793] The pre-utterance determiner 151 extracts actions corresponding to the pre-utterance context (814). Fig.25C As shown, the action corresponding to the pre-utterance context can be pre-stored in the pre-utterance condition table 145a. The pre-utterance determiner 151 can obtain the action corresponding to the pre-utterance context from the pre-utterance condition table 145a. In addition, the pre-utterance determiner 151 can generate the action corresponding to the pre-utterance context according to the established rules.

[0794] When the pre-discourse determiner 151 sends a pre-discourse trigger signal having an action corresponding to the pre-discourse context to the dialog input manager 111c, the dialog input manager 111c sends the action corresponding to the pre-discourse context to the dialog manager 120 (815). In this case, the pre-discourse trigger signal may be sent using a signal indicating the pre-discourse context.

[0795] like Fig.47 As shown, after the action corresponding to the pre-discourse context is sent to the dialog manager 120, a series of processes such as the generation of dialog tasks and action tasks and the acquisition of action parameters may be performed. When other dialog tasks or action tasks are being executed, the dialog flow manager 121 may first generate and process tasks related to the pre-discourse context, or may select a priority according to established rules.

[0796] When the dialogue manager 120 transmits information about the first executed action to the result processor 130, the dialogue response generator 132 may acquire a dialogue response template by searching the response template 149, and generate a dialogue response by filling the extracted dialogue response template with parameter values. The generated dialogue response may be transmitted to the output manager 133 via the response generation manager 131. The output manager 133 may output the generated dialogue response via a speaker provided in the vehicle 200 or the mobile device 400.

[0797] In addition, the pre-utterance message itself corresponding to the pre-utterance context may be obtained or generated. Fig.50 , the context information collector 112a and the context information collection manager 112b collect context information (820), and the pre-utterance determiner 151 determines a pre-utterance condition based on the context information (821).

[0798] When the context information transmitted from the context information DB 142 , the long-term storage 143 , or the short-term storage 144 satisfies the pre-utterance condition ( YES in 822 ), the pre-utterance determiner 151 determines it as the pre-utterance context and generates a pre-utterance trigger signal ( 823 ).

[0799] The pre-utterance determiner 151 extracts the pre-utterance message corresponding to the pre-utterance context (824). Fig.25A , Fig.25B , Fig.25C and Fig.25D As shown, the pre-speech message corresponding to the pre-speech context can be pre-stored in the pre-speech condition table 145a. The pre-speech message stored in advance can be content indicating the current context or content that first suggests a specific function or service required to execute the pre-speech context. In addition, the pre-speech determiner 151 can generate a pre-speech message according to an established rule.

[0800] When the pre-speech determiner 151 sends the pre-speech trigger signal together with the pre-speech message to the dialog input manager 111c, the dialog input manager 111c may send the pre-speech message to the dialog manager 120 (825). In this case, the pre-speech trigger signal may be sent together with a signal indicating a pre-speech context.

[0801] The dialog manager 120 may generate a dialog task for outputting the transmitted pre-utterance message, and transmit the dialog task to the result processor 130. The result processor 130 may output the input pre-utterance message via the speaker 232.

[0802] Furthermore, virtual user utterances corresponding to the pre-utterance context can be extracted. Fig.51, the context information collector 112a and the context information collection manager 112b collect context information (830), and the pre-utterance determiner 151 determines a pre-utterance condition based on the context information (831).

[0803] When the context information transmitted from the context information DB 142 , the long-term storage 143 , or the short-term storage 144 satisfies the pre-utterance condition ( YES in 832 ), the pre-utterance determiner 151 determines it as the pre-utterance context and generates a pre-utterance trigger signal ( 833 ).

[0804] The pre-discourse determiner 151 extracts a virtual user utterance corresponding to the pre-discourse context (834). Although not shown in the drawings, the virtual user utterance corresponding to the pre-discourse context may be pre-stored in the pre-discourse condition table 145a. The pre-discourse determiner 151 obtains the virtual user utterance corresponding to the pre-discourse context from the pre-discourse condition table 145a. In addition, the pre-discourse determiner 151 may generate a virtual user utterance corresponding to the pre-discourse context according to the established rules.

[0805] When the pre-utterance determiner 151 sends the virtual user utterance in text form to the natural language understanding part 111b (835), the natural language understanding part 111b can obtain the domain and action from the virtual user utterance in the same manner as when the user actually speaks.

[0806] The dialog input manager 111c sends the pre-utterance trigger signal together with the natural language understanding result to the dialog manager 120 (836). The result of the natural language understanding may include the domain and action extracted from the virtual user utterance, and the extracted domain and action may become the domain and action corresponding to the pre-utterance context.

[0807] For example, according to the mobile gateway mode in which the mobile device 400 acts as a gateway between the vehicle and the dialogue system 100, the dialogue system client 470 of the mobile device 400 may perform some operations of the pre-utterance determiner 151. In this case, the dialogue system client 470 may generate a virtual user utterance corresponding to the pre-utterance context and send the virtual user utterance to the natural language understanding part 111b.

[0808] like Fig.47 As shown, after the pre-discourse trigger signal is sent to the dialog manager 120 together with the natural language understanding result, a series of processes such as the generation of dialog tasks and action tasks and the acquisition of action parameters may be performed. When other dialog tasks or action tasks are being executed, the dialog flow manager 121 may first generate and process tasks related to the pre-discourse context, or may select a priority according to established rules.

[0809] When the dialogue manager 120 transmits information about the first executed action to the result processor 130, the dialogue response generator 132 may acquire a dialogue response template by searching the response template 149, and generate a dialogue response by filling the extracted dialogue response template with parameter values. The generated dialogue response may be transmitted to the output manager 133 via the response generation manager 131. The output manager 133 may output the generated dialogue response via a speaker provided in the vehicle 200 or the mobile device 400.

[0810] Fig.52 The flowchart is a flowchart showing processing of a repetitive task when a dialog system outputs a pre-utterance before a user inputs an utterance in a dialog processing method according to an embodiment.

[0811] refer to Fig.52 , the context information collector 112a and the context information collection manager 112b collect context information (840), and the pre-utterance determiner 151 determines a pre-utterance condition based on the context information (841).

[0812] The pre-discourse determiner 151 determines whether the context information sent from the context information DB 142, the long-term memory 143 or the short-term memory satisfies the pre-discourse condition, and when the context information satisfies the pre-discourse condition (yes in 842), the repetitive task processor 152 determines whether the task related to the pre-discourse context currently occurring is repeated (843).

[0813] Specifically, the repetitive task processor 152 may determine whether a task such as a dialogue and action related to a currently occurring pre-discourse context has been or is currently being performed based on information related to tasks previously or currently performed in the dialogue system 100 stored in the task processing DB 145b.

[0814] For example, when a conversation related to the currently occurring pre-discourse context has been performed, and when the reference time period has not passed since the conversation time point, the repetitive task processor 152 may determine that the task related to the current pre-discourse context is a repetitive task. In addition, when a conversation and an action related to the current pre-discourse context are currently being performed, the repetitive task processor 152 may determine that the task related to the current pre-discourse context is a repetitive task.

[0815] That is, the repetitive task processor 152 can determine whether the pre-utterance has been output and the user's intention regarding the pre-utterance context based on the conversation history stored in the task processing DB 145b and whether the task is executed. The repetitive task processor 152 can determine whether it is a repetitive task based on the stored conversation time...

Claims

1. A dialogue system for a vehicle, comprising: an input processor configured to receive a conversation between occupants of the vehicle including a driver and at least one passenger, detect vehicle operation information, identify at least one passenger based on the conversation between the occupants or the vehicle operation information, generate passenger number information estimating a change in the number of passengers in the vehicle when the vehicle arrives at a stop point based on the conversation between the occupants, and acquire a pre-speech message based on the passenger number information; as well as A result processor configured to output a pre-speech according to the pre-speech message; Wherein: the pre-speech message indicates at least one of the following: the possibility of each of the at least one passenger leaving the vehicle at the stop point, the possibility of each of the at least one passenger boarding the vehicle again after leaving the vehicle at the stop point, and the possibility of potential passengers boarding the vehicle at the stop point.

2. The dialogue system according to claim 1, wherein: The input processor comprises: a voice input processor configured to determine whether the at least one passenger is boarding the vehicle based on one or more voice features of the at least one passenger's voice included in the conversation between the occupants; and A context information processor is configured to determine whether the at least one passenger is boarding the vehicle based on the vehicle operation information.

3. The dialogue system according to claim 2, wherein: When it is determined that the at least one passenger is boarding the vehicle, the input processor is configured to obtain a pre-utterance message corresponding to the at least one passenger boarding the vehicle, receive an utterance of the at least one passenger related to the pre-utterance message, and identify the at least one passenger by applying a natural language understanding algorithm to the utterance of the at least one passenger, and When it is determined that the at least one passenger has not boarded the vehicle, the input processor is configured to obtain a pre-speech message corresponding to the at least one passenger not boarding the vehicle, receive the driver's speech related to the pre-speech message, and verify the presence of the at least one passenger by applying a natural language understanding algorithm to the driver's speech.

4. The dialogue system according to claim 1, wherein: The input processor is configured to determine a likelihood of each of the at least one passenger leaving the vehicle at the stop point and a likelihood of each of the at least one passenger getting on the vehicle again after leaving the vehicle at the stop point by applying a natural language understanding algorithm to the conversation between the occupants, and to generate the passenger quantity information based on the determined likelihood of the at least one passenger leaving the vehicle at the stop point and a likelihood of the at least one passenger getting on the vehicle again after leaving the vehicle at the stop point.

5. The dialogue system according to claim 1, wherein: The input processor is configured to receive a call session in the vehicle, determine the likelihood of a potential passenger boarding the vehicle at the stop by applying a natural language understanding algorithm to the received call session, and generate the passenger quantity information based on the likelihood of the potential passenger boarding the vehicle at the stop.

6. The dialogue system according to claim 4, wherein: After the vehicle leaves the stop point, the input processor is configured to determine a change in the number of passengers in the vehicle based on a conversation between the occupants and the vehicle operation information, compare an estimated change in the number of passengers in the vehicle based on the passenger number information with the determined change in the number of passengers, and acquire the pre-speech message based on the comparison result.

7. The dialogue system according to claim 6, wherein: After the vehicle leaves the stop, the input processor is configured to obtain the pre-speech message to determine a change in the number of passengers in the vehicle, receive an utterance of the at least one passenger related to the pre-speech message, and determine the change in the number of passengers in the vehicle by applying a natural language understanding algorithm to the utterance of the at least one passenger.

8. The dialogue system according to claim 1, further comprising: A storage device is configured to store travel-related information of the vehicle and passenger information of each of the at least one passenger when the vehicle stops traveling.

9. The dialogue system according to claim 8, wherein: The passenger information includes at least one of passenger identification information, voice feature information, seat position information, vehicle boarding time information, vehicle boarding position information, departure time information, or information related to a location of departure from the vehicle.

10. The dialogue system according to claim 8, wherein: The input processor is configured to receive a conversation between the occupants and the vehicle operation information, determine whether the at least one passenger has boarded the vehicle based on the conversation between the occupants and the vehicle operation information, determine whether a feature of each of the at least one passenger corresponds to the passenger information, and obtain the pre-speech message by verifying whether a first passenger having features corresponding to the passenger information participated in a previous trip.

11. The dialog system according to claim 10, wherein: The input processor is configured to receive an utterance of at least one passenger related to the pre-utterance message, verify whether the first passenger participated in a previous trip by applying a natural language understanding algorithm to the utterance of the passenger, and generate the passenger quantity information based on the conversation between the occupants and the passenger information when the first passenger participated in the previous trip.

12. A conversation processing method for a vehicle, comprising: receiving a conversation between a vehicle occupant including a driver and at least one passenger; Detect vehicle operation information; identifying the at least one passenger based on the conversation between the occupants or the vehicle operation information; generating passenger number information estimating a change in the number of passengers in the vehicle when the vehicle arrives at a stop point based on the conversation between the occupants; Acquire a pre-speech message according to the passenger quantity information; and Outputting a pre-speech according to the pre-speech message; Wherein: the pre-speech message indicates at least one of the following: the possibility of each of the at least one passenger leaving the vehicle at the stop point, the possibility of each of the at least one passenger boarding the vehicle again after leaving the vehicle at the stop point, and the possibility of potential passengers boarding the vehicle at the stop point.

13. The dialog processing method according to claim 12, further comprising: determining whether the at least one passenger is boarding the vehicle based on one or more speech characteristics of the at least one passenger's speech included in the conversation between the occupants; and A determination is made based on the vehicle operation information as to whether the at least one passenger is boarding the vehicle.

14. The dialog processing method according to claim 13, further comprising: When it is determined that the at least one passenger is boarding the vehicle, obtaining a pre-speech message corresponding to the at least one passenger boarding the vehicle; receiving an utterance of the at least one passenger associated with the pre-utterance message; and identifying the at least one passenger by applying a natural language understanding algorithm to an utterance of the at least one passenger; as well as When it is determined that the at least one passenger has not boarded the vehicle, obtaining a pre-speech message corresponding to the at least one passenger not boarding the vehicle; receiving an utterance of the driver associated with the pre-utterance message; and The presence of the at least one passenger is verified by applying a natural language understanding algorithm to the driver's utterance.

15. The dialog processing method according to claim 13, further comprising: determining, by applying a natural language understanding algorithm to the conversation between the occupants, a likelihood that each of the at least one passenger will leave the vehicle at the stop point and a likelihood that each of the at least one passenger will re-enter the vehicle after leaving the vehicle at the stop point; and The passenger quantity information is generated based on the determined likelihood of the at least one passenger leaving the vehicle at the stop point and the determined likelihood of the at least one passenger boarding the vehicle again after leaving the vehicle at the stop point.

16. The method for processing a dialogue according to claim 13, further comprising: receiving a call session in the vehicle; determining a likelihood of the potential passenger boarding the vehicle at the stop by applying a natural language understanding algorithm to the received call session; and The passenger quantity information is generated based on the likelihood of the potential passengers to board the vehicle at the stop point.

17. The method for processing a dialogue according to claim 15, further comprising: determining a change in the number of passengers in the vehicle based on a conversation between the occupants and the vehicle operation information after the vehicle leaves the stop point; comparing an estimated change in the number of passengers in the vehicle based on the passenger number information with the determined change in the number of passengers; and The pre-utterance message is acquired based on the comparison result.

18. The dialog processing method according to claim 17, further comprising: After the vehicle leaves the stop point, acquiring the pre-speech message to determine a change in the number of passengers in the vehicle; receiving an utterance of the at least one passenger associated with the pre-utterance message; and A change in the number of passengers in the vehicle is determined by applying a natural language understanding algorithm to an utterance of the at least one passenger.

19. The dialog processing method according to claim 12, further comprising: When the vehicle stops traveling, traveling related information of the vehicle and passenger information of each of the at least one passenger are stored.

20. The method for processing a conversation according to claim 19, wherein: The passenger information includes at least one of passenger identification information, voice feature information, seat position information, vehicle boarding time information, vehicle boarding position information, departure time information, or information related to a location of departure from the vehicle.

21. The dialog processing method according to claim 20, further comprising: receiving the conversation between the passengers and the vehicle operation information; determining whether the at least one passenger is boarding the vehicle based on the conversation between the occupants and the vehicle operation information; determining whether a characteristic of each of the at least one passenger corresponds to the passenger information; The pre-speech message is obtained by verifying whether a first passenger having a characteristic corresponding to the passenger information participated in a previous trip.

22. The dialog processing method according to claim 21, further comprising: receiving an utterance of the at least one passenger related to the pre-utterance message; verifying whether the first passenger participated in the previous ride by applying a natural language understanding algorithm to the passenger's utterance; and The passenger number information is generated based on a conversation between the occupants and the passenger information when the first passenger participated in the previous travel.

Images