Method and apparatus for performing response actions to voice commands in consideration of passenger profiles

Abstract

Methods and apparatus to perform a response action to a voice command in consideration of a passenger profile are disclosed. A method for performing a response action to a voice command based on a passenger profile includes: identifying a passenger in a vehicle; and performing a response action corresponding to the voice command based on a passenger profile of the passenger, wherein the passenger profile includes at least one of a vehicle control range, a POI search option, or preset information.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is based upon and claims the benefit of priority from Korean Patent Application No. 10-2023-0136629 filed in the Korean Intellectual Property Office on October 13, 2023, the disclosure of which is incorporated herein in its entirety by reference. Technical Field

[0002] The present disclosure relates to devices configured to perform actions in response to voice commands based on a passenger profile. Background Art

[0003] With the advancement of artificial intelligence technology in recent years, the scope of application of artificial intelligence technology is also expanding. Specifically, conversation systems that interact with users through natural language (such as chat systems or virtual assistants) are widely used in various fields, and related technologies are also steadily advancing. For a dialogue system for conducting a dialogue with a user, it is necessary to understand the user's utterance (i.e., input message) from the perspective of the dialogue system. In order to achieve natural language understanding (NLU), the dialogue system needs to derive the current context from the dialogue between the dialogue system and the user and the user's intended intent from the context; and analyze the input message based on the derived current context and / or intent.

[0004] The application scope of voice recognition services is expanding from homes to different fields, including the automotive field. For example, voice recognition assistant services and telematics services operate in conjunction with each other, and voice commands generated by user utterances are delivered to the car for controlling the car. Through this operation, the user can lock / unlock the doors or control the temperature inside the vehicle by turning on the air conditioner in advance.

[0005] However, conventional voice recognition functions generally ignore passengers because they focus primarily on conveying responsive actions to the driver's words. When a passenger is present in the vehicle, the driver experiences the inconvenience of having to speak a voice command in consideration of the passenger's presence.

[0006] Therefore, there is a need to research voice recognition functions that can provide response actions based on the driver's voice commands while taking the presence of passengers into consideration. Summary of the invention

[0007] According to one aspect of the present disclosure, a method for performing a response action to a voice command by referring to a passenger profile may include: identifying a passenger in a vehicle; and performing a response action corresponding to a voice command issued by a driver or a passenger in the vehicle based on the passenger profile of the passenger. In some implementations, the passenger profile includes at least one of a vehicle control range, a point of interest (POI) search option, or preset information.

[0008] According to another aspect of the present disclosure, a device configured to perform a response action to a voice command based on a passenger profile may include: a memory storing a passenger profile including at least one of a vehicle control range, a POI search option, or preset information; and a processor configured to (i) identify a passenger in a vehicle; and (ii) perform a response action corresponding to a voice command issued by a driver or passenger in the vehicle based on the passenger profile of the passenger. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a diagram illustrating an embodiment of a passenger specific scenario.

[0010] Figure 2 is a block diagram illustrating an embodiment of a response device.

[0011] Figure 3 is a diagram illustrating an embodiment of a passenger profile.

[0012] Figure 4 is a block diagram illustrating an embodiment of a speech recognition system.

[0013] Figure 5 is a diagram illustrating an exemplary relationship between a vehicle and a voice recognition system.

[0014] Fig. 6A and Figure 6B is a diagram illustrating an embodiment of a passenger identification process and a response action process.

[0015] Figure 7 is a flow chart illustrating an embodiment of a method for operating a response device. DETAILED DESCRIPTION

[0016] The present disclosure is directed to methods and apparatus for providing passenger-specific services by performing response actions based on voice commands with reference to pre-stored passenger profiles.

[0017] The embodiments of the present disclosure are described in detail below using various drawings. In addition, when the components are marked with reference numerals in each figure, the same reference numerals are used as much as possible even for components shown in different drawings. In addition, in the description of the present disclosure, if it has been determined that the specific description of the relevant known configuration or function may obscure the key points of the present disclosure, its detailed description is omitted.

[0018] When describing components according to embodiments of the present disclosure, symbols such as first, second, i), ii), a), and b) may be used. These symbols are used only to distinguish components from other components. The identity or sequence or order of the components is not limited by the symbols. In this specification, when a component "includes" or "is equipped with" an element, it means that the component may further include other elements, without excluding other elements unless explicitly stated to the contrary. In addition, when an element in the written description and claims is described as "for" performing or implementing a stated function, step, instruction set, etc., the element may also be regarded as "configured to" do so.

[0019] Each component of the device or method can be implemented in hardware or software, or in a combination of hardware and software. In addition, the function of each component can be implemented in software. A microprocessor or processor can execute the function of the software corresponding to each component.

[0020] Figure 1 is a diagram illustrating an embodiment of a passenger specific scenario.

[0021] See also Figure 1 , both the driver and the passengers can be present in the vehicle. For example, the driver can be an adult child taking his parent to the hospital.

[0022] The driver may say the phrase "Hey, Hyundai, guide me to a <hospital>" to find a route to a specific hospital that the passenger frequently visits. "Hey, Hyundai" may be the wake-up word that triggers voice recognition. At this point, it may be assumed that the driver did not say the name of the specific hospital.

[0023] If no additional information is given, the vehicle may not know the name of the specific hospital, so that the vehicle can only provide information about hospitals around the current location. In this case, the driver must say the name of the specific hospital or select the specific hospital from a list of nearby hospitals provided to find a route to the hospital.

[0024] An exemplary response device can reduce such inconvenience by performing a response action to a voice command based on a passenger profile. The response device can apply a passenger profile to the vehicle, enabling information to be received from a simple speech of the driver or passenger.

[0025] For example, without explicitly naming a particular hospital, the response device may identify hospitals that the passenger frequently visits based on point of interest (POI) data within the passenger profile.

[0026] In another embodiment, when a child or pet is in the vehicle, the response device may provide services specifically for the child or pet using the child's or pet's profile.

[0027] Therefore, the response device can provide passenger-specific services by pre-storing the profile of each passenger and performing a response action to the voice command based on the passenger's profile.

[0028] The response device may be implemented by at least one of a voice recognition system or a vehicle.

[0029] Figure 2 is a block diagram illustrating an embodiment of a response device. Figure 3 is a diagram illustrating an embodiment of a passenger profile.

[0030] See also Figure 2 , the response device 200 may include a memory 250 and a processor 280, the memory 250 stores information for providing a service requested by a user, and the processor 280 is configured to control the operation of the vehicle.

[0031] When the response device 200 is implemented inside a vehicle, the response device 200 may further include a microphone 210 configured to receive a user's voice, a speaker 220 configured to output sound to provide a service requested by the user, a camera 230 configured to obtain an image around the vehicle, an interface 240 configured for input or output to provide a service requested by the user, a communication module 260 configured to communicate with an external device, and vehicle control components 271, 272, 273, 274, 275.

[0032] The microphone 210 may be provided at a position inside the response device 200 where the user's voice may be received. The user who inputs the voice into the microphone 210 provided in the response device 200 may be a driver. The microphone 210 may be installed at a position such as a steering wheel, a center instrument panel, a roof lining, or a mirror in the cabin to receive the driver's voice or speech.

[0033] In some embodiments, two or more microphones 210 may be provided to receive speech from passengers in the rear seats. The microphone 210 for receiving speech from the rear seat occupants may be disposed in the armrest of the front seat or the rear seat, or may be disposed in the rear seat door, the B-pillar or the C-pillar.

[0034] In addition to the user's voice, various audio sounds generated around the microphone 210 may be input to the microphone 210. The microphone 210 may output an audio signal corresponding to the input audio signal; the output audio signal may be processed by the processor 280 or transmitted to an external server device through the communication module 260.

[0035] In addition to microphone 210, response device 200 may include interface 240 configured to receive user commands through manual input such as touch. Interface 240 may include input devices in the form of buttons or push-to-go switches in the AVN area of ​​the center instrument panel, in the transmission area, or on the steering wheel.

[0036] In some implementations, the interface 240 may include an input device provided on a door of each seat or an input device provided on an armrest of a front seat or a rear seat and receiving a control command related to a passenger seat.

[0037] In some implementations, interface 240 may include a touch pad that implements a touch screen by integrating an input device with a display.

[0038] The camera 230 may obtain at least one of an internal image or an external image of the response device 200. Therefore, the camera 230 may be provided inside the response device 200, outside the response device 200, or inside and outside the response device 200.

[0039] The interface 240 may include an AVN display, a cluster display, or a head-up display (HUD) disposed on the center instrument panel of the response device 200. Additionally or alternatively, the interface 240 may include a rear seat display disposed on the rear of the headrest of the front seat, allowing passengers in the rear seat to see the interface; in the case of a multi-seat vehicle, the response device 200 may include a display mounted on the headliner.

[0040] The displays may be located anywhere as long as they are visible to the user of response device 200, without any limitation on the number or location of the displays.

[0041] The memory 250 may store a program that controls the processor 280 to execute the method. For example, the program may include a plurality of instructions that can be executed by the processor 280, and the method may be executed when the processor 280 executes the plurality of instructions.

[0042] The communication module 260 may exchange signals with other devices by adopting at least one of various wireless communication methods such as Bluetooth, 4G communication, 5G communication, or Wi-Fi. Additionally or alternatively, the communication module 260 may exchange information with other devices via a cable connected to a universal serial bus (USB) port, an auxiliary (AUX) port, etc. In some implementations, the communication module 260 may include at least one of a transceiver or an antenna.

[0043] In some implementations, the communication module 260 may be equipped with two or more communication interfaces supporting different communication methods, thereby enabling information and signals to be exchanged with two or more other devices.

[0044] For example, the communication module 260 can communicate with a mobile device located inside the response device 200 through Bluetooth communication to receive information acquired by the mobile device or stored in the mobile device (such as user image, user voice, contact information, schedule, etc.), and can communicate with a server through 4G or 5G communication to transmit user voice and receive signals to provide the requested service to the user. In some implementations, the communication module 260 can exchange signals with the server through a mobile device connected to the response device 200.

[0045] In some implementations, the response device 200 may include: a navigation device configured to provide route guidance; an air conditioning device 271 configured to control the interior temperature; a window control device 272 configured to open or close the windows; and a seat heating device 273 configured to heat the seats; a seat control device 274 configured to adjust the position, height or angle of the seat; and a lighting device 275 to adjust the interior lighting.

[0046] The above devices may provide functions related to the response device 200, and some devices may be omitted depending on the vehicle model and options. In addition, other devices may be included in addition to the above devices. Because the drive-related configuration of the response device 200 is well known, its description will be omitted in this disclosure.

[0047] The processor 280 may turn on / off the microphone 210 , process or store a voice signal input to the microphone 210 , or transmit the input voice to another device through the communication module 260 .

[0048] In some implementations, processor 280 may control the display to display images and control speaker 220 to produce audio output.

[0049] In some implementations, the processor 280 may perform various controls related to the response device 200. For example, based on a user command input through the microphone 210 or the interface 240, the processor 280 may control at least one of the navigation device, the air conditioning device 271, the window control device 272, the seat heating device 273, the seat control device 274, or the lighting device 275.

[0050] In some implementations, processor 280 may perform at least a portion of the functionality of a speech recognition system to analyze the speech of a driver or passenger. Figure 4 Describe the speech recognition system in detail.

[0051] The processor 280 may include at least one memory storing a program for performing the above-mentioned operations and operations described later and at least one processor executing the stored program.

[0052] In some implementations, the processor 280 may perform a process for registering a passenger profile, a passenger identification process, and a response action process to provide passenger-specific services.

[0053] In the process for registering passenger profiles, each passenger's profile may be stored in the memory 250. The passenger profile may record settings customized for a particular passenger.

[0054] See also Figure 3 The profile of passenger 1 includes at least one of identification data, vehicle control range, POI search option or preset information.

[0055] Responding device 200 may use the identification data to identify occupant 1 within the vehicle.

[0056] The identification data may include at least one of the name, voice characteristics, or device information of the passenger 1. Various identification data may be used to identify the passenger 1.

[0057] The vehicle control range may refer to a control range of the vehicle customized for the passenger 1 .

[0058] The vehicle control range may include control ranges for various components of the vehicle. The vehicle can be controlled within the vehicle control range. The vehicle control range may include at least one of a window control range, a seat control range, a temperature control range, an air conditioning control range, a lighting control range, a volume control range, or a media control range. For example, the window control range may limit the window to one-half or one-third open. As another example, the media control range may limit content to only adult content or only children's content.

[0059] The POI search option may refer to information for providing customized POIs to the passenger 1 in response to a voice command related to a POI search.

[0060] The POI search options may include a preferred POI search option, a no-child area exclusion option, or a no-pet area exclusion option. The POI search options may include various search options.

[0061] In some implementations, the preference POI search option is an option to search for POIs frequently visited by the passenger 1 among POIs related to the keyword in the voice command or to search for preset POIs according to the corresponding keyword.

[0062] The child-free area exclusion option may be an option for excluding child-free area POIs among those POIs associated with a specific keyword within a voice command.

[0063] The no-pet-zone exclusion option may be an option for excluding no-pet-zone POIs among those POIs associated with a specific keyword within a voice command.

[0064] The preset information may refer to a preset of the vehicle customized for the passenger 1 .

[0065] The preset information may include presets for various components of the vehicle. The preset information may include at least one of a window control preset, a seat control preset, a temperature control preset, an air conditioning control preset, a lighting control preset, a volume control preset, or a media preset. When the passenger 1 is identified or when the profile of the passenger 1 is applied to the vehicle, the vehicle's windows, seat posture, temperature, air conditioning, lighting, media, etc. may be controlled based on the preset information. For example, the media preset may control running a specific channel or a specific content application.

[0066] In some embodiments, the profile of passenger 1 may include Figure 3 The profile of passenger 1 in Figure 3 A specific part of the profile of passenger 1 in .

[0067] The passenger profile may be registered in different ways. The driver or passenger may register the passenger profile through the interface 240 of the response device 200. Alternatively, the passenger profile may be registered to the passenger's own device and then transmitted to the response device 200. In some implementations, when the response device 200 is implemented in a server, the passenger profile may be registered in the vehicle and then transmitted to the response device 200 in the server.

[0068] A passenger profile can be created by the driver or the passenger. A passenger profile can be created before or during a vehicle journey.

[0069] refer to Figure 2 In the passenger identification process, the processor 280 may identify the passenger in the vehicle using at least one of the initial speech received from the microphone 210 or the connection status of the passenger device. For example, the initial speech may refer to speech from the driver or the passenger.

[0070] In some implementations, the processor 280 identifies a passenger in the vehicle by comparing the name in the initial utterance with the name in the passenger profile. For example, the processor 280 converts the initial utterance into text using a speech recognition system, classifies the intent of the initial utterance from the text, and extracts a slot associated with the intent from the text. When the intent is "provide passenger information", the processor 280 may identify the extracted slot as the name of the passenger. As another embodiment, the processor 280 may identify the initial utterance (i.e., the name in the converted text) as the name of the passenger using a predetermined grammar. The processor 280 may identify the passenger by comparing the identified name with the name in the passenger profile.

[0071] In some implementations, the processor 280 identifies a passenger in the vehicle by comparing the speech features of the initial utterance with the speech features in the passenger profile. Specifically, the processor 280 may use an extraction model to extract the speech features of the passenger's initial utterance. The speech features may be in vector form. The processor 280 may identify the passenger by comparing the extracted speech features with the speech features in the passenger profile. For example, the speech features may be represented by an embedded vector or a spectrogram, which conveys an audio signal or frequency characteristics.

[0072] In some embodiments, the processor 280 can identify a passenger in the vehicle by comparing the device information in the passenger profile with the device connected to the vehicle. Specifically, the processor 280 can identify the passenger by comparing the ID (identification) of the device connected to the vehicle with the device identification information in the passenger profile. The vehicle can be connected to the driver's device and the passenger's device at the same time.

[0073] Between the passenger identification process and the response action process, the processor 280 may inquire whether to apply the profile of the identified passenger to the vehicle. If the application of the passenger profile is approved, the processor 280 applies the passenger profile to the vehicle.

[0074] On the other hand, processor 280 may immediately apply the profile of the identified passenger to the vehicle in response to identification of the passenger.

[0075] In some implementations, in response to the identification of the passenger, the processor 280 controls the vehicle based on preset information in the passenger profile. Various control presets in the preset information of the passenger profile may be applied to the vehicle.

[0076] For example, if the passenger is identified as a child or a pet, the processor 280 may perform a response action such as closing the window, setting the temperature of the air conditioning device near the passenger to 20 degrees, and setting the wind direction of the air conditioning device to downward. Because the vehicle is controlled according to the preset information, the driver or passenger does not need to control the window or seat individually and manually. Therefore, the response action improves convenience.

[0077] In some embodiments, the passenger profile may further include a preferred music playlist for the passenger, and the processor 280 may play the preferred music playlist in response to the identification of the passenger.

[0078] In some embodiments, the preset information of the passenger profile includes a preset for each vehicle state. In addition to the preset for the vehicle to which the passenger profile is applied, the preset information may include a preset for the vehicle when the engine is turned off. For example, the preset information may include a window control preset for the vehicle when the engine is turned off. When the engine of the vehicle is turned off, the windows of the vehicle may be slightly opened based on the window control preset. When the passenger is identified as a child or a pet, the windows may be opened even when the engine of the vehicle is turned off, thereby preventing the temperature inside the vehicle from rising due to strong sunlight.

[0079] During the response action, the microphone 210 may receive a voice command from a driver or a passenger in the vehicle, and the processor 280 may perform the response action according to the voice command with reference to the passenger profile.

[0080] Specifically, processor 280 may use a voice recognition system to identify the intent and slot of a voice command, and determine a response action based on the identified intent and slot.

[0081] The voice command may be converted into text, the text may be classified into one of the preset intents, and slots associated with the intent may be extracted from the phrases in the text. For example, the intent may be classified into at least one of vehicle control or POI search; vehicle control-related slots may include driver window, passenger seat, temperature, or air conditioning; and POI search-related slots may include area name, building name, restaurant name, or abbreviations thereof.

[0082] Specifically, processor 280 may reference a passenger profile applied to the vehicle when determining a response action to a voice command.

[0083] In some embodiments, when the voice command is related to vehicle control, the processor 280 can control the vehicle by referring to the vehicle control range within the passenger profile. When the vehicle is controlled by a voice command from the driver or passenger, the vehicle can be controlled within the range specified in the vehicle control range.

[0084] For example, the window control range in the passenger profile of a pet may be set to one-half or one-third. Even if a voice command such as "open the window" is received, the processor 280 may generate an action to open the vehicle window by half or one-third. When the passenger is a child or a pet, the response device 200 may be controlled with reference to the vehicle control range, thereby ensuring the safety of the passenger.

[0085] In some implementations, when the voice command is related to a POI search, the processor 280 can provide at least one POI that satisfies the POI search options within the passenger profile.

[0086] The POI search options may include a preferred POI search option, a no-child area exclusion option, or a no-pet area exclusion option. The POI search options may include various search options.

[0087] The processor 280 may search for POIs using the preferred POI search options within the passenger profile and provide POIs that satisfy the POI search options.

[0088] For example, in response to the "hospital" keyword in the preferred POI search option of the passenger profile, "hospital A" may be pre-registered as a preferred POI. When the driver says "guide me to <hospital>", even if the vehicle is far from hospital A, the processor 280 may provide "hospital A" as a response action by referring to the preferred POI search option in the passenger profile. The processor 280 may first provide "hospital A".

[0089] The child-free zone exclusion option in the passenger profile may be activated. For example, when the driver says "guide me to <restaurant>", the processor 280 may consider the child-free zone exclusion option and provide search results that exclude restaurants designated as child-free zones as a response action.

[0090] The no-pet zone exclusion option in the passenger profile may be activated. For example, when the driver says "guide me to <restaurant>", the processor 280 may consider the no-pet zone exclusion option and provide search results that exclude restaurants designated as no-pet zones as a response action.

[0091] In this manner, the response device 200 can provide a passenger-specific service by pre-storing a profile of each passenger and performing a response action with reference to the passenger's profile according to a voice command.

[0092] Figure 4 is a block diagram illustrating an embodiment of a speech recognition system.

[0093] See also Figure 4 , the speech recognition system 400 can recognize and understand the user's speech and provide a response corresponding to the user's speech. In some embodiments, the user can refer to the driver or the passenger.

[0094] In some implementations, the speech recognition system 400 may include: a speech recognition module 410, configured to convert a user's speech and voice commands into text; a natural language understanding module 420, configured to determine the intent of the user's speech; and a response generation module 430, configured to perform processing to provide a response corresponding to the user's speech intent.

[0095] In some implementations, the speech recognition system 400 may further include a dialog manager configured to manage the overall dialog between the speech recognition system 400 and the user.

[0096] The speech recognition module 410 may acquire user speech received by a microphone in the vehicle and convert the user's speech into an input sentence using at least one speech-to-text (STT) engine. The STT engine may convert the speech signal into text by applying a speech recognition algorithm or a deep learning model to the speech signal representing the user's speech.

[0097] For example, the speech recognition module 410 may extract a feature vector from the user utterance using a feature vector extraction technique such as cepstrum, linear prediction coefficient (LPC), Mel-frequency cepstral coefficients (MFCC), or filter bank energy.

[0098] The speech recognition module 410 can obtain a recognition result by comparing the extracted feature vector with a trained reference pattern. In some implementations, the speech recognition module 410 can use an acoustic model that models and compares the signal features of the speech, or a language model that models the language order relationship of words or syllables corresponding to the vocabulary of the recognition.

[0099] The speech recognition module 410 may convert the user's speech into an input sentence in text form based on a model using machine learning or deep learning.

[0100] In addition, before applying voice recognition, the voice recognition module 410 may pre-process the voice signal corresponding to the user's speech. For example, the voice recognition module 410 may perform pre-processing to reduce noise in the voice signal.

[0101] The natural language understanding module 420 may classify the intent of the user utterance included in the input sentence using at least one natural language understanding (NLU) engine, and extract slots indicating meaningful information related to the utterance intent.

[0102] For example, a slot may refer to a semantic object used to provide a response according to an utterance intent. A slot may be predefined for each utterance intent. The role of a slot may be determined by an utterance intent. For example, in the input sentence "Lead me to Yanghwa Bridge", the role of "Yanghwa Bridge" represents a point of interest, but in the input sentence "Play Yanghwa Bridge", the role of "Yanghwa Bridge" may refer to a song name.

[0103] In some implementations, the NLU engine can determine the utterance intent and slot of the user who input the sentence by comparing the input sentence with a predetermined grammar. For example, when the predetermined grammar is "call <someone>" and the input sentence is "call Hong Gil-Dong", the NLU engine determines that the utterance intent is "make a call" and the slot value is "Hong Gil-Dong".

[0104] In some implementations, the NLU engine may use tokenization, deep learning models, and other related techniques to determine the utterance intent and slots of a sentence input by a user.

[0105] Specifically, the NLU engine can segment the input sentence into morpheme-level tokens. A morpheme can refer to the smallest unit of meaning that cannot be further analyzed. In addition, the NLU engine can label each token with a portion of speech.

[0106] The NLU engine can project tokens into a vector space. Each token or combination of tokens can be converted into an embedding vector. To improve performance, sequence embedding and position embedding can be performed together.

[0107] The NLU engine can determine the speech intent and slot of the input sentence by grouping the embedding vectors or applying the first deep learning model and the second deep learning model to the embedding vectors respectively. For example, the first deep learning model can be a recursive neural network pre-trained to classify speech intent in response to the input of the embedding vector. The second deep learning model can be a recursive neural network pre-trained to determine the slot in response to the input of the embedding vector.

[0108] In addition, the natural language understanding module 420 may extract information such as a domain, an entity name, or a voice action from an input sentence using an NLU engine.

[0109] The domain may refer to information for identifying the subject of the user's utterance. For example, domains representing different subjects such as vehicle control, information provision, text transmission, and navigation functions may be determined based on the input sentence.

[0110] Entity names can refer to proper nouns, such as names of people, places, organizations, time, date, and currency. Named Entity Recognition (NER) is the task of identifying entity names in a sentence and determining the type of the recognized entity name. Through entity name recognition, important keywords can be extracted from a sentence to understand the meaning of the sentence.

[0111] Speech action analysis can be the task of analyzing the intent of an utterance. Speech action analysis can be used to determine the intent of an utterance, such as whether the user is asking a question, making a request, responding, or simply expressing an emotion.

[0112] Information such as domain, entity name, or voice action can be used for at least one of operations such as classification of user utterance intent, determination of slots, or generation of a response based on the user utterance.

[0113] The response generation module 430 may perform processing to provide a response corresponding to the user's utterance intent.

[0114] The response generation module 430 can provide responses in different forms. The response generation module 430 can provide responses through visual, auditory, or tactile interfaces based on the user's speech.

[0115] The response generation module 430 can use a generative model to generate response information that can be easily recognized by passengers. The response generation module 430 can use a generative model to generate a complete sentence from information such as utterance intent, slots, domains, entity names, and voice actions.

[0116] For example, if the intention of the user's utterance is “vehicle-related control”, the response generation module 430 may transmit a signal processing result of performing the vehicle-related control to the vehicle.

[0117] In another embodiment, if the user's utterance is intended to "provide specific information", the response generation module 430 may use the slot to search for specific information and provide the searched information to the user terminal. If necessary, the information search may be performed on another external server.

[0118] In yet another embodiment, if the intention of the user's utterance is to “provide specific content,” the response generation module 430 may request that the target content be transmitted from an external server that provides the content.

[0119] In yet another embodiment, if the intention of the user's utterance is “continuation of a simple conversation”, the response generation module 430 may generate response content according to the user's utterance and output the response visually or audibly.

[0120] Hereinafter, an operational embodiment of the speech recognition system 400 will be described.

[0121] For example, when the input sentence is "When should I change the engine oil?" the NLU engine splits the input sentence into "engine", "oil", "when", "change", and "year" morphemes, and converts each morpheme into a vector. Afterwards, the NLU engine classifies the speech intent corresponding to the vector based on the similarity between the vectors and their positions in the vector space. In the above embodiment, the classified speech intent is "confirm the replacement of disposable parts". The NLU engine extracts the word elements "engine" and "oil" into slots according to the intent of "confirming the replacement of disposable parts". Subsequently, the response generation module 430 can provide a sentence such as "the engine oil change cycle is 15,000km" based on the intent of confirming the replacement of disposable parts and the slot values ​​of "engine" and "oil".

[0122] In another embodiment, if the input sentence is “Let's go home”, the domain is identified as “navigation”, the utterance intention is “route setting”, and the slot for executing the control corresponding to the utterance intention is “departure point, destination”.

[0123] In yet another embodiment, if the input sentence is "turn on the air conditioner", the domain is identified as "vehicle control", the speech intention is "air conditioner power on", and the slot for executing the control corresponding to the speech intention is "air conditioner". Another slot may be "temperature, air volume".

[0124] Meanwhile, the speech recognition system 400 may include at least one processor and a memory including at least one command, and execute the command by the at least one processor to perform the functions of the speech recognition module 410, the natural language understanding module 420, and the response generation module 430. The speech recognition system 400 may further include a communication unit for communicating with an external device.

[0125] Figure 5 The relationship between the vehicle and the speech recognition system is shown.

[0126] refer to Figure 5 , the response device and the voice recognition system can be implemented on at least one of the vehicle 510 or the server 520.

[0127] In some implementations, both the response device and the voice recognition system can be implemented in the vehicle 510. The controller of the response device can directly perform the functions of the voice recognition system. The response device can perform the process for registering the passenger profile, the passenger identification process, and the response action process.

[0128] In some implementations, the response device may be implemented in the vehicle 510, and the voice recognition system may be implemented in the server 520. The response device in the vehicle 510 transmits the speech or voice command from the driver or the passenger to the voice recognition system in the server 520. The voice recognition system processes the speech or voice command to generate information or control commands required by the passenger, and transmits the information or control commands to the response device in the vehicle 510. The response device coordinates the information or control commands with reference to the passenger profile.

[0129] As an example of a passenger identification process, the response device may transmit an utterance to a voice recognition system, receive passenger information recognized by the voice recognition system, and identify the passenger based on the received passenger information.

[0130] As an example of a response action process, the response device may transmit the speech to a speech recognition system, receive a control command determined by the speech recognition system, modify the control command with reference to the passenger profile, and control the vehicle using the modified control command.

[0131] As another example, the response device may transmit the speech to a speech recognition system, receive the intent and slot determined by the speech recognition system, generate a control signal corresponding to the intent and slot by referring to the passenger profile, and control the vehicle using the control signal.

[0132] The function of the response generation module in the speech recognition system is performed by the response device.

[0133] In some implementations, the response device and the voice recognition system may be implemented in the server 520. The response device may perform a process for registering a passenger profile, a passenger identification process, and a response action process. The response device receives an initial speech from the vehicle and identifies the passenger based on the initial speech. Thereafter, the response device receives a voice command of the vehicle and performs a response action corresponding to the voice command. Here, the response action is to transmit a control signal or information corresponding to the voice command to the vehicle.

[0134] In addition to the above, the speech recognition module, the natural language understanding module, and the response generation module in the speech recognition system may be distributed on the vehicle 510 and the server 520 .

[0135] Fig. 6A and Figure 6B The passenger identification process and response action process are shown.

[0136] refer to Fig. 6A , during the passenger identification process, the speech from the passenger and the display device 600 installed in the response device are shown.

[0137] The screen of the display device 600 may be divided into two divided screens 610, 620. The first divided screen 610 displays various information. The second divided screen displays information related to passenger-specific services.

[0138] The response device is installed in the vehicle.

[0139] exist Fig. 6A , the speaker says "Hey Hyundai, I'm riding with Hong Gil-Dong."

[0140] The response device extracts a name from the utterance and identifies a passenger in the vehicle by comparing the extracted name with a name in identification data of a pre-stored passenger profile.

[0141] Specifically, the response device converts the speech in the form of audio data into input text using the STT function. The response device extracts the name from the input text.

[0142] For example, to extract a name from the input text, the response device may use a predefined grammar. The response device may store a sentence "Hey, Hyundai, I'm riding with <who>" and extract the name corresponding to <who> from the input text by referring to the stored sentence.

[0143] In another embodiment, the response device may perform the function of the speech recognition system to extract the name in the utterance. The response device classifies the utterance intention and extracts the slot by applying the function of the speech recognition system to the input text. When the intention of the utterance is "provide passenger information", the response device uses the extracted slot as the name in the utterance.

[0144] exist Fig. 6A middle,<Hong Gil-Dong> is extracted as a name.

[0145] The responding device then searches the passenger profile for the name<Hong Gil-Dong> If you find a file containing <honggil-dong>If a passenger profile with a matching name is found, the response device designates the identified passenger profile as the profile of the passenger in the vehicle.

[0146] In some implementations, the response device may perform a passenger identification process using a speech recognition system installed in a server. The response device transmits an input speech to the speech recognition system in the server. The speech recognition system converts the input speech into an input text, extracts a name in the input text using a predefined grammar or intent / slot, and transmits the extracted name to the response device. The speech recognition system may store a passenger profile, search for a passenger profile corresponding to the extracted name, and transmit the searched passenger profile to the response device. The response device obtains a profile of a passenger in the vehicle based on the name or passenger profile received from the speech recognition system.

[0147] In addition to the above, the responding device can identify the passenger by comparing the result obtained by applying the deep learning model to the voice features extracted from the input speech with the voice features registered in the passenger profile. Alternatively, the responding device can determine whose device is connected to the vehicle by referring to the passenger profile.

[0148] The responding device may display the name of the identified passenger on the second split screen 620<Hong Gil-Dong> .

[0149] refer to Figure 6B , during the response action process, the passenger's speech and the display device 600 installed in the response device are shown.

[0150] Identify passengers in the vehicle<Hong Gil-Dong> , and will<Hong Gil-Dong> The profile is applied to the vehicle.<Hong Gil-Dong> In the profile of , the window control range can be set to half.

[0151] The speaker says, "Hey Hyundai, open the right rear window."

[0152] The response device may control the window as a response action corresponding to the input speech. When controlling the window, the response device refers to<Hong Gil-Dong> The vehicle control scope in the profile.

[0153] Specifically, the response device converts the input utterance into input text.

[0154] The response device uses the function of the speech recognition system to extract the speech intent and slots from the input text. The intent of the utterance is "open the car window", and the slots are "right", "back seat" and "car window".

[0155] The response device generates a control signal for opening the right rear window of the vehicle based on the speech intention and the slot.<Hong Gil-Dong> profile and generates a control signal to control the right rear window to open halfway.

[0156] The response device sends a control signal to the window control device to control the right rear window of the vehicle.

[0157] The response device may display content related to the window control applied to the vehicle on the second split screen 640 .

[0158] Figure 7 is a flow chart illustrating a method for operating a response device.

[0159] In the process for registering passenger profiles, the response device stores the profile of each passenger in advance.

[0160] The passenger profile includes at least one of identification data, vehicle control range, POI search option, or preset information.

[0161] The identification data includes at least one of the passenger's name, voice characteristics, or device information.

[0162] The vehicle control range includes at least one of a window control range, a seat control range, a temperature control range, an air-conditioning control range, a lighting control range, a volume control range, and a media control range.

[0163] The POI search option includes at least one of a preferred POI search option, a no-child area exclusion option, or a no-pet area exclusion option.

[0164] The preset information includes at least one of a window control preset, a seat control preset, a temperature control preset, an air conditioning control preset, a lighting control preset, a volume control preset, or a media preset.

[0165] See also Figure 7 , the response device identifies the passengers in the vehicle S710.

[0166] The responding device uses at least one of the passenger's name, voice characteristics, or device information in the passenger profile to identify the passenger from at least one of speech from the driver or the passenger, a connection status of the passenger's device, or any combination thereof.

[0167] At the same time, the response device can control the vehicle based on preset information in the passenger profile in response to identifying the passenger.

[0168] The response device performs a response action corresponding to the voice command issued by the driver or the passenger in the vehicle by referring to the passenger profile of the identified passenger S720.

[0169] When the voice command is related to vehicle control, the response device controls the vehicle by referring to the vehicle control range in the passenger profile.

[0170] When the voice command relates to a POI search, the response device provides at least one POI searched using a POI search option in the passenger profile.

[0171] In some implementations, passenger specific services may be provided by performing response actions based on voice commands with reference to pre-stored passenger profiles.

[0172] The technical effects of the present disclosure are not limited to the above-mentioned technical effects, and those skilled in the art to which the present disclosure belongs may understand other technical effects not mentioned herein from the following description.

[0173] The different techniques described herein may be implemented in digital electronic circuits, or in computer hardware, firmware, software, or a combination thereof. The implementation may be in the form of a computer program tangibly embodied in a computer program product (i.e., an information carrier, such as a machine-readable storage device (computer-readable medium) or a propagation signal) for processing or controlling the operation of a data processing device (e.g., a programmable processor, a computer, or multiple computers). Computer programs such as the computer programs mentioned above may be written in any form of programming language, including compiled or interpreted languages, and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment. The computer program may be deployed to run on a single computer or multiple computers at a site or distributed on multiple sites and interconnected by a communication network.

[0174] In addition, the components of the present disclosure may use integrated circuit structures such as memories, processors, logic circuits, lookup tables, etc. These integrated circuit structures perform each function described herein by controlling one or more microprocessors or other control devices. In addition, the components of the present disclosure may be specifically implemented by a part of a program or code including one or more executable instructions for performing specific logical functions, and executed by one or more microprocessors or other control devices. In addition, the components of the present disclosure may include or be implemented as a central processing unit (CPU), a microprocessor, etc. that performs various functions. In addition, the components of the present disclosure may store instructions executed by one or more processors in one or more memories.

[0175] For example, processors suitable for processing computer programs include general and special microprocessors, and one or more processors of any kind of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer may include at least one processor that executes instructions and one or more memory devices that store instructions and data. Typically, the computer will also include one or more large-capacity storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data or operably coupled to receive data from or transmit data to or both. For example, information carriers suitable for embodying computer program instructions and data include semiconductor memory devices, such as magnetic media (such as hard disks, floppy disks, and tapes), optical media (such as compact disk read-only memory (CD-ROM) and digital video disks (DVD)), magneto-optical media (such as floppy disks), read-only memory (ROM), random access memory (RAM), flash memory, erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), etc. The processor and memory can be supplemented or incorporated into a dedicated logic circuit.

[0176] The processor may execute an operating system and software applications executed on the operating system. In addition, the processor device may access, store, manipulate, process, and generate data in response to software execution. For convenience, there are cases where a single processor device is used, but those skilled in the art will understand that the processor device may include multiple processing elements and / or multiple types of processing elements. For example, the processor device may include multiple processors or a single processor and a single controller. Other processing configurations (such as parallel processors) are also possible.

[0177] Additionally, non-transitory computer readable media may be any available media that can be accessed by a computer and may include both computer storage media and transmission media.

[0178] This specification includes details of various specific implementations, but they should not be construed as limiting the scope of any invention or the scope of what is claimed, and should be construed as descriptions of features that may be unique to a particular embodiment of a particular invention. In the context of each embodiment, the specific features described herein may also be implemented in conjunction with a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented in multiple embodiments independently or in any suitable subcombination. Further, although features may operate in a particular combination and may initially be described as being claimed as such, in some cases, one or more features from the claimed combination may be excluded from the combination, and the claimed combination may be modified into a subcombination or a variant of the subcombination.

[0179] Likewise, although operations are described in a particular order in the accompanying drawings, it should not be understood that such operations must be performed in the particular order shown or in sequential order to achieve the desired results or that all described operations should be performed. In some cases, multitasking and parallel processing can be advantageous. In addition, the separation of various device components of the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and devices can generally be integrated into a single software product or packaged into multiple software products.

[0180] The above description is only an illustration of the technical concept of the present embodiment. A person of ordinary skill in the art can make various modifications and changes without departing from the basic features of each embodiment. Therefore, the present embodiment is not intended to limit but to describe the technical concept of the present embodiment. The scope of the technical concept of the embodiment is not limited by these embodiments. The protection scope of various embodiments should be interpreted by the attached claims. All technical concepts falling within the scope of their equivalents should be interpreted as being included within the scope of the present embodiment.

Claims

1. A method for performing an action in response to a voice command based on a passenger profile, the method comprising: Identify passengers in a vehicle; as well as performing a response action corresponding to a voice command issued by the identified passenger or driver based on the passenger profile of the passenger, The passenger profile includes at least one of a vehicle control range, a point of interest (POI) search option, and preset information.

2. The method according to claim 1, wherein: Performing the response action includes controlling the vehicle according to the vehicle control range within the passenger profile based on the voice command related to vehicle control.

3. The method according to claim 2, wherein: The vehicle control range includes at least one of a window control range, a seat control range, a temperature control range, an air conditioning control range, a lighting control range, a volume control range, and a media control range.

4. The method according to claim 1, wherein: Performing the responsive action includes providing at least one POI searched using the POI search option within the passenger profile based on the voice command associated with POI search.

5. The method according to claim 4, wherein: The POI search option includes at least one of a preferred POI search option, a no-child area exclusion option, and a no-pet area exclusion option.

6. The method according to claim 1, further comprising: In response to the identification of the passenger, the vehicle is controlled based on the preset information in the passenger profile.

7. The method according to claim 6, wherein: The preset information includes at least one of a window control preset, a seat control preset, a temperature control preset, an air conditioning control preset, a lighting control preset, a volume control preset, and a media preset.

8. The method according to claim 1, wherein: The passenger profile includes at least one of the passenger's name, voice characteristics, and device information, and Wherein, identifying the passenger comprises: The passenger is identified based on at least one of the name of the passenger within the passenger profile, the voice feature, and the device information using at least one of a connection status of the passenger's device or an utterance from the passenger or the driver.

9. A device configured to perform an action in response to a voice command based on a passenger profile, the device comprising: A memory storing instructions and a passenger profile, the passenger profile including at least one of a vehicle control range, a point of interest (POI) search option, and preset information; as well as A processor configured to execute the instructions to perform operations including: Identify the occupants of the vehicle, and A response action corresponding to a voice command issued by the identified passenger or driver is performed based on the passenger profile of the passenger.

10. The device according to claim 9, wherein: The processor is configured to control the vehicle according to the vehicle control range within the passenger profile based on the voice command related to vehicle control.

11. The device according to claim 10, wherein: The vehicle control range includes at least one of a window control range, a seat control range, a temperature control range, an air conditioning control range, a lighting control range, a volume control range, and a media control range.

12. The device according to claim 9, wherein: The processor is configured to provide at least one POI searched using the POI search option within the passenger profile based on the voice command related to POI search.

13. The device according to claim 12, wherein: The POI search option includes at least one of a preferred POI search option, a no-child area exclusion option, and a no-pet area exclusion option.

14. The device according to claim 9, wherein: The processor is configured to control the vehicle based on the preset information within the passenger profile in response to the identification of the passenger.

15. The device according to claim 14, wherein: The preset information includes at least one of a window control preset, a seat control preset, a temperature control preset, an air conditioning control preset, a lighting control preset, a volume control preset, and a media preset.

16. The device according to claim 9, wherein: The passenger profile includes at least one of the passenger's name, voice characteristics, and device information, and Wherein, the processor is configured to identify the passenger based on at least one of the name of the passenger, the voice characteristics and the device information within the passenger profile, using the connection status of the passenger's device or at least one of the speech from the passenger or the driver.

Images