Artificial intelligence partner interacting with people in motor vehicle

Abstract

The present invention relates to an artificial intelligence partner for conversation with a person in a vehicle using a chat robot, a prompter for providing a prompt to the chat robot, and a context source for providing a context to the chat robot.

Description

Technical Field

[0001] The present invention relates to automotive electronic devices, and in particular to electronic devices for alleviating the effects of loneliness on drivers. Background Art

[0002] Soon after the advent of motor vehicles, people discovered that long drives quickly became boring and tiring. To some extent, the burden of long drives can be alleviated by having a like-minded passenger along for the ride. The journey seems to go by faster when there is a talkative person around.

[0003] Unfortunately, traveling doesn’t always involve meeting people you can talk to. Not surprisingly, one of the first accessories to gain popularity was the car radio. This soon evolved into tape players, CD players, and most recently, MP3 players, with the invention of portable audio storage media. What all of these products have in common is the ability to provide audio entertainment to ease the fatigue of long journeys.

[0004] While these devices do provide effective audio entertainment, they are essentially passive forms. There is no real way for people to interact with them, other than singing along to the radio. As such, they cannot replace passengers who are skilled in the art of conversation. Summary of the invention

[0005] In one aspect, the present invention relates to an artificial intelligence companion configured to have a conversation with a person. In an embodiment of the present invention, the artificial intelligence companion includes a chatbot and a prompter that provides a prompt to the chatbot. The prompt is at least partially derived from a microphone audio signal provided by a microphone that receives a first round of conversation from the person. The chatbot generates a second round of conversation to be delivered to the person through a speaker. In some embodiments, a context source provides context to the chatbot. In these embodiments, the chatbot uses the context to generate a second round of conversation to be delivered to the person through the speaker.

[0006] In some embodiments, the chatbot is a robot that implements a generative model. In other embodiments, the chatbot implements a large language model.

[0007] In other embodiments, the context used by the chatbot to generate the second turn includes personal information about the person; the chatbot uses the context-generated second turn to advance the conversation by inviting the person to participate in another person's turn.

[0008] In other embodiments, the chatbot is further configured to generate a second round of dialogue based on context obtained from the first round of dialogue or a previous round of the first round of dialogue.

[0009] In some embodiments, the person is a vehicle occupant. In these embodiments, the person is the driver of the vehicle; the person is a passenger. In some embodiments, the chatbot has been trained to provide entertainment to a young child, in which case the person is a young child in the back seat of a vehicle in need of attention, in which case the AI ​​companion plays the role of an AI soother.

[0010] In other embodiments, the person is walking, such as after leaving the vehicle, and the AI ​​companion is implemented on a portable device carried by the person. In this way, the AI ​​companion can be used at home, when going out for a walk, when going out shopping (e.g., buying clothes), and after leaving the vehicle. In the latter case, in an embodiment of the present invention, the AI ​​companion is implemented in the vehicle, but when the AI ​​companion learns that the person is leaving the vehicle, it smoothly transitions to being implemented on the portable device of the person.

[0011] In some embodiments, the first round of conversation is also the first round of the entire conversation. However, in other embodiments, the zeroth round provided by the artificial intelligence companion is before the first round. Accordingly, there are conversations initiated by humans and conversations initiated by artificial intelligence companions in the operation.

[0012] In the embodiment of the present invention, the context includes information related to the vehicle's motion state; the context includes the vehicle's future position; the context includes information related to the person; the context includes sensor readings in the vehicle; the context includes biometric information related to the person; the context involves the person's emotional state; the context includes prosodic information related to the person's voice. In each of the above cases, the second round of dialogue is generated at least partially relying on the context.

[0013] In some other embodiments, the context includes one or more of the following relevant information: current affairs related information, vehicle environment related information, personnel related information, local news, new movie related information, book related information and weather related information.

[0014] In some other embodiments, the chatbot is configured to adjust the cognitive load of the conversation in response to the context; the chatbot is trained to have a non-judgmental conversation; the chatbot is trained to debate with a person; the chatbot selects the topic of the conversation based on the context. In these embodiments, the degree of non-judgment of the conversation can be adjusted; the degree of persistence of the artificial intelligence companion in the debate can also be adjusted.

[0015] In other embodiments, the AI ​​companion is configured to sense a person's lack of attention and initiate a conversation in response to the sensed lack of attention.

[0016] In some other embodiments, the artificial intelligence companion also includes a classifier for receiving information related to the person. The information comes from one or more of the following: microphone audio signals, camera signals of a camera pointed at the person, tactile sensors that sense the force exerted by the person's activities, and sensors that measure the physiological signals of the person. The camera includes, for example, a camera that is sensitive to visible radiation and a camera that is sensitive to radiation outside the visible light range (such as infrared radiation, ultraviolet radiation, and ionizing radiation, such as X-rays).

[0017] The classifier classifies the person as belonging to one or more categories and provides this information to the context source for use as additional context, or to the prompter for generating prompts, or to both to achieve the above two purposes. The claims and description mention that the "first round" is provided by the person and the "second round" is provided by the AI ​​companion. But this does not imply that the conversation must be initiated by a person. It is entirely possible that the "zeroth round" is initiated by the AI ​​companion, and the "first round" is just a response to the "zeroth round".

[0018] As used herein, "round two" does not include perfunctory responses to questions asked in "round one." Therefore, "round two" is configured to use contextual information to advance the conversation.

[0019] On the other hand, the present invention proposes a method, including: collecting context information for conducting a conversation with a person in a vehicle; determining whether the person in the vehicle shows signs of inattention; and causing an artificial intelligence companion to conduct a conversation with the person. In such practice, causing the artificial intelligence companion to conduct a conversation includes: receiving an audio signal from a microphone, the audio signal including a first round of conversation; using the first round of conversation to provide prompts to a chatbot; providing context to the chatbot; causing the chatbot to generate a second round of conversation based on the prompts and the context; and causing the second round of conversation to be output to the person through a speaker.

[0020] In the practice of the above method, the chatbot implements a large language model; the chatbot implements a generative model.

[0021] In the practice of the above method, the first round of dialogue is preceded by the zeroth round of dialogue generated by the chatbot; the first round starts the dialogue; the zeroth round of dialogue before the first round is generated by the chatbot.

[0022] In the zero-turn initiated conversation practice, the chatbot proactively starts a conversation when the context indicates that something is happening. In this practice, for example, the context indicates that the driver is about to embark on a long trip because a destination has been set, or because the vehicle is identified as traveling along a route that has historically led to long trips. By proactively initiating the conversation, the chatbot immediately begins communicating with the driver, delaying the onset of driver fatigue. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and other features of the present invention will be clearly understood in conjunction with the accompanying drawings and specific embodiments, in which:

[0024] Figure 1 The vehicle is shown to be equipped with an artificial intelligence companion that can have a conversation with the person.

[0025] Figure 2 Shows Figure 1 An alternative embodiment of the illustrated artificial intelligence companion.

[0026] Figure 3 Shows Figure 1 and Figure 2 The method performed by the artificial intelligence companion shown. DETAILED DESCRIPTION

[0027] Figure 1 An AI-implemented artificial intelligence companion 10 is shown engaging in a conversation 12 with a person 14 in a vehicle 16. In such a conversation 12, the artificial intelligence companion 10 issues a companion turn 18, and the person issues a person turn 20. Person turns 20 and partner turns 18 are alternated to form the conversation 12. Insofar as the companion turns 18 and person turns 20 are built on common semantic content, the conversation 12 develops desirable characteristics for entertaining the person 14. In contrast to the passive entertainment provided by listening to music or audiobooks, the conversation 12 provides interactive entertainment.

[0028] In order for the AI ​​companion 10 to be able to meaningfully participate in the conversation 12, it is beneficial to equip it with a camera 22 and a microphone 24 as eyes and ears. This also provides the AI ​​companion 10 with a basis for evaluating the person's turn 20. It is also beneficial to provide the AI ​​companion 10 with a speaker 26, allowing the companion 10 to communicate the companion's turn 18 to the person 14.

[0029] The microphone 24 provides a microphone audio signal 28 to a speech recognition unit 30. The output of the speech recognition unit 30 is provided to a prompter 32 together with a video signal 34 from the camera 22 and an in-cabin sensor signal 36 from an in-cabin sensor 38. These three signals work together to provide the prompter 32 with a basis for assessing the status of the person 14, thereby providing appropriate prompts 40 to the AI ​​chatbot 42.

[0030] The role of the chatbot is to formulate the companion turn 18 to be delivered to the person 14, aiming to keep the conversation 12 at an appropriate cognitive load level. In order to achieve this, the chatbot 42 needs something to talk about. It is therefore beneficial to have the chatbot 42 communicate with a context source 44.

[0031] In order to promote a more realistic conversation 12 and eliminate the illusion that the chatbot 42 is a stranger to the unfamiliar person 14, it is beneficial for the chatbot to have some personal information 46 about the person 14. Such personal information 46 is obtained by providing a classifier 48 that receives signals from the microphone 24, camera 22, and in-cabin sensors 38.

[0032] The classifier 48 extracts information from the above signals indicating the person's age, emotional state, socioeconomic clues inferred from the voice (such as the person's accent), and any other biometric information provided by the signals. Such personal information helps provide the chatbot 42 with information about how to manage the conversation 12 with a particular person 14.

[0033] Figure 1 In the illustrated embodiment, the classifier 48 provides the background information 46 to the context source 42, which aggregates the background information 46 with other information and provides the resulting context 50 to the chatbot 42. Based on the context 50 provided by the context source 44, the video signal 34 provided by the camera 22, and the prompt 40 provided by the prompter 26, the chatbot 42 provides the partner turn 18 to the text-to-speech converter 52, which then outputs the corresponding audio speaker signal 54 to the speaker 26.

[0034] Figure 2 In the illustrated alternative embodiment, the classifier 48 provides the context information to the prompter 32, which then uses the context information as a basis for the chatbot 42 to generate the prompt 40. Figure 2 The embodiment shown is combined with Figure 1 The operation of the embodiments described is the same.

[0035] To promote good interaction with the person 14, the AI ​​companion 10 preferably conducts the conversation in a relatively predictable manner. Therefore, when the AI ​​companion 10 needs to respond to a person turn 20 to advance the conversation 12 through a partner turn 18, it is preferred that the partner turn 18 is a turn that would have occurred in the context of one or more previous person turns 20 and / or partner turns 18. It is therefore advantageous that the chatbot 42 is implemented as a language generation model that has been trained to talk about a variety of topics and do so in a non-judgmental manner.

[0036] In addition, the chatbot 42 has been trained to conduct the conversation in a manner that avoids excessive cognitive load. Specifically, the chatbot 42 has been trained to maintain enough cognitive load to stimulate the person 14, but not enough to tend to monopolize the person's attention. The former is, for example, a conversation 12 about sports or current events. The latter is, for example, a conversation 12 about the foundations of quantum mechanics.

[0037] Of course, the optimal cognitive load and topic of conversation 12 are inherently individual. Therefore, it is beneficial that context source 44 provides chatbot 42 with intelligence about person 14. For example, if person 14 is known to be interested in a particular area based on context information (e.g., the person is an opera fan), chatbot 42 can adjust conversation 12 accordingly (e.g., discuss a selected opera plot).

[0038] In addition, the chatbot 42 has also been trained to be an active listener, prompting the person 14 to talk about the events of the day or various difficulties that the person 14 may be facing.

[0039] In some embodiments, the chatbot 42 is also configured to imitate a specific individual. For example, the chatbot can be trained using text from a collection of Jane Austen books, so that the chatbot 42 can imitate Jane Austen's speaking style and conduct conversations with a British Regency accent.

[0040] The context source 44 includes different types of context 50. The context 50 provides the chatbot 42 with material for the conversation 12.

[0041] In some cases, context 50 is provided by camera 22. For example, if video signal 34 indicates that person 14 is wearing golf attire, chatbot 42 may recognize that conversation 12 centered around golf may be exciting to person 14. Similarly, in-cabin sensors 38 provide usable information from within the vehicle cabin that is relevant to determining the status of the person but is not available through camera 22 or microphone 24.

[0042] In some embodiments, the in-cabin sensor 38 is a tactile sensor. Some examples include a tactile sensor coupled to a vehicle seat, in which case a person frequently moving in the seat may indicate restlessness. Other examples include a tactile sensor coupled to a vehicle steering wheel that can measure the person's grip strength, thereby providing a stress indicator.

[0043] In some cases, the in-cabin sensors 38 measure physiological signals, which can infer the state of the person. Some examples include sensors that measure skin electrical properties (such as skin conductance and / or capacitance). Such sensors include, for example, skin electrodermal activity sensors, skin electrodermal response sensors, and sweat sensors. Based on the information provided by these sensors, the emotional state, wakefulness level, or stress response of the person can be inferred. When placed on the steering wheel or armrest, the sweat sensor analyzes the composition of human sweat and provides information about hydration level and electrolyte balance.

[0044] Other examples of in-cabin sensors 38 that measure physiological signals include in-cabin sensors 38 that measure electrical activity in the human brain. Such in-cabin sensors 38 can be used to identify drowsiness and provide signals that can be used by the brain-vehicle interface. Examples of such in-cabin sensors 38 include electroencephalographs. Electroencephalographs built into vehicle seats provide a particularly suitable location for such measurements.

[0045] Other examples of in-cabin sensors 38 include sensors that measure and record the electrical activity of the human heart. Examples of such in-cabin sensors 38 include electrocardiographs for diagnosing and monitoring heart conditions. When the artificial intelligence companion 10 detects an abnormal situation, it will suggest to the person 14 to seek medical assistance or suggest calling an ambulance.

[0046] A common feature of the in-cabin sensors 38 to date is the use of electrical signals as the basis for collecting information. However, there are also in-cabin sensors that rely on optical signals. Examples of such in-cabin sensors include pulse oximeters, which are used to monitor blood oxygen levels and heart rate. Such sensors rely on modulated electromagnetic radiation in the visible light range and / or infrared range.

[0047] The context source 44 may use other forms of context 50 , including readings from various sensors 52 within the vehicle 16 , such as seat weight sensors, cabin air temperature, and various in-vehicle settings.

[0048] In some embodiments, the chatbot 42 changes the topic of the conversation 12 in response to turns received from the person 14. Thus, a conversation 12 that begins with the weather may eventually turn into a discussion of the Earth's convective system and its impact on the weather.

[0049] One useful context 50 is derived from the vehicle's motion history. For example, a vehicle velocity vector change (such as a turn or speed change) may indicate that the person 14 is not paying close attention to the road.

[0050] The vehicle movement history also includes the vehicle's current and past locations, which can be obtained from the GPS unit. In the case where the GPS has been programmed to reach a destination, future locations also form part of the vehicle movement history. In this case, the AI ​​companion 10 can use the location information combined with context-related knowledge to comment on the availability of nearby services (such as nearby restaurants or gas stations).

[0051] Another source of information includes information about the physical and mental state of the person, including biometric information. Such information can be obtained from the video signal 34, which can be used to understand whether the person is slouching excessively, thereby indicating fatigue. Additional information can be obtained from the microphone audio signal 28, which can reveal aspects of the person's speech prosody that may indicate fatigue, such as slurred or incoherent speech. More information can also be obtained from the tactile signal 36, which can indicate, for example, that the person 14 is gripping the steering wheel with abnormal force.

[0052] In some embodiments, biometric information provides a basis for identifying a person, which in turn provides the AI ​​companion 10 with an opportunity to personally greet the person as they enter the vehicle, thereby establishing rapport. In some cases, the AI ​​companion 10 may also be able to recognize that the person is not seated in the usual position within the vehicle, but rather that an unidentifiable person is driving. In this case, the AI ​​companion 10 has an opportunity to inquire about the new driver and perhaps seek an introduction.

[0053] More examples of context 50 include current events related information, vehicle environment related information, people related information, local news, new movie information or book information, and weather information. Such information is usually available from a cloud-based server 56.

[0054] The biometric information combined with the vehicle movement history helps to assess the state of the person in order to identify when intervention through dialogue is needed to stimulate the person's alertness level. Accordingly, in some embodiments, the artificial intelligence companion 10 continuously monitors such information to determine whether it should actively engage in dialogue with the person 14.

[0055] In some embodiments, certain components of the AI ​​companion 10 or portions thereof are located at a remote location of the vehicle, such as on a cloud-based server 54. These components include the chatbot 42, the speech recognition unit 30, and the prompter 32.

[0056] Reference Figure 2 , the method of using the artificial intelligence companion 10 to entertain the person 14 through the conversation 12 includes receiving the person turn 20 from the person (step 58), and generating prompts 40 and then providing them to the chatbot 42 (step 60). At the same time, the chatbot 42 also receives the context 50 (step 62). Based on the context 50 and the prompts 40, the chatbot 42 generates the companion turn 18 of the conversation 12 (step 64).

[0057] The foregoing describes the novel technical solutions and preferred embodiments of the present invention.

Claims

1. A device comprising an artificial intelligence companion, wherein the artificial intelligence companion is configured to have a conversation with a person in a vehicle, the artificial intelligence companion comprising: Chatbots; a prompter for providing a prompt to the chatbot, the prompt being derived from a microphone audio signal provided by a microphone receiving the first round of the conversation from the person; as well as a context source, providing context to the chatbot, Therein, the chatbot uses the context to generate a second round of the conversation to be delivered to the person via a speaker.

2. The device according to claim 1, wherein: The chatbot implements a large language model.

3. The device according to claim 1, wherein: The context used by the chatbot to generate the second round includes personal information about the person.

4. The device according to claim 1, wherein: The chatbot is further configured to generate the second round based on context obtained from at least one of the first round and a previous round occurring before the first round.

5. The device according to claim 1, wherein: The context includes information about the motion state of the vehicle.

6. The device according to claim 1, wherein: The context includes a future position of the vehicle.

7. The device according to claim 1, wherein: The context includes information about the person, and the chatbot generates the second round of the conversation using the information about the person.

8. The device according to claim 1, wherein: The context includes readings from sensors in the vehicle, and the chatbot uses the readings from the sensors when generating the second turn of the conversation.

9. The device according to claim 1, wherein: The context includes biometric information about the person, and the chatbot uses the biometric information about the person in generating the second round of the conversation.

10. The device according to claim 1, wherein: The context includes prosodic information about the person's speech, and the chatbot conducts the second round of the conversation based at least in part on the prosodic information.

11. The device according to claim 1, wherein: The context includes emotional state information about the person, and the chatbot uses the emotional state information to generate the second round of the conversation.

12. The device according to claim 1, wherein: The context includes one or more of the following relevant information: current affairs related information, vehicle environment related information, personnel related information, local news, new movie related information, book related information and weather related information.

13. The apparatus according to claim 1, wherein: The chatbot is configured to adjust a cognitive load of the conversation in response to the context.

14. The apparatus according to claim 1, wherein: The chatbot has been trained to have a non-judgemental conversation.

15. The apparatus according to claim 1, wherein: The chatbot has been trained to engage in a debate with the person.

16. The apparatus according to claim 1, wherein: The chatbot selects a topic for the conversation based on the context.

17. The apparatus according to claim 1, wherein: The artificial intelligence companion is configured to sense a lack of attention by the person and to initiate the conversation in response to sensing the lack of attention.

18. The apparatus according to claim 1, wherein: The artificial intelligence companion also includes a classifier for receiving information about the person and using the information to provide context to the context source, wherein the information is obtained from the microphone audio signal.

19. The apparatus according to claim 1, wherein: The artificial intelligence companion also includes a classifier for receiving information about the person and using the information to provide context to the prompter for generating the prompt, wherein the information is obtained from a camera signal of a camera pointed at the person.

20. The apparatus of claim 1, wherein: The artificial intelligence companion also includes a classifier for receiving information about the person and using the information to provide context to the prompter for generating the prompt, wherein the information is obtained from a tactile sensor in mechanical communication with the person.

21. The apparatus of claim 1, wherein: The artificial intelligence companion also includes a classifier for receiving information about the person and using the information to provide context to the prompter for generating the prompt, wherein the information is obtained from a sensor that obtains physiological signals of the person.

22. A method for enabling an artificial intelligence companion to have a conversation with a person in a vehicle, wherein: Enabling the artificial intelligence companion to conduct the conversation includes: receiving a first round of the conversation via a microphone; providing a prompt to the chatbot, the prompt derived at least in part from the first round; providing context to the chatbot; Based on the prompt and the context, a second round of the dialogue is generated.

23. The method according to claim 22, further comprising: Determining that the person in the vehicle is showing signs of inattention, and causing the artificial intelligence companion to initiate the conversation with the person.

24. The method according to claim 22, wherein: The first round is preceded by a zero-round dialogue generated by the chatbot.

25. The method according to claim 22, wherein: The first turn initiates the conversation.

26. The method of claim 22, wherein: The conversation was initiated by a zeroth turn prior to the first turn, the zeroth turn having been generated by the chatbot.

27. The method of claim 22, wherein: The chatbot implements a generative model.

28. The method of claim 22, wherein: The zeroth round before the first round starts the conversation, and the zeroth round is actively generated by the chat robot when the incident occurs.

29. The method of claim 22, wherein: The conversation is initiated by a zeroth round prior to the first round, the zeroth round being actively generated by the chatbot based on the expected journey length of the person.

Images