Method and device for selectively ignoring voice by automobile assistant
Through line-of-sight detection technology and voice filters, car assistants accurately distinguish discourse in a multi-person environment, solving the problem of car assistants misunderstanding or ignoring user instructions in a multi-person occupant environment, and improving response accuracy.
Patent Information
- Application Number
- CN202410815327.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-06-22
- Filing Date
- 2024-06-24
- Publication Date
- 2024-12-24
AI Technical Summary
Hyundai Motor Assistant has difficulty accurately distinguishing between the words directed to themselves and the words not directed to themselves in a multi-person environment, resulting in the possibility of misunderstanding or ignoring user instructions.
Through the line of sight detection technology, the camera and logic circuit in the vehicle are used to determine whether the occupant's line of sight intersects with a predetermined eye area, thereby determining whether to respond to the occupant's voice commands, and a speech filter is used to combine the line of sight detector and a machine vision system to distinguish the speech.
Effectively avoids misunderstandings or ignoring user commands by car assistants, improves accuracy when interacting with individual occupants, and reduces dialogue ambiguity in a multi-person environment, ensuring the correct response of voice commands.
Smart Images
Figure CN119176098A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of automobile assistants, and in particular to a method and device for selectively ignoring voice by an automobile assistant. Background Art
[0002] In modern vehicles, it is not uncommon to have a car assistant with a voice interface. In response to the vehicle occupant's words, the car assistant can answer certain questions or perform certain commands.
[0003] To use a car assistant, people usually wake it up in some way. This can be done by pressing a button. Alternatively, it can be done by saying a special word called a "wake-up word".
[0004] The new generation of car assistants eliminates this step. In such cases, people can speak to the car assistant in much the same way as people might speak to a person (address). Therefore, people can simply say the question and expect the car assistant to respond. Summary of the invention
[0005] The present invention attempts to help the car assistant distinguish those utterances that are directed to another entity and those that are not directed to another entity.To do this, the present invention relies on the observation that a person who wishes an utterance to be directed to a target entity will often look at the target entity.
[0006] In one aspect, the invention includes causing a car assistant in a vehicle to ignore a first utterance, the first utterance being uttered by a first occupant of the vehicle. In this method, causing the car assistant to ignore the first utterance includes receiving the first utterance, determining a first line of sight of the first occupant, and causing the car assistant to ignore the first utterance based at least in part on the first line of sight.
[0007] Among these practices are those that further include causing the car assistant to respond to a second utterance. In such practices, causing the car assistant to respond to the second utterance includes receiving the second utterance from a second occupant of the vehicle, determining a line of sight of the second occupant, and causing the car assistant to respond to the second utterance based at least in part on the line of sight of the second occupant.
[0008] In yet other practices, the method further includes causing the car assistant to respond to the second utterance and causing the car assistant to respond to the second utterance by receiving the second utterance from the first occupant, determining a second line of sight of the first occupant, and causing the car assistant to respond to the second utterance based at least in part on the second line of sight.
[0009] Other practices include those in which the method includes determining that the first line of sight fails to intersect an eye region that has been defined for the automotive assistant for use by a first occupant of the vehicle, and causing the automotive assistant to ignore the first utterance based at least in part on the line of sight failing to intersect the eye region.
[0010] In other practices, the method further includes causing the car assistant to respond to the second utterance by: receiving the second utterance; determining a second line of sight, which is the line of sight of anyone who made the second utterance; determining that the second line of sight intersects with an eye region designated for anyone who made the second utterance; and causing the car assistant to respond to the second utterance based at least in part on the line of sight having intersected with the eye region.
[0011] In another aspect, the invention features a device that includes a car assistant and a speech filter that causes the car assistant to ignore speech from an occupant based on the occupant's line of sight.
[0012] These embodiments are those in which the speech filter includes a gaze detector that determines whether the occupant's gaze intersects an eye region that has been defined for use by the occupant when speaking to the automotive assistant.
[0013] In an embodiment is also an embodiment wherein the speech filter comprises a logic circuit that receives a signal indicative of an utterance and a signal indicative of whether a line of sight of the occupant uttering the utterance intersects an eye region of the automotive assistant.
[0014] In all of the foregoing, the method steps are performed in a non-abstract manner and all device components are non-abstract in nature. As used herein, "non-abstract" shall mean the opposite of "abstract" as the courts in the United States have defined that term as of the filing date of this application. Abstract implementations and practices are hereby abandoned. Also abandoned are practices in which humans perform the method without the use of a writing instrument and a substrate on which to write. Anyone who interprets the claims in the opposite way will demonstrate that the claims can be interpreted without regard to the specification and without derogating from the applicant's rights as its own lexicographer.
[0015] These and other features of the invention will become apparent from the following embodiments and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Figure 1 A vehicle cabin equipped with a speech filter is shown.
[0017] Figure 2 Shows Figure 1 An interior view of a vehicle cabin in which an occupant is looking toward the eye area.
[0018] Figure 3 Shows Figure 2 configuration, but the occupants are looking away from the passenger area.
[0019] Figure 4 Shown for Figure 1An exemplary architecture of a speech filter is shown in .
[0020] Figure 5 A flow chart illustrating an embodiment of ignoring utterances uttered by an occupant of a vehicle based on the occupant's line of sight. DETAILED DESCRIPTION
[0021] Figure 1 A vehicle 10 is shown having a passenger compartment 12 with occupants 14 seated in seats 16. Each seat 16 has an associated microphone 18, speaker 20, camera 22, and display 24.
[0022] The vehicle further includes a car assistant 26. In response to spoken utterances from the occupant 14, the car assistant 26 performs various tasks.
[0023] In some embodiments, the car assistant 26 is an assistant that does not require a wake word. This allows the occupant 14 of the vehicle to interact with the car assistant 26 in much the same way as the occupant 14 might interact with another occupant.
[0024] The foregoing features are useful when there is only one occupant 14 in the vehicle 10. In this case, the car assistant 26 faces little ambiguity in determining whether it is being spoken to.
[0025] When multiple occupants 14 are present, conversations may occur between them. Although the car assistant 26 can detect the resulting speech, it does not have a good basis for determining when it is being spoken to. As a result, the car assistant 26 may find itself entering into a conversation in which it has no right to participate. This can occur even if only one passenger 14 is present. For example, when the occupant 14 is using a phone, the voice assistant 26 may still misinterpret the occupant's speech as a command to be executed.
[0026] To address this difficulty, the car assistant 26 attempts to adopt the human practice of using eye contact to determine whether it is being spoken to, and therefore anticipate a response to a particular utterance. Of course, this is simply impossible. After all, the car assistant 26 lacks eyes to make contact.
[0027] However, an "eye volume 30" may be specified to enable the automotive assistant 26 to simulate an effect similar to eye contact. The automotive assistant 26 is configured to respond to an utterance of the occupant 14 only if the occupant 14 has a line of sight 28 intersecting the eye volume 30 during a time interval contemporaneous with the utterance.
[0028] Figure 2 The situation where the occupant's line of sight 28 intersects the eye volume 30 is shown. In this case, the occupant's speech will be accepted by the automotive assistant 26.
[0029] In contrast, Figure 3 It is shown that the occupant's line of sight 28 misses the eye volume 30. Therefore, the car assistant 26 will infer that the occupant 14 is talking to someone else. After doing so, the car assistant 14 will ignore the occupant's words.
[0030] In practice, different occupants 14 have different eye volumes 30. After all, an occupant 14 sitting in the back seat of a vehicle may find it inconvenient to use an eye volume 30 at the opposite end of the vehicle cabin 12.
[0031] A useful location for a particular occupant's eye volume 30 is slightly below or to the side of the line of sight 28 for viewing the exterior of the vehicle 12, such as Figure 2 and 3 Embodiments also include those in which the eye volume 30 is coextensive with the microphone 18 located at the occupant seat 16 or coextensive with the display 24 closest to the occupant seat 16. Other useful locations for locating the eye volume 30 may be experimentally determined by careful observation of the interactions of the occupants 14 in the vehicle 12.
[0032] Generally, it is not required that the eye volume 30 be continuous. Thus, in some embodiments, the eye volume 30 is formed by the union of a first volume coextensive with the seat's display 24 and a second volume coextensive with the seat's microphone 18.
[0033] See now Figure 1 and Figure 4 , the above configuration of the car assistant 26 is produced by a speech filter 32 standing between the car assistant 26 and the microphone 18. The speech filter 32 sends a trigger 34 to the car assistant 26 only when there is a high probability that the occupant 14 is speaking to the car assistant 26. As used herein, "high probability" means that the occupant 14 exhibits the first attribute and the second attribute.
[0034] The first attribute is that the occupant 14 has uttered a speech. To detect this, the speech filter 32 includes a speech detector 36 that receives signals from the microphones 18. Based on these signals, the speech detector 36 determines whether speech has occurred and which microphone 18 is associated with the strongest speech signal. This provides a basis for inferring the position of the occupant 14 who is speaking.
[0035] When speech is detected, the speech detector 36 provides a speaker signal 38 to the gaze detector 40. The speaker signal 38 indicates that the occupant 14 has uttered a word. It also identifies the microphone 18 that is considered to be closest to the occupant 14.
[0036] The speech detector 36 also sends an arming signal 42 to the logic circuit 44. As a result of the arming signal 42, if the gaze detector 40 determines that the gaze 28 passes through the eye volume 30, the logic circuit 44 prepares to send the trigger 34.
[0037] Based on the speaker signal 38, the gaze detector 40 attempts to determine the presence of the second of the two attributes described above, namely, whether the speaking occupant 14 has a line of sight 28 that intersects or passes through the occupant's eye volume 30. To do this, the gaze detector 40 observes the output of whichever camera 22 is best positioned to assess the occupant's line of sight 28. Based on that output, the gaze detector 40 determines whether the occupant's line of sight 28 intersects the eye volume 30.
[0038] If the gaze detector 40 determines that the second attribute is present, it sends an activation signal 46 to the logic circuit 44. This causes the logic circuit 44 to send the trigger 34 to the car assistant 26, which then proceeds to respond to the occupant 14 to whom it is speaking.
[0039] Although it may not be obvious, the speech filter 32 is not a conventional filter that removes specific frequency components from the frequency domain representation of the speech signal. Rather, the illustrated speech filter 32 filters the ambient acoustic environment in order to detect a signal having the attributes of human speech.
[0040] In some embodiments, the speech filter 32 omits the logic circuit 44. Instead, the automotive assistant 26 responds to the occupant's speech as long as the occupant 14 is looking toward the eye volume 30. If no one is actually speaking, the automotive assistant 26 will not respond. If the occupant 14 utters a speech while looking away from the eye volume 30, the automotive assistant 26 will not respond to the speech.
[0041] Embodiments of the gaze detector 30 include those incorporating a machine vision system that detects the location of a feature in the occupant's eye and tracks the location of that feature, thereby inferring the gaze 28. In this case, a suitable feature is the occupant's pupil.
[0042] Also among the embodiments of the line of sight detector 30 are those that illuminate the eyes of an occupant using electromagnetic radiation outside the visible range and then observe the position of the reflection as a basis for inferring the line of sight 28 of the occupant.
[0043] A typical occupant 14 has a face vector and an eye vector. The face vector is perpendicular to the face, while the eye vector is perpendicular to the plane defined by the pupil of the eye.
[0044] Different methods can be used to control a person's line of sight 28. One method is to turn the person's head, i.e., change the direction of the face vector. This method achieves a more dramatic and obvious effect. Another method is a more subtle method of rotating the person's eyes about their respective sockets, thereby changing the direction of their eye vector. This is useful, for example, if a person is driving and trying to establish eye contact with the briefest of glances. Yet another method that requires more effort is to maintain the person's line of sight 28 by rotating the person's face vector and eye vector in opposite directions. A suitable line of sight detector 30 is one that adapts to these different movements of the face vector and the eye vector to correctly determine the line of sight 28.
[0045] Figure 5 A flow chart is shown of an implementation with two entry points: wake word input after detection of the wake word (step 48), and push-to-talk input after use of the push-to-talk button (step 50).
[0046] Upon detecting the wake word (step 48), the speech filter 32 determines the location of the occupant 14 who uttered the wake word within the vehicle 10 (step 52). The speech filter 32 then determines the line of sight 28 of the speaker (step 54), and based at least in part on the line of sight 28, the speech filter 32 determines whether the speech is likely to be directed to the car assistant (step 56). If this is not possible, the car assistant 26 ignores the speech (step 58). Otherwise, the car assistant 26 begins a voice conversation (step 60).
[0047] As used herein, if for a preselected threshold in the interval defined by zero and one, inclusive, the probability that the speech is directed to the car assistant 26 exceeds the threshold, then the speech is likely to have been directed to the car assistant 26. As used herein, if the speech is unlikely to be directed to the car assistant 26, then the speech is considered unlikely to be directed to the car assistant 26.
[0048] The voice conversation includes waiting for speech (step 62), and when speech is detected (step 64), determining the location of the speaker within the vehicle (step 66) and the line of sight 28 of the speaker (step 68).
[0049] The speech filter 32 determines whether the speech may have been directed to the car assistant (step 70). If this turns out to be impossible, the car assistant 26 ignores the speech (step 72) without closing the voice session (step 74). Otherwise, the car assistant accepts the speech (step 76) and processes the speech (step 78). If the car assistant 26 determines that no further information is needed (step 80), the requested action is performed (step 82). Otherwise, the car assistant 26 waits for further speech (step 62).
[0050] In one example, the occupant 14 looks at his designated eye volume 30 and asks, “Are there any restaurants nearby?”, and the car assistant 26, having recognized that it is being spoken to, responds, “Yes, here is a list of restaurants. The first one is the Etruscan restaurant half a mile away. The second one is…” and begins enumerating the list.
[0051] There may be three scenarios at this time.
[0052] In the first scenario, the occupant 14, who has a particular fondness for Etruscan cuisine, looks at his designated eye volume 30 and says, “I want to go to this place, Etruria.” The car assistant 26 recognizes that he is being spoken to again, and the car assistant 26 interrupts his delivery and responds, “OK, I’ll plan a route to the Etruscan restaurant.”
[0053] In the second scenario, occupant 14 is with another occupant. Recognizing that perhaps not all share his love of Etruscan cuisine, occupant 14 looks away from his eye region 30 and asks, "Do you like Etruscan cuisine?" The car assistant detects this change in the occupant's gaze 28 and infers that he is no longer being addressed. Therefore, it does not attempt to answer the question.
[0054] The third scenario proceeds similarly to the second scenario, except that the occupant 14 does not take his eyes off the road. Therefore, the line of sight 28 may graze the eye volume 30. In this case, the car assistant 26 recognizes the ambiguity and perhaps seeks clarification by asking, "Please say that again. Are you talking to me?"
[0055] Upon detecting that the push-to-talk button has been actuated (step 50), the automotive assistant 26 determines whether a voice recognition session has already begun (step 84). If not, it initiates one (step 60). The control flow then proceeds as described above.
[0056] In some cases, while the car assistant 26 is saying something (i.e., playing a prompt), the occupant 14 barges in (step 86). For example, as noted above in three scenarios, after having been asked for a list of nearby restaurants, the car assistant 26 may be listing a long list of such restaurants.
[0057] In this case, it is useful to determine whether the occupant's utterance is an attempt to speak to the automotive assistant 26. If it is not such an attempt, the automotive assistant 26 ignores the utterance (step 88) and continues its prompting. On the other hand, if the speech filter 32 recognizes the utterance as an attempt to speak to the automotive assistant 26 because the line of sight 28 has been observed, the automotive assistant 26 interrupts its prompting (step 90) and begins to wait for speech (step 62). At this point, the control flow proceeds as already discussed.
[0058] Having described the invention and its preferred embodiments, what is claimed as new and protected by patent is defined in the claims.
Claims
1. A method for selectively ignoring speech by a car assistant, comprising causing the car assistant in a vehicle to ignore a first utterance, the first utterance being made by a first occupant in the vehicle, wherein causing the car assistant to ignore the first utterance comprises receiving the first utterance, determining a first line of sight of the first occupant, and causing the car assistant to ignore the first utterance based at least in part on the first line of sight.
2. The method according to claim 1, further comprising: Causing the automotive assistant to respond to a second utterance, wherein causing the automotive assistant to respond to the second utterance comprises: receiving the second utterance from a second occupant of the vehicle, determining a line of sight of the second occupant, and causing the automotive assistant to respond to the second utterance based at least in part on the line of sight of the second occupant.
3. The method according to claim 1, further comprising: Causing the automotive assistant to respond to a second utterance, wherein causing the automotive assistant to respond to the second utterance comprises: receiving the second utterance from the first occupant, determining a second line of sight of the first occupant, and causing the automotive assistant to respond to the second utterance based at least in part on the second line of sight.
4. The method according to claim 1, wherein: The method further includes determining that the first line of sight fails to intersect an eye region that has been defined for the automotive assistant for use by the first occupant of the vehicle, and causing the automotive assistant to ignore the first utterance based at least in part on the line of sight failing to intersect the eye region.
5. The method according to claim 1, further comprising: Cause the car assistant to respond to a second utterance, wherein causing the car assistant to respond to the second utterance comprises: receiving the second utterance; determining a second line of sight, which is the line of sight of anyone who utters the second utterance; determining that the second line of sight intersects with an eye region designated for anyone who utters the second utterance; and causing the car assistant to respond to the second utterance based at least in part on the intersection of the line of sight with the eye region.
6. A device for selectively ignoring speech by a car assistant, comprising a car assistant and a speech filter, wherein the speech filter enables the car assistant to ignore speech from a passenger based on the passenger's line of sight.
7. The device according to claim 6, wherein: The speech filter includes a gaze detector that determines whether the occupant's gaze intersects an eye region that has been defined for use by the occupant when speaking to the automotive assistant.
8. The device according to claim 6, wherein: The speech filter includes logic circuitry that receives a signal indicative of an utterance and a signal indicative of whether a line of sight of an occupant uttering the utterance intersects an eye region of the automotive assistant.
9. The device according to claim 6, wherein: The speech filter includes a speech detector, a line of sight detector, and a logic circuit, wherein the speech detector sends an arming signal to the logic circuit when detecting the occupant's speech, wherein the line of sight detector sends an activation signal in response to determining that the occupant's line of sight intersects with an eye volume, wherein the logic circuit sends a trigger to the automotive assistant when both the activation signal and the arming signal are detected, wherein the automotive assistant interacts with the occupant in response to the trigger signal.
10. The device according to claim 6, wherein: The speech filter filters the ambient acoustic environment of a microphone in the vehicle to detect signals having attributes of human speech.
11. The device according to claim 6, wherein: The speech filter includes a gaze detector that sends a signal to the automotive assistant indicating that the occupant's gaze intersects an eye region that has been defined for use by the occupant when speaking to the automotive assistant, wherein the automotive assistant is configured to interact with the occupant in response to receiving the signal from the gaze detector and detecting speech from the user.
12. The device according to claim 6, wherein: The passenger is one of a plurality of occupants in a vehicle, wherein the plurality of occupants include a first occupant and a second occupant, wherein a first eye region is assigned to the first occupant, wherein a second eye region is assigned to the second occupant, wherein the speech filter is configured to detect a first utterance uttered by the first occupant and a second utterance uttered by the second occupant, and to cause the car assistant to ignore the first utterance and respond to the second utterance, wherein at the time of the first utterance, the line of sight of the first occupant misses the first eye region, and at the time of the second utterance, the line of sight of the second occupant intersects with the second eye region.
13. The device according to claim 6, wherein: The speech filter is configured to ignore the utterance when the occupant's line of sight fails to intersect an eye volume that has been designated for the occupant.
14. The device of claim 6, further comprising a microphone, wherein The speech filter is configured to ignore the utterance when the occupant's line of sight fails to intersect a volume coextensive with the microphone.
15. The device of claim 6, further comprising a display, wherein The speech filter is configured to ignore the utterance when the occupant's line of sight fails to intersect the display.
16. The device according to claim 6, wherein: The speech filter is configured to ignore the utterance when the occupant's line of sight fails to intersect an eye volume that has been designated for the occupant.
17. The apparatus according to claim 6, wherein: The occupant includes eyes, wherein the apparatus further includes a gaze detector comprising a machine vision system that detects a position of features in the occupant's eyes and tracks the position to infer the gaze.
18. The apparatus according to claim 6, wherein: The occupant includes eyes, wherein the device further includes a line of sight detector configured to illuminate the eyes with electromagnetic radiation outside the visible range and observe the position of the reflection as a basis for inferring the occupant's line of sight.
19. The apparatus according to claim 6, wherein: The occupant includes a face and an eye located in a socket on the face, wherein there is a facial vector perpendicular to the face, wherein there is an eye vector perpendicular to the eye, wherein the eye is configured to rotate in the socket so that the facial vector and the eye vector can move independently of each other, wherein the device also includes a line of sight detector, which is configured to determine a line of sight based on the facial vector and the eye vector.
20. The apparatus according to claim 6, wherein: The car assistant is configured to ignore the utterance if there is a high probability that the passenger is not speaking to the car assistant.