Dynamic text-to-speech responses from smart speakers
By integrating audio sensors in smart speakers, analyzing environmental parameters and adjusting audio responses, the problem that existing smart speakers cannot customize responses is solved, improving the adaptability and user experience of responses.
Patent Information
- Application Number
- CN201880034610.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2017-05-26
- Filing Date
- 2018-05-17
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2038-05-17
AI Technical Summary
Existing smart speakers lack situational awareness and cannot customize responses to user queries or commands based on environmental parameters around the user, such as factors such as noise level, user location and identity.
By integrating audio sensors in smart speakers, receiving and analyzing environmental parameters, adjusting the volume, tone and presentation speed of the audio response to better match the user's query or command.
Achieve more appropriate audio responses and improve user experience, such as increasing volume when noise is high or reducing volume at night to adapt to different environments and user needs.
Smart Images

Figure CN110678921B_ABST
Abstract
Description
Technical Field
[0001] Various aspects and embodiments of the present disclosure as a whole relate to customizing a smart speaker's response to a user's command or query based at least in part on one or more parameters of the environment surrounding the smart speaker. Background Art
[0002] A smart speaker that can access a virtual personal assistant (VPA) service is a device that responds to a user query (which can be in the form of a spoken query) by, for example, using an Internet search for a response to the user query, and typically provides the response to the user in the form of an audible response (such as synthesized speech). A smart speaker that can access a VPA service can also respond to user commands to play audio from a specified audio source (e.g., an Internet radio station), or control a smart device, such as to turn a light on or off or change a setting of another smart device, which the smart speaker can access, for example, directly via a Wi-Fi signal or through the user's Internet router. The user typically provides a query or command to the VPA through a smart speaker or other device after the user presses a button or says a wake word or phrase (e.g., "Alexa"), which indicates to the smart speaker or other device that the user is speaking to the VPA. VPA-enabled devices are becoming increasingly common among a variety of companies that offer competing devices, such as the Amazon Alexa and the Amazon Siri. Devices, Google from Google devices, and from Apple Various devices for applications. Current smart speakers do not yet have situational awareness. They lack the ability to detect parameters of their surroundings, such as the location of people, the number of people around the smart speaker, or the ambient noise level. Current smart speakers cannot customize VPA responses to user queries or commands based on environmental parameters. Summary of the invention
[0003] According to one aspect of the present disclosure, a method of operating a situational awareness speaker is provided. The method includes receiving an indication of at least one parameter of an environment proximate to the situational awareness speaker, receiving audio information at the situational awareness speaker from a virtual personal assistant, and modifying the audio information based on the indication of the at least one parameter. Receiving the audio information may include receiving an audio response to a query spoken by a user to the situational awareness speaker. The method may also include presenting the audio response through the situational awareness speaker.
[0004] In some implementations, the method includes modifying the audio information by setting a volume of the response based on an indication of at least one parameter. The at least one parameter may be one or more of a volume of the query, a volume of background noise sensed by an audio sensor of the situational awareness speaker, or an identity of a user.
[0005] Modifying the audio information may include adjusting one or more of a volume, pitch, balance, tone, or presentation speed of the audio information. The at least one parameter may include one or more of: a volume of a query provided to the situational awareness speaker, a volume of background noise, a spectrum of background noise, an identity of a user, a location of a user, a time of day, a physiological parameter of a person proximate to the situational awareness speaker, a response of a user to a previous response provided by the situational awareness speaker, or an activity state of one or more devices proximate to the situational awareness speaker.
[0006] In some implementations, the method includes setting the volume of the response based on the user's location. The method also includes selecting a speaker through which the response is delivered based on the user's location, or setting the volume of the response based on the user's distance from the situational awareness speaker.
[0007] In some embodiments, the method includes setting the volume of the response based on one or more of: the time of day, a physiological parameter of a person within a hearing distance of the situational awareness speaker, a user's response to a previous response provided by the situational awareness speaker, or an activity state of one or more devices proximate to the situational awareness speaker.
[0008] In some embodiments, the method includes setting a pitch of the response based on an indication of at least one parameter. The method may include formatting the response to simulate a whisper, simulate a shout, or remove low frequency components of the response. The pitch of the response may be set based on one or more of: a volume of the vocal query, a volume of background noise sensed by an audio sensor, an identity of the user, a location of the user, a time of day, a physiological parameter of a person within hearing distance of the situational awareness speaker, a frequency spectrum of background noise sensed by an audio sensor, or a response of the user to a previous response provided by the situational awareness speaker.
[0009] In some embodiments, the method includes setting a speed of the simulated voice of the response based on the indication of the at least one parameter. The speed of the simulated voice of the response may be set based on one of a voice speed of the voice query or an identity of the user.
[0010] According to another aspect, a method for dynamically formatting a response of a virtual personal assistant (VPA) service provider to a query of a user is provided. The method includes receiving an indication of a voice query received from a user via an audio sensor of a device that can access the VPA service provider, receiving a response to the voice query, and delivering the response formatted as speech to the user via an audio output of the device, at least one audio parameter of the response being set based on at least one parameter of an environment proximate to the device.
[0011] In some embodiments, the method includes formatting the volume of the response and / or the tone of the response based on at least one parameter. The audio parameters of the response may be formatted based on the identity of the user. The audio parameters of the response may be formatted based on the time of day.
[0012] According to another aspect, a smart speaker is provided. The smart speaker includes a microphone, at least one speaker, and a processor. The processor is configured to recognize a verbal user query received at the microphone, transmit the user query to a virtual personal assistant service provider, receive a response to the user query from the virtual personal assistant service provider, format the response as speech, and present the response to the user through at least one speaker. At least one audio parameter of the response is set based on at least one parameter of an environment proximate to the smart speaker.
[0013] The processor may be configured to set the volume of the response based on one or more of the following: the volume of the spoken user query, the volume of background noise sensed by the microphone, the identity of the user, the location of the user, the time of day, a physiological parameter of the user, or a response of the user to a previous response provided by the smart speaker. The processor may be configured to set the pitch of the response based on one or more of the following: the pitch of the spoken user query, the spectrum of background noise sensed by the microphone, the identity of the user, the location of the user, or the time of day. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] The drawings are not intended to be drawn to scale. In the drawings, each identical or nearly identical component shown in various figures is represented by a like number. For clarity, not every component is labeled in every drawing. In the drawings:
[0015] Figure 1A is a simplified schematic diagram of an example of a smart speaker including functionality to access a VPA service provider;
[0016] Figure 1B is a simplified schematic diagram of another example of a smart speaker including functionality to access a VPA service provider;
[0017] Figure 1CA user is shown providing a query to a smart speaker including functionality to access a VPA service provider, and the smart speaker responding to the user;
[0018] Figure 1D A system including a smart speaker in communication with a VPA service provider is shown;
[0019] Figure 2 shows communication between the smart speaker and various devices through a router;
[0020] Figure 3 shows communication between a smart speaker and a user's device;
[0021] Figure 4 A vehicle equipped with functionality to access a VPA service provider is shown; and
[0022] Figure 5 A headset equipped with functionality to access a VPA service provider is shown. DETAILED DESCRIPTION
[0023] The aspects and implementations disclosed herein are not limited to the details of construction and the arrangement of components set forth in the following description or shown in the drawings. The aspects and implementations disclosed herein can be practiced or implemented in various ways.
[0024] The various aspects and embodiments disclosed herein may be applicable to various smart speakers that can access a virtual personal assistant (VPA) service. The various aspects and embodiments of the smart speakers disclosed herein include the function of giving the smart speakers situational awareness. Based on one or more measured or monitored parameters, the various aspects and embodiments of the smart speakers may customize the response (VPA response) provided by the VPA service to the user query or command to be more appropriate given one or more measured or monitored parameters. Customizing the response to the user query or command may include providing the same content (e.g., the same words) in the same order as provided in the uncustomized response, but wherein one or more audio components (such as volume, pitch, speed, etc.) are modified from the default volume, pitch, speed, etc. Examples of measured or monitored parameters that the smart speaker can use to customize the VPA response may include any one or more of the following: the volume of the user query or command, the tone of the user query or command, the voice speed in the user query or command, the ambient noise level, the time of day, the day of the week, the location of the user providing the query or command, the identity of the user providing the query or command, or the user's response to the response given by the VPA service through the smart speaker.
[0025] FIG. 1 shows a simplified schematic diagram of a smart speaker (generally indicated at 100). The smart speaker 100 includes a body 105. One or more audio sensors (e.g., microphones 110) are mounted on or within the body 105. In some embodiments, providing multiple microphones 110 enables the smart speaker to triangulate the direction of origin of the audio signal (e.g., the user's query or command or the location of the audio source (e.g., the user)). In other embodiments, a single directional microphone 110 may be provided to the smart speaker to determine the direction of origin of the audio signal. One or more microphones 110 (hereinafter, for simplicity, represented in singular form) are electrically connected to a circuit 115 together with other components of the smart speaker 100, which includes at least one processor disposed within the body 105 of the smart speaker. The circuit 115 is shown as a single box in FIG. 1, but may include multiple modules, subcircuits, or processors for performing various functions of the smart speaker 100, such as audio signal processing circuits, communication circuits, memory, etc.
[0026] The circuit 115 may include or be electrically coupled to one or more antennas 120 (in Figure 1A Only one of the antennas is shown in the figure). One or more antennas 120 (hereinafter referred to in the singular for simplicity) can be used by the smart speaker 100 to connect to a VPA service, the Internet, or another source to search for information in response to a user's query. The smart speaker 100 can use the antenna 120 to connect to the Internet via a cellular signal or to connect to the Internet via Wi-Fi through an Internet router.
[0027] The antenna 120 may also be used by the smart speaker 100 to communicate with other devices. In a home or building equipped with smart devices (e.g., smart thermostats, lights, or appliances that can be controlled via Wi-Fi, etc.), the smart speaker 100 may use the antenna 120 to relay commands from the user to the smart device. For example, a user may provide an audio command to the smart speaker 100 to turn on the lights in a room of the house, and the smart speaker 100 may interpret the command and send an appropriate signal to the controller of the lights in the room via Wi-Fi (in some embodiments, through a Wi-Fi router) to turn on the lights. In other embodiments, the smart speaker 100 may use the antenna 120 to locate or communicate with a device, such as a cell phone or other mobile computing device, to collect information or send commands to the device. For example, in some embodiments, the smart speaker 100 may use the antenna 120 to identify the IP address of a cell phone or other computing device near the smart speaker 100, and may identify a user associated with the cell phone or other computing device. The smart speaker 100 may communicate via Wi-Fi, cellular, Bluetooth, or other communication protocols using a single antenna 120, or may include dedicated antennas 120 for different communication protocols.
[0028] The smart speaker 100 also includes a power supply 125. The power supply 125 may receive power from an electrical outlet via a plug 130 and / or may include one or more batteries or other power storage devices.
[0029] At least one audio output (e.g., speaker 135) is included in the smart speaker 100 to output an audio response to a user's query and / or provide a prompt or verification of receipt or completion of a command, or provide information about the condition or setting of the smart speaker 100. In some embodiments, Figure 1B As shown, a plurality of speakers 135 may be provided. The plurality of speakers 135 may be used to control the direction of the audio output to the user.
[0030] A user interface 140 (which may include manually activated controls and, in some embodiments, a display) may be provided on the body of the smart speaker 100 to allow a user to change settings (e.g., power on / off, volume, etc.) or program the smart speaker 100. In other embodiments, settings and / or programming of the smart speaker 100 may be adjusted wirelessly via the antenna 120, for example, using an application on a user's phone or other computing device. Additionally or alternatively, settings and / or programming of the smart speaker 100 may be adjusted using a cable connected to the user interface 140 and coupled to an external device (e.g., a user's phone or other computing device).
[0031] The smart speaker 100 can be implemented in a variety of form factors. In some embodiments, the smart speaker 100 is designed to be placed in a fixed position on a table or countertop. In other embodiments, the smart speaker 100 is included in a vehicle. In other embodiments, the smart speaker 100 is implemented in headphones, or in software in a mobile computing device (e.g., a cell phone). Figure 1A The embodiments of the smart speaker 100 described above may be modified appropriately for a particular embodiment. For example, when included in a vehicle, headset, or mobile phone, the smart speaker 100 may not include a separate power supply 125 or plug 130, and the various components of the smart speaker 100 may be distributed throughout the vehicle, headset, or mobile phone, or implemented in software or hardware modules shared with other systems of the vehicle, headset, or mobile phone.
[0032] Figure 1CAn example of user interaction with a smart speaker 100 is shown. As shown, a user 195 may speak an audio query 100A to the smart speaker 100. The smart speaker 100 receives the audio query 100A through a microphone and performs a speech-to-text conversion of the audio query 100A. Alternatively, the smart speaker 100 records the audio query and sends it as an audio file to another system, such as a cloud-based VPA service provider 1000. The smart speaker 100 requests the VPA service provider 1000 to search for information in a database (e.g., the Internet or the cloud) to respond to the user's query 100A. The smart speaker 100 may access the VPA service provider or other information source via an Internet router 150. The smart speaker 100 retrieves the information required to respond to the user's query 100A from the VPA service provider in text form, performs text-to-speech conversion on the received information, and outputs an audio response 100B to the user 195. Alternatively, text-to-speech conversion of the response may be performed by the VPA service provider, and the response may be sent from the VPA service provider to the smart speaker 100 as an audio file.
[0033] In other embodiments, VPA functions (e.g., sending requests for information to a VPA service provider or other information source, and receiving responses to information requests from a VPA service provider or other information source) may be performed in a device separate from the device that receives or presents responses to user queries or commands. Figure 1D As shown, user 195 may provide a verbal query 100A to smart speaker 101. User 195 may speak a wake-up word to smart speaker 101 before providing verbal query 100A, so that smart speaker 101 will interpret verbal query 100A as a query to which user 195 expects a response. Smart speaker 101 may include a reference Figure 1A and Figure 1B The smart speaker 100 described above has substantially the same components, but may lack the functionality to send queries to a VPA service provider or other data source and receive responses to the queries from the VPA service provider or other data source. In one non-limiting example, the smart speaker 101 is a Bose Corporation available from audio playback device or similar streaming audio player. The smart speaker 101 may optionally relay the verbal query 100A to the smart speaker 100 after recording the verbal query 100A, which has the ability to request and receive a response to the user query 100A from a VPA service provider or other information source as described above (e.g., a service provider or other information source in the cloud 1000). The smart speaker 100 may receive a response to the user query from the VPA service provider or other information source and transmit the response to the smart speaker 101 for presentation. After applying appropriate signal conditioning to the response to change one or more audio parameters of the response based on one or more environmental variables of the environment surrounding the smart speaker 101, the smart speaker 101 may present the response to the user 195 as an audio response 100B.
[0034] The communication between the smart speaker 101, the smart speaker 100 and the VPA service provider 1000 can be done through Figure 1D The router 150 shown may alternatively include direct communication (wired or wireless) between the smart speaker 101 and the smart speaker 100.
[0035] It should be understood that references herein to the smart speaker 100 include systems in which a single component receives spoken user queries and provides audio responses to the user, as well as requesting and receiving responses to queries from external sources, as well as systems in which a single component receives spoken user queries and provides audio responses to the user. Figure 1D A system is shown in which a first device receives a user query and presents a response to the user, and a second device requests and receives a response to the user query and transmits the response to the first device for presentation.
[0036] As described above, the smart speaker 100 may have situational awareness and may customize responses to user queries or commands based on one or more measured or monitored parameters. One parameter may be the volume of the user's voice issuing a verbal query or command to the smart speaker 100. In some embodiments, the smart speaker 100 may include a volume control that the user can set to a default level. In some cases, the smart speaker 100 may respond to queries or commands at a volume different from the set volume or default volume. For example, if the user gets up early and other members of the user's family are still sleeping, the user can verbally query the smart speaker 100 at a low volume to obtain information such as the weather forecast for the day. The smart speaker 100 may receive the query through the microphone 110 and determine whether the volume of the user's voice is higher or lower than the typical value when interacting with the smart speaker 100 or higher or lower than a predetermined threshold. If the query is provided by the user speaking softly, the smart speaker 100 may provide an audible response to the query at a volume lower than the default volume or set volume. In another example, the smart speaker 100 may provide a louder response than usual, for example, to overcome background noise, or when the user is far away from the smart speaker 100. The user can provide a query or command to the smart speaker 100 at an increased volume, and the smart speaker 100 will respond to the query or command at a corresponding volume higher than the default volume or set volume. In some embodiments, the smart speaker 100 may have a set number of volume adjustment levels, for example, 25%, 50%, 100%, 125%, 150%, and 200% of the default volume or set volume. In other embodiments, the smart speaker 100 may respond to the user's query at a volume within a continuous volume range based on the volume of the user's query.
[0037] In other embodiments, the smart speaker 100 may respond to a query or command with a tone based on the volume or tone of the command or query provided by the user to the smart speaker 100. For example, if the user provides a command or query to the smart speaker 100 at a low volume or in a low voice, the smart speaker may provide an audible response to the command or query that sounds like a whisper. Conversely, if the user shouts a command or query to the smart speaker 100, the smart speaker 100 may respond with an audible response that sounds like a shout. In some embodiments, if the user provides a query or command in a specific tone, for example, if a child provides a query or command in a typical tone of a child's voice, the smart speaker 100 may respond with a response that sounds like a child's voice. Similarly, based on whether the user query or command is provided in a female voice or a male voice, the tone of the response provided by the smart speaker 100 can be provided as a female voice or a male voice. The signal processing circuit in the circuit 115 of the smart speaker 100 may apply audio filtering / signal processing to the response so as to generate a response with an alternative tone.
[0038] The smart speaker 100 may additionally or alternatively modify the pitch of a response to a command or query from a default pitch by applying a bandpass filter to the response. Low-frequency sounds tend to propagate farther through a building than high-frequency sounds. Thus, if a user provides a command or query to the smart speaker 100 at a low volume, or if the smart speaker 100 receives some other indication that it should provide an audio response that will not propagate far, the smart speaker 100 may apply a high-pass filter to the audio response, thereby removing low-frequency wavelengths from the response.
[0039] In some embodiments, the smart speaker 100 may include functionality to determine the speed of a user's voice providing a command or query to the smart speaker 100, and may adjust the speed of reproduction of an audio response to the command or query based on the user's voice speed. If the user provides a command or query by speaking slowly, the smart speaker 100 may provide an audio response to the command or query slowly. Such functionality may be useful if the user is not proficient in English or any other language that the smart speaker 100 is programmed to respond to, and the user wishes to receive a response from the smart speaker slower than the default response speed or set response speed to help the user understand the response. Conversely, if the user provides a command or query by speaking quickly, the smart speaker 100 may provide an audio response to the command or query in a manner that replicates fast speech. Such functionality may be useful if the user is in a hurry, and the user wishes to receive a response from the smart speaker faster than the default response speed or set response speed.
[0040] In other embodiments, when a wake-up phrase is received and before or during receiving a command or query from a user, the smart speaker 100 may monitor or measure the volume level and / or spectrum of the background or ambient noise. When formatting an audio response to a user command or query, the smart speaker 100 may take into account the volume level and / or spectrum of the background or ambient noise. If a large or high volume of background noise is generated by multiple people talking, or a TV or radio near the smart speaker 100 generates a large or high volume of background noise, or because the smart speaker 100 is in an outdoor environment, the smart speaker 100 may output a response at a volume higher than the default volume or set volume. In some embodiments, the smart speaker 100 may have a set number of volume adjustment levels, for example, 25%, 50%, 100%, 125%, 150% and 200% of the default volume or set volume. In other embodiments, the smart speaker 100 may respond to the user's query at a certain volume within a continuous volume range based on the volume of the background or ambient noise. Additionally or alternatively, the smart speaker 100 may analyze the spectrum of the background or ambient noise. The smart speaker 100 may modify the volume of only certain frequencies of the audio response, for example, frequencies where background or ambient noise is louder than other frequencies. In other embodiments, the smart speaker 100 may increase the volume of the response at frequencies where the background or ambient noise is softer or less loud than at other frequencies, so that the user can more easily distinguish the audio response from the background or ambient noise.
[0041] The smart speaker 100 may dynamically adjust the volume and / or pitch of a response to a command or query based on factors other than the parameters of the user's voice providing the command or query to the smart speaker 100. In some embodiments, the smart speaker 100 may include a clock (e.g., within circuit 115 of FIG. 1 ) or receive an indication of the time, and may be programmed to provide an audio response to a command or query at a reduced volume that is lower than a default or set volume during nighttime periods when it is expected that family members may be sleeping. The reduced volume may be a volume set by a user of the smart speaker 100. Additionally or alternatively, the smart speaker 100 may be programmed to provide an audio response to a command or query at a tone similar to a whisper or with a lower frequency of the audio response suppressed (e.g., with a high pass filter applied to the audio response) during nighttime periods or other periods set by the user.
[0042] In addition, the smart speaker 100 may query the Wi-Fi router 150 (see Figure 2) to communicate with, such as lights 160 or entertainment system 170 (e.g., other smart speakers, televisions, radios, etc.), or even a physiological monitor 180 of a household resident. If the lights 160 or entertainment system 170 are active and / or if the physiological monitor 180 indicates that the household resident is awake, this can provide an indication that someone in the household is awake and a reduced volume response to a query or command may not be warranted, and the smart speaker 100 can respond to the query or command at a default volume or set volume. If the lights 160 or entertainment system are inactive and / or if the physiological monitor indicates that the household resident is sleeping, this can provide an indication that someone in the household is sleeping and a reduced volume response to a query or command may be warranted. In other embodiments, the user's physiological monitor 180 can provide information about the user's heart rate or breathing rate. An increase in the user's heart rate or breathing rate can indicate that the user is in an excited state or is in a state of exercise or has just finished exercise. In response to the smart speaker 100 receiving an indication of an elevated heart rate or breathing rate of a user from the user's physiological monitor 180 , the smart speaker 100 may provide an audio response to the user's query or command at a volume elevated above a default or set volume.
[0043] The smart speaker 100 may also include the ability to communicate with one or more other devices 190 of the user 195 (e.g., a mobile phone, a smart watch, etc.). (See Figure 3 . ) The smart speaker 100 may query the device 190 for information from which the identity of the user can be derived. For example, the smart speaker 100 may request the IP address of the device 190 and attempt to match the IP address of the device 190 with a user in a lookup list in the memory of the circuit 115 of the smart speaker 100 or in the cloud 1000 ( Figure 1B ). Once the VPA determines the identity of the user 195, it can customize the response to the query or command from the user in a manner that is appropriate for the user. For example, if the smart speaker 100 determines that the user is a middle-aged or elderly person who may have some hearing loss, the smart speaker 100 can respond to the query or command from the user with an audio response provided at a volume that is increased relative to the default volume or set volume. In some embodiments, different users can program the smart speaker 100 with different preferences for responses to queries or commands, such as the pitch of the audio response, whether the audio response should be a female voice or a male voice, the volume of the audio response, the speed of the audio response, etc.
[0044] In some embodiments, the smart speaker 100 may listen for an indication that the user does not understand an audio response provided by the smart speaker 100 to a command or query. If such an indication is detected, the smart speaker 100 may repeat the audio response at a lower "speaking" rate and / or a higher volume. For example, if the smart speaker 100 provides an audio response to a user's command or query at a first speaking rate and a first volume, if the user responds with a vocal phrase indicating that the user does not understand the response, such as by saying "what?", "what's that?", "hmm?", "I don't understand", "please repeat", etc., the smart speaker 100 may repeat the response at a lower speaking rate and / or a higher volume.
[0045] A smart speaker 100 equipped with multiple microphones 110 or equipped with a directional microphone can detect the direction of origin of a user command or query and / or the location of the user providing the command or query. The smart speaker 100 can use the direction or location information to direct the response to the user who gave the command or query. For example, in a system including more than one speaker 135 (see, e.g. Figure 1B ), the relative volumes of the different speakers may be modulated to direct most of the acoustic energy in the response in the direction of the user providing the command or query. In some embodiments, the smart speaker 100 may be in a master-slave relationship with other smart speakers 100, or may have the ability to control a separate smart speaker or audio device. In such embodiments, a smart speaker 100 that receives a query result from a user may direct a smart speaker 100 or a separate smart speaker or audio device that is closer to the user than the smart speaker 100 that receives the query result to provide an audible response to the query to the user.
[0046] In one particular embodiment, a smart speaker 100 with functionality to access VPA services may be implemented in the electronics of a vehicle, such as a car or SUV. The smart speaker 100 may include dedicated circuitry 115, or may be implemented as software in a computer of the vehicle 200. As disclosed in international applications PCT / US2017 / 021521 and PCT / US2017 / 021625, assigned to the same assignee as the present application, the vehicle 200 may include a shared speaker 205 and a dedicated speaker 210 for each passenger in the vehicle. The dedicated speaker 210 may be built into a headrest or other portion of a seat of the vehicle 200. (See Figure 4. ) The vehicle 200 may include one or more microphones 215 in a portion of the vehicle 200, such as in the dashboard or in the headrest of a seat of the vehicle. The microphone 215 may be used to receive queries or commands to a smart speaker implemented in the vehicle's electronics, and may be used by the smart speaker to determine which user / seat the query or command originated from. The smart speaker may provide an audio response to the query or command to a dedicated speaker associated with the user / seat from which the query or command originated. In some embodiments, dedicated speakers associated with users / seats other than the user / seat from which the query or command originated may utilize crosstalk cancellation or noise cancellation techniques to at least partially reduce the perceived volume of audio responses at users / seats other than the user / seat from which the query or command originated.
[0047] In some embodiments, a smart speaker with the ability to access VPA services may be included in the headset. Figure 5 As shown, headphones, generally indicated at 300, may include a pair of ear cups 310 coupled to each other via a headband 320. Each of the ear cups 210 includes at least one speaker (not shown for clarity) to deliver audio (e.g., music) to the user's ears. VPA functionality may be provided in the headphones 300 by incorporating a VPA access circuit 115 and one or more microphones 110 into the headphones 300, such as into one or more of the ear cups 310. The VPA access circuit 115 may include one or more antennas 120 to enable communication with the Internet or other devices, as described above with reference to various embodiments of the smart speaker 100. In some embodiments, the headphones 300 are wireless headphones, and the antenna 120 may be shared between the VPA access circuit 115 and circuitry of the headphones 300 for providing wireless connectivity. In some embodiments, the headphones 300 may also include a location determination system, such as a GPS receiver 330 and associated circuitry and / or a physiological monitor 180 (e.g., a heart rate monitor). It should be understood that Figure 5 The various components shown in the figure (e.g., VPA access circuit 115, antenna 120, GPS receiver 330, microphone 110, or physiological monitor 180) may be located in different locations within the headset 300, such as at least partially located in the headband 320, or may be at least partially included in a device (e.g., a portable music player or a mobile phone) to which the headset 300 is wirelessly connected. For clarity, Figure 5 The electrical connections between components are omitted.
[0048] In use, a user may provide a voice query or command to the headset 300 to which the VPA function provides a response. The response may be provided through a speaker in the ear cup 310, optionally while lowering or muting the volume of any music or other audio content that the user is listening to through the headset 300. The voice volume or speed of the response may be modified based on the voice volume or speed of the user's query or command, as discussed above with reference to various embodiments of the smart speaker 100. The headset may also change the volume or speed of the audio response to the user's query or command based on other factors (such as the user's heart rate, the user's identity, etc.), as discussed above with reference to various embodiments of the smart speaker 100. The headset 300 may be able to use a GPS receiver 330 to determine the location of the user wearing the headset 300, and may adjust the volume of the audio response to the user's query or command based on the user's location, such as providing a louder response if the user is in an urban environment, and a softer response if the user is in a suburban area.
[0049] In any of the above embodiments, the user can cause the smart speaker 100 to respond in a manner that takes into account any of the above parameters by providing a modified alternative wake-up phrase to the smart speaker 100 before providing a command or query. For example, if the user wants the smart speaker to adjust the volume of the response to take into account the volume of the user's voice, the user can say the wake-up phrase "Volume, Siri" instead of "Hey, Siri". Alternatively, the smart speaker 100 may include a switch or may be responsive to a verbal command to enable or disable modification or formatting of the response based on one or more monitored or measured parameters.
[0050] Hypothetical Example 1 :
[0051] A smart speaker located in the kitchen of a user's home has a volume selector set to a first level. The user wakes up in the middle of the night while other family members are sleeping and goes to the kitchen for a midnight snack. The user whispers a wake-up word to the smart speaker and asks the smart speaker what the weather forecast for the next day is. The smart speaker compares the volume of the user's request for the weather forecast with a preset volume threshold and determines that the user is speaking at a reduced volume compared to the preset volume threshold. The smart speaker sends the user request to the VPA-enabled device over the Wi-Fi network in the user's home. The VPA queries the VPA service provider in the cloud for the next day's weather report and receives the weather report from the VPA service provider. The VPA transmits the weather report to the smart speaker. The smart speaker presents the weather report to the user at a lower volume than the volume at which the smart speaker would present the audio based on the set volume level.
[0052] Hypothetical Example 2 :
[0053] The middle-aged and elderly user says the wake-up word and requests information about the preparation of the prescription from the smart speaker. The smart speaker detects that the mobile phone has entered the vicinity of the smart speaker. The smart speaker queries the mobile phone for identification information, such as an IP address, and receives identification information from the mobile phone. The smart speaker transmits the information request and identification information from the mobile phone to the VPA-enabled device through the Wi-Fi network in the user's home. The VPA-enabled device queries the VPA service provider in the cloud for information about the preparation of the prescription and receives a response that the prescription is ready for extraction. The VPA service provider also accesses the account associated with the VPA and searches for information that associates the identification information from the mobile phone with a specific user, and determines that the mobile phone belongs to the middle-aged and elderly user. The VPA transmits information about the preparation of the prescription and the identity of the user to the smart speaker. The smart speaker presents information about the preparation of the prescription to the user at a preset volume associated with the middle-aged and elderly user in the memory of the smart speaker, and a preset speed associated with the middle-aged and elderly user in the memory of the smart speaker, the preset volume is increased compared to the default volume level, and the preset speed is reduced compared to the default response voice speed.
[0054] Having thus described several aspects of at least one embodiment, it will be appreciated that various changes, modifications and improvements will readily occur to those skilled in the art. Such changes, modifications and improvements are intended to be a part of the present disclosure and are intended to fall within the spirit and scope of the present disclosure. The operations of the methods disclosed herein may be performed in a different order than that shown, and one or more operations may be omitted, replaced or added. One or more features of any one of the examples disclosed herein may be combined with or replace one or more features of any other example disclosed. Therefore, the foregoing description and drawings are provided by way of example only.
[0055] The wording and terminology used herein are for the purpose of description and should not be considered as limiting. As used herein, the term "multiple" refers to two or more items or parts. As used herein, the size described as "substantially similar" should be considered to be within about 25% of each other. Whether in a written description or in a claim, the terms "include", "comprising", "with", "having", "including" and "involving" are open terms, that is, "including but not limited to". Therefore, the use of such terms is intended to cover the items listed thereafter and their equivalents, as well as other items. For claims, only the connecting phrases "consisting of..." and "consisting essentially of..." are closed or semi-closed connecting phrases, respectively. Using ordinal terms such as "first", "second", "third" in a claim to modify a claim element itself does not mean that one claim element exceeds any priority, priority or order of another claim element, or the time sequence of performing a method action, but is only used to distinguish a claim element with a certain name from another element with the same name (but used to use ordinal terms) to distinguish the label of the claim element.
Claims
1. A method of operating a situational awareness speaker, the method comprising: receiving an indication of a parameter of an environment proximate to the situational awareness speaker; receiving audio information at the situational awareness speaker from a virtual personal assistant, wherein receiving the audio information comprises receiving an audio response to a query spoken by a user into the situational awareness speaker; The method includes modifying the audio information by setting a volume of the response based on the indication of the parameters, the parameters including: a volume of background noise, a location of the user, and a volume of the query spoken into the situational awareness speaker, wherein setting the volume of the response includes setting a volume of a particular frequency of the response, and the response to the query being at a volume adjustment level among a plurality of volume adjustment levels, the plurality of volume adjustment levels being determined based on a default volume of a volume control of the situational awareness speaker set by the user or a set volume of the situational awareness speaker, The volume adjustment level is different from the set volume or the default volume. 2 . The method of claim 1 , wherein modifying the audio information comprises adjusting one or more of a pitch, a balance, or a presentation speed of the audio information.
3. The method of claim 1 , wherein the parameters further include one or more of: a spectrum of background noise, an identity of the user, a time of day, a physiological parameter of a person proximate to the situational awareness speaker, a response of the user to a previous response provided by the situational awareness speaker, or an activity state of one or more devices proximate to the situational awareness speaker. 4 . The method of claim 1 , further comprising presenting the audio response through the situational awareness speaker.
5. The method of claim 1 , wherein setting the volume of the response comprises: The volume of the response is set based on a volume of background noise sensed by an audio sensor of the situational awareness speaker.
6. The method of claim 1 , wherein setting the volume of the response comprises: The volume of the response is set based on the identity of the user of the situational awareness speaker.
7. The method of claim 1 , wherein setting the volume of the response comprises: The volume of the response is set based on the location of the user of the situational awareness speaker.
8. The method of claim 7, further comprising selecting a speaker through which to deliver the response based on the location of the user.
9. The method of claim 1, wherein setting the volume of the response comprises setting the volume of the response based on a time of day.
10. The method of claim 1, wherein setting the volume of the response comprises setting the volume of the response based on a physiological parameter of a person within a hearing distance of the situational awareness speaker.
11. The method of claim 1 , wherein setting the volume of the response comprises setting the volume of the response based on a response of the user to a previous response provided by the situational awareness speaker. 12 . The method of claim 1 , wherein setting the volume of the response comprises setting the volume of the response based on an activity state of one or more devices proximate to the situational awareness speaker.
13. The method of claim 1, comprising setting a pitch of the response based on the indication of the parameter.
14. The method of claim 13, comprising formatting the response to one of simulated whispering, simulated shouting, or removing low frequency components of the response.
15. The method of claim 13, comprising setting the pitch of the response based on one or more of: the volume of the vocal query, the volume of background noise sensed by the audio sensor, the identity of the user, the location of the user, the time of day, physiological parameters of people within hearing distance of the situational awareness speaker, the frequency spectrum of background noise sensed by the audio sensor, or the user's response to a previous response provided by the situational awareness speaker.
16. The method of claim 1, comprising setting a speed of the simulated voice of the response based on one of a speech speed of the vocal query or an identity of the user.
17. A method for dynamically formatting a response of a virtual personal assistant (VPA) service provider to a user's query, the method comprising: receiving, via an audio sensor of a device accessible to the VPA service provider, an indication of a voice query received from a user; receiving a response to the voice query; as well as delivering the response formatted as speech to the user via an audio output of the device, at least one audio parameter of the response being set based on a parameter of an environment proximate to the device, wherein a volume of the response is set based on an indication of the parameter, the parameter including a volume of background noise, a location of the user, and a volume of the vocal query, wherein setting the volume of the response includes setting a volume of a particular frequency of the response, and the response to the query being at a volume adjustment level among a plurality of volume adjustment levels, the plurality of volume adjustment levels being determined based on a default volume of a volume control of the device set by the user or a set volume of the device, The volume adjustment level is different from the set volume or the default volume.
18. The method of claim 17, wherein setting the volume of the response comprises setting the volume of the response based on one or more of: time of day, a physiological parameter of the user, a response of the user to a previous response provided by the device, or an activity state of one or more devices proximate to the device.
19. The method of claim 17, comprising formatting a tone of the response based on the parameter.
20. The method of claim 17, comprising formatting audio parameters of the response based on the identity of the user.
21. The method of claim 19, comprising formatting audio parameters of the response based on time of day.
22. A smart speaker, comprising: microphone; at least one speaker; and A processor, the processor being configured to: recognizing a spoken user query received at the microphone; transmitting the user query to a virtual personal assistant service provider; receiving a response to the user query from the virtual personal assistant service provider; formatting the response as speech, at least one audio parameter of the response being set based on a parameter of an environment proximate to the smart speaker; Setting a volume of the response based on a volume of background noise, a location of the user, and a volume of the user query, wherein setting the volume of the response includes setting a volume of a specific frequency of the response; wherein the response to the user query is at a volume adjustment level among a plurality of volume adjustment levels, the plurality of volume adjustment levels being determined based on a default volume of a volume control of the smart speaker set by the user or a set volume of the smart speaker, the volume adjustment level being different from the set volume or the default volume, and The response is presented to the user via the at least one speaker.
23. The smart speaker of claim 22, wherein the processor is further configured to set the volume of the response based on one or more of: time of day, a physiological parameter of the user, a response of the user to a previous response provided by the smart speaker, or an activity state of one or more devices proximate to the smart speaker.
24. The smart speaker of claim 22, wherein the processor is configured to set the pitch of the response based on one or more of: the pitch of the verbal user query, the spectrum of background noise sensed by the microphone, the identity of the user, the location of the user, or the time of day.
