Drop-In functionality on computing devices based on event detection

Abstract

Techniques are disclosed for monitoring an audio stream for a trigger event, wherein the trigger event may include a particular sound identified in the audio stream. Further, the technique includes detecting the particular sound in the audio stream and / or providing an event notification to a user device via a network connection. A user device may request permissions to initiate bi-directional audio streaming between the computing device and the user device. Further, the technique may include receiving an indication to initiate the bi-directional audio stream with the user device. Further, the technique may include providing an alert that has been granted permission in order to initiate the bi-directional audio stream. In addition, the device may include initiating the bi-directional audio stream with the user device.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is an international application and claims the benefit of and priority under 35 U.S.C. 119(e) to U.S. Provisional Application No. 63 / 365,842, filed on June 3, 2022, entitled “Drop-In On Computing Devices Based On Event Detections,” and U.S. Non-Provisional Application No. 18 / 138,652, filed on April 24, 2023, entitled “Drop-In On Computing Devices Based On Event Detections,” the contents of each of which are incorporated herein by reference in their entirety for all purposes. Technical Field

[0003] The present disclosure relates to audio communication. In particular, the present disclosure relates to two-way audio communication technology. Background Art

[0004] Two-way audio communication between electronic devices can allow for improved crisis detection, notification, and communication. Electronic devices can be used to monitor other devices to detect triggering events. Electronic devices can also provide notifications or communications to other electronic devices. Therefore, it is desirable to improve two-way audio communication technology. Summary of the invention

[0005] A system consisting of one or more computers may be configured to perform a specific operation or action by virtue of having software, firmware, hardware, or a combination thereof installed on the system, which in operation causes the system to perform the action. One or more computer programs may be configured to perform a specific operation or action by virtue of including instructions that, when executed by a data processing device, cause the device to perform the action.

[0006] In a general aspect, a computer-implemented method may include monitoring a one-way audio stream for one or more triggering events. The one or more triggering events may include a specific sound identified in the one-way audio stream. In addition, the method may include detecting the specific sound in the one-way audio stream. The method may also include providing an event notification to a user device via a network connection. The event notification may request permission to start a two-way audio stream between the computing device and the user device. The method may also include receiving an indication to start the two-way audio stream with the user device. The indication may be based at least in part on the event notification. In addition, the method may include providing a first alarm via a speaker of the computing device. The alarm may indicate that permission has been granted to start the two-way audio stream. The method may also include starting the two-way audio stream with the user device. Other embodiments of this aspect include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each of which is configured to perform the actions of the method.

[0007] Specific implementations may include one or more of the following features. A technique for providing an event notification includes determining a distance between the computing device and the source of a detected specific sound. The technique may include sending the determined distance to multiple monitoring devices associated with the computing device. In addition, the technique may include receiving one or more determined distances from a subset of monitoring devices that detected the specific sound. The technique may include comparing the determined distance with the one or more determined distances to determine whether the computing device is closer to the source than the subset of monitoring devices. The technique may include providing an event notification to the user device when it is determined that the computing device is closer than the subset of monitoring devices.

[0008] Particular implementations may include providing a notification to the second user device.

[0009] Specific implementations may include a technology in which the two-way audio stream is a multi-directional audio stream. The multi-directional audio stream may be located between the computing device, the user device, and the second user device.

[0010] Specific implementations may include providing event notification and also include starting an event timer. The two-way audio stream may be started during the event timer.

[0011] Particular implementations may include techniques for the event notification to identify the particular sound recognized in the audio stream.

[0012] Specific implementations may include techniques for initiating the two-way audio stream including providing a second alarm at regular intervals during the duration of the two-way audio stream. The second alarm may announce that the two-way audio stream has been initiated. Specific implementations of the described techniques may include hardware, techniques or processes, or computer tangible media.

[0013] In a general aspect, a technique implemented by a non-transitory computer-readable medium may include monitoring a one-way audio stream for one or more triggering events. The one or more triggering events may include a specific sound identified in the one-way audio stream. The technique may include detecting the specific sound in the audio stream. The technique may also include providing an event notification to a user device via a network connection. The event notification may request permission to start a two-way audio stream between the computing device and the user device. The technique may also include receiving an indication to start the two-way audio stream with the user device based at least in part on the event notification. In addition, the technique may include providing a first alarm that permission has been granted to start the two-way audio stream via a speaker of the computing device. The technique may also include starting the two-way audio stream with the user device. Other embodiments of this aspect include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each of which is configured to perform the actions of the method.

[0014] In a general aspect, a computing device may include a storage device; a speaker; and one or more processors configured to execute program instructions stored in the storage device to monitor a one-way audio stream for one or more triggering events. The one or more triggering events may include a specific sound identified in the one-way audio stream. The instruction may cause the one or more processors to detect the specific sound in the audio stream. The instruction may cause the one or more processors to provide an event notification to a user device via a network connection. The event notification requests permission to start a two-way audio stream between the computing device and the user device. The instruction may cause the one or more processors to receive an indication based at least in part on the event notification to start the two-way audio stream with the user device. The instruction may cause the one or more processors to provide a first alarm that permission has been granted to start the two-way audio stream via the speaker. The instruction may cause the one or more processors to start the two-way audio stream with the user device. Other embodiments of this aspect include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each of which is configured to perform the actions of the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 Diagrams and methods for initiating an audio stream in response to a detected event according to an embodiment are shown.

[0016] Figure 2 A diagram and method for initiating multi-directional audio streaming according to an embodiment are shown.

[0017] Figure 3 A method for selecting a computing device in a monitored environment according to an embodiment is shown.

[0018] Figure 4 is a simplified block diagram illustrating an example architecture of a system for detecting triggering events and taking actions based on the triggering events according to some embodiments.

[0019] Figure 5 A method 500 for initiating a call with a smart speaker according to an embodiment is shown. DETAILED DESCRIPTION

[0020] In the following description, various examples will be described. For the purpose of explanation, many specific configurations and details are set forth in order to provide a thorough understanding of the examples. However, it will also be apparent to those skilled in the art that the examples may be implemented without these specific details. In addition, well-known features may be omitted or simplified to avoid confusing the described examples.

[0021] The various embodiments of the present disclosure may provide a technique for initiating a call using a computing device in response to a detected event according to an embodiment. Example computing devices for performing the techniques described herein include smart speakers, smart media players, tablet computers, or user devices such as smart phones. In some examples, a smart speaker or smart media player may be a computing device having at least a microphone and a speaker. Unlike many computing devices, a smart speaker or smart media player may not necessarily have a display device. Therefore, input may be provided to a smart speaker or media player as a verbal natural language command, and output may be delivered from a speaker as a simulated voice (or a recording of an actual voice). In order to protect privacy, input to a smart speaker may be provided as a stylized interaction, which includes one or more wake-up words / phrases and a command or query to follow. For example, the wake-up word "Hello, device." may be followed by a query "Will it rain today?". Although many of the following examples are provided in the context of a computing device being a smart speaker or media player without a display, other types of computing devices may also perform the described techniques. For example, a computing device may be a smart media streaming device (e.g., connected to a TV), a tablet device, or a smart phone, etc.

[0022] During the monitoring phase, the computing device may listen for a wake word while ignoring other audio input. After the computing device detects the wake word, the device may enter the command phase, in which the speaker may monitor for verbal user commands. For example, the computing device may query a search engine in response to a user command, use a command (e.g., "Hello, device, please turn off the kitchen lights.") to control a smart device associated with the computing device, or communicate with a user device. The computing device may generate an auditory or visual response when the wake word is detected to notify the user that the command phase has begun and that their voice will be monitored. This two-stage monitoring and command configuration protects user privacy because the user consciously uses the wake word to initiate interaction with the smart speaker.

[0023] In addition to receiving verbal commands, the computing device may also be configured to perform event monitoring. Rather than just listening for a wake word, the computing device may also monitor for specific sounds associated with an event. For example, the computing device may listen for various warnings or alarms. The warnings or alarms may correspond to smoke detector warnings, i.e., carbon monoxide warnings, security warnings (e.g., motion detected or window broken). Upon detecting an event, the computing device may transmit a notification to one or more user devices associated with the computing device, thereby notifying one or more users that the event has been detected.

[0024] A triggering event may be a specific sound associated with an event. A triggering event may be a warning or alarm generated by an electronic device, as well as, for example, a warning from a smoke detector, a safety warning, a carbon monoxide warning, a flood detection warning, etc. A triggering event is not necessarily a sound generated by an electronic device, and other sounds may also be a triggering event. For example, a triggering event may include sounds caused by physical damage to a structure such as breaking materials (e.g., shattered glass), sounds of burning materials (e.g., burning wood), and sounds of water, such as those caused by a broken pipe or a fire sprinkler. Sounds caused by people or animals may be triggering events, and triggering events may include sounds of people falling, sounds of animals or people in distress, animal noises (e.g., barking dogs), etc.

[0025] In response to a triggering event, a user may initiate a call between a user device and a computing device. The computing device may broadcast a notification that a call has been initiated before the user device connects to the computing device. In addition, the computing device may provide periodic audio or visual notifications for the duration of the call. For example, a light on top of the computing device may be illuminated and a tone may sound at 30 second intervals. Periodic notifications during a call may mitigate the risk of someone near the computing device participating in the call without their consent.

[0026] In an illustrative example, a homeowner's smart speaker may be located in their kitchen. The smart speaker is configured to listen for several specific sounds associated with triggering events, including smoke detector warnings associated with fires. While at work, the homeowner receives a phone notification that the smart speaker has detected a kitchen smoke warning. The homeowner is unable to contact his partner, who is working from home, by phone, and the homeowner decides to contact his partner via the smart speaker. The homeowner initiates the call via the smart speaker and learns from his partner that the warning was caused by burnt toast and that emergency services are not needed.

[0027] I. Event Monitoring

[0028] Microphones on one or more computing devices (such as smart speakers) can be used to monitor specific sounds associated with the event. One or more computing devices can provide a notification to the user device in response to the detected triggering event. The user can initiate an audio stream (e.g., a call) between the user device and the computing device in response to the notification. If there are multiple computing devices, the call can be initiated using the device closest to the event. After the call is initiated, additional users can join the audio stream in response to the notification.

[0029] A. Two-way audio streaming

[0030] Figure 1 A diagram 101 and method 100 for starting an audio stream in response to a detected event according to an embodiment are shown. The method is illustrated as a logic flow diagram, each operation of which can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, an operation may represent computer-executable instructions stored on one or more computer-readable storage media, which perform the operation when executed by one or more processors. Generally speaking, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the described process or method.

[0031] Turning to method 100 in more detail, at box 102, an audio stream is monitored for a triggering event. A microphone on a computing device 110 may monitor an audio stream for one or more specific sounds associated with an event (e.g., a triggering event). The computing device 110 may be located in a monitored area, and the electronic device may be any computing device that can receive audio input. The monitored area may be an enclosed area, such as a room, a house, a store, etc. An event may be any event that can be detected by sound. For example, an event may include a fire, water damage, smoke damage, a burglary, a person at the door, pet damage, a person in distress, etc.

[0032] At block 104, a triggering event may be detected. Computing device 110 may detect a sound in the monitored audio stream, and the detected sound may be a triggering event. Triggering events may include burning material, an alarm, breaking material (e.g., glass, wood, fabric, etc.), running water, knocking, a doorbell, a collision, an animal noise, furniture movement, etc. For example, computing device 110 may hear an alarm, which is a triggering event 120 generated by a smoke detector.

[0033] At block 106, computing device 110 may provide a notification to user device 140. The notification may be provided to user device 140 via network 150. The notification may be delivered as a text message, a push notification, an email, a pre-recorded phone call, a pre-recorded Voice over Internet Protocol (VOIP) call, etc. The user device may be a computing device such as a smart phone, a smart watch, a tablet computer, a personal computer, etc. The user device 140 may be enabled to communicate over a network 408 (e.g., including a LAN or WAN) using one or more network protocols (e.g., a Bluetooth connection, a Thread connection, a Zigbee connection, a WiFi connection, etc.) and a network path.

[0034] The notification may include an audio clip containing the triggering event. The recording of the triggering event may allow the user to better understand and respond to the event. If the computing device or an electronic device communicatively connected to or associated with the electronic device includes a camera, the notification may include a video clip recorded during the triggering event. The notification may include one or more pictures captured by the camera. The notification may present options that the user can select using user device 140. For example, the notification may prompt the user to call emergency services (e.g., 911). The notification may prompt the user to call, text, email, or otherwise notify an emergency contact associated with user device 140 or computing device 110.

[0035] In some cases, the computing device 110 may automatically call, text, email, or otherwise notify emergency services or emergency contacts. Certain triggering events may cause the electronic device 110 or the user device 140 to automatically contact emergency services or emergency contacts. For example, if a fire or a break is detected, emergency services may be automatically contacted, but if an animal is detected gnawing furniture, emergency services may not be automatically contacted. Depending on the severity of the triggering event, emergency services or emergency contacts may be automatically contacted. For example, if a smoke warning triggering event is detected, emergency services or emergency contacts may not be automatically contacted, but if a smoke warning, burning material, and fire sprinkler triggering event is detected, emergency services and emergency contacts may be automatically contacted. In some cases, information from a camera that is communicatively connected to or associated with the computing device 110 may determine whether to automatically contact emergency services or emergency contacts. For example, if a camera detects a sufficient amount of smoke, visible flames, or intruders in the monitored area, emergency services or emergency contacts may be automatically contacted.

[0036] At box 108, an audio stream may be started. The audio stream may be started by user device 140. The audio stream may be sent between user device 140 and computing device 110 via network 150. The voice recorded at user device 140 may be broadcast as audio 160 by a speaker on computing device 110 or controlled by the computing device. In some cases, computing device 110 and user device 140 may communicate using typed messages. Typed messages (e.g., email short message service (SMS) messages, etc.) may be transmitted between user device 140 and computing device 110. Typed messages received by computing device 110 may be converted into audio 160 using natural language processing techniques. Computing device 110 may transcribe the spoken message and send the text to user device 140.

[0037] The audio 160 broadcast by the computing device 110 may include an alert notifying anyone in the monitored area that an audio stream has been initiated. The alert may include an alert when the audio stream begins, an alert at regular intervals during the audio stream, or an alert after the audio stream ends. The computing device 110 may also provide one or more visual alerts before, during, or after the call. For example, a flashing light on the computing device 110 may serve as an alert that a call (e.g., an audio stream) is in progress.

[0038] The user may listen or monitor for trigger events after joining the audio stream. For example, a graphical user interface running on user device 140 may prompt the user to listen or monitor for trigger events after joining the audio stream. The recorded audio or video containing the trigger event may be shown to the user after requesting the audio stream via user device 140 but before starting the audio stream.

[0039] B. Multi-directional audio streaming

[0040] Figure 2 A diagram 201 and method 200 for initiating multi-directional audio streaming according to an embodiment are shown. The method is illustrated as a logic flow diagram, each operation of which can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations may represent computer-executable instructions stored on one or more computer-readable storage media, which perform the operations when executed by one or more processors. Generally speaking, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the described process or method.

[0041] Turning to method 200 in more detail, at block 202, a computing device may detect a triggering event. The triggering event may be detected by computing device 210 in a monitored area according to the method disclosed at block 104. At block 204, computing device 210 may provide a notification to one or more user devices 250 via network 240. Some or all of user devices 250 may be notified, and in some cases, location data or other information may be used to determine which user devices 250 to notify. For example, if a user device is connected to the same WiFi network as computing device 210, the user device may not receive the notification.

[0042] The notification may include an audio clip containing the triggering event. The recording of the triggering event may allow the user to better understand and respond to the event. If the computing device or an electronic device communicatively connected to or associated with the electronic device includes a camera, the notification may include a video clip recorded during the triggering event. The notification may include one or more pictures captured by the camera. The notification may present options that the user can select using the user device 250. For example, the notification may prompt the user to call emergency services (e.g., 911). The notification may prompt the user to call, send a text message, send an email, or otherwise notify an emergency contact associated with the user device 250 or computing device 210.

[0043] In some cases, the computing device 210 may automatically call, text, email, or otherwise notify emergency services or emergency contacts. Certain triggering events may cause the electronic device 210 or the user device 250 to automatically contact emergency services or emergency contacts. For example, if a fire or a break is detected, emergency services may be automatically contacted, but if an animal is detected gnawing furniture, emergency services may not be automatically contacted. Depending on the severity of the triggering event, emergency services or emergency contacts may be automatically contacted. For example, if a smoke warning triggering event is detected, emergency services or emergency contacts may not be automatically contacted, but if a smoke warning, burning material, and fire sprinkler triggering event is detected, emergency services and emergency contacts may be automatically contacted. In some cases, information from a camera that is communicatively connected to or associated with the computing device 210 may determine whether to automatically contact emergency services or emergency contacts. For example, if a camera detects a sufficient amount of smoke, visible flames, or intruders in the monitored area, emergency services or emergency contacts may be automatically contacted.

[0044] At block 206, an audio stream may be initiated with the first user device. User device 250 may initiate the stream with computing device 210 via network 240. The audio stream may be initiated in response to a notification from block 204. The notification may include a license file that grants permission for the mobile device to initiate an audio stream with the computing device. In some cases, notifications may be provided to multiple user devices, but the license file may vary between user devices. For example, one version of the license file may allow the mobile device to initiate a stream, and a different version of the license file may allow the mobile device to join an existing call that has already been initiated.

[0045] At box 208, one or more additional user devices may be added to the audio stream. Audio 260 received as an audio stream from two or more user devices in user device 250 may be broadcast from a speaker on computing device 210 or controlled by the computing device. The user device in user device 250 may receive and broadcast an audio stream from one or more user devices in computing device 210 and user device 250. The audio stream may include audio recorded by at least one of computing device 210 or user device 250. When another device attempts to join the audio stream, computing device 210 and any user device 250 participating in the audio stream may receive a notification. In some cases, the notification may include a request to grant permission for another device to join the audio stream. Devices participating in the audio stream (e.g., computing device 210, user device 250, etc.) may be able to use notifications to invite one or more other devices to join the audio stream.

[0046] The user may listen or monitor for trigger events after joining the audio stream. For example, a graphical user interface running on one or more user devices 250 may prompt the user to listen or monitor for trigger events after joining the audio stream. The recorded audio or video containing the trigger event may be shown to the user after requesting the audio stream via the user device 250 but before starting the audio stream.

[0047] C. Select speakers

[0048] In some cases, multiple computing devices may be used in a monitored environment. It may be desirable to determine the computing device closest to the sound source so that an audio stream may be initiated with the device that is closest to the triggering event.

[0049] Figure 3 A method 300 for selecting a computing device in a monitored environment 301 according to an embodiment is shown. The method is illustrated as a logic flow diagram, each operation of which can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, an operation may represent computer-executable instructions stored on one or more computer-readable storage media, which perform the operation when executed by one or more processors. Generally speaking, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the described process or method.

[0050] Turning to method 300 in more detail, at step 302, a distance to a source may be determined. The distance may be a distance between a computing device (e.g., computing devices 310a-310c) and a source 312 of a trigger event 314 (e.g., audio, a particular sound, etc.). Computing devices 310a-310c may use information about trigger event 314 to determine a relative distance between the computing device and source 312. For example, computing devices 310a-310c may measure the length of the reverberation of trigger event 314 at each computing device to determine the distance to source 312.

[0051] For example, computing device 310a may have a measurable reverberation length because computing device 310a is located in a room with source 312. Computing devices 310b to 310c may have negligible reverberation lengths because both devices are not located in a room with source 312. If two or more computing devices are located in a room with source 312, the computing device with the shortest reverberation length may be closest to the source. In some cases, if the reverberation length is above a threshold, the computing device may be determined to be located in a room with a source. For example, the threshold length may be 1 millisecond (ms), 10ms, 50ms, 100ms, 200ms, 500ms, 1 second, 2 seconds, etc. The reverberation length may be determined by measuring the time it takes for the sound level to decrease by a certain amount of decibels (dB). For example, the reverberation length may be the time it takes for the sound to decrease by 60 dB.

[0052] Other methods for determining the distance between computing devices 310a-310c and source 312 are contemplated. Other methods may be used alone, in combination with each other, or with calculated reverberations to determine the distance. For example, the arrival time of trigger event 314 may be used to determine the distance between source 312 and computing devices 310a-310c. The clocks of computing devices 310a-310c may be synchronized, and the arrival time of trigger event 314 at each device may be determined. The device that first receives trigger event 314 may be the device closest to source 312.

[0053] At block 304, the determined distance may be received from a computing device. For example, computing device 310b may receive the determined distance from computing devices 310a and 310b. Computing device 310b may receive information about triggering event 314 from other computing devices, and computing device 310b may use the information to calculate the distance of one or more devices. In some cases, computing devices 310a to 310c may provide information about triggering event 314 to other devices (e.g., server device 410, user device 140, user device 250, user device 404, etc.) that can calculate the distance between the computing devices 310a to 310c and source 312. Computing devices 310a to 310c may provide distances or information about triggering event 314 via a network (e.g., network 408, etc.).

[0054] At block 306, the closest device may be identified. The closest device may be the computing device of computing devices 310a to 310c that has the shortest determined distance to source 312. These distances may be compared to determine the closest distance to source 312. These distances may be compared by one or more of computing devices 310a to 310c, server computers (e.g., server device 410, etc.), user devices (e.g., user device 404, etc.), etc. At block 308, a notification may be provided from the closest device. Notifications may be provided to user devices (e.g., user device 140, user device 250, user device 404, etc.) via a network (e.g., network 150, network 240, network 408, etc.). The user device may use the notification to initiate an audio stream as described herein.

[0055] D. Exemplary Computing Devices

[0056] Figure 4 400 is a simplified block diagram illustrating an example architecture of a system for detecting trigger events and taking actions based on trigger events according to some embodiments. The diagram includes a representative user device 402, one or more user devices 404, one or more additional computing devices 406, one or more networks 408, and a server device 410. Figure 4 Each of these elements depicted in FIG400 may be similar to one or more elements depicted in other figures described herein. In some embodiments, at least some elements of diagram 400 may be in a monitored environment (e.g., Figure 3 The monitored environment 301 is operated in the background.

[0057] The user device 704 can be any suitable computing device (e.g., a smart phone, a smart watch, a laptop computer, a tablet computer, etc.). In some embodiments, the user device can perform any one or more of the operations of the user device described herein. Depending on the type of user device and / or the location of the accessory device (e.g., within the monitored environment or outside the monitored environment), the user device can be enabled to communicate over a network 408 (e.g., including a LAN or WAN) using one or more network protocols (e.g., Bluetooth connection, Thread connection, ZigBee connection, WiFi connection, etc.) and network paths, as further described herein.

[0058] In some embodiments, server device 410 can be a computer system including at least one memory, one or more processing units (or processors), a storage unit, a communication device, and an I / O device. In some embodiments, server device 410 can perform any one or more operations of the operations of the server device described herein. In some embodiments, these elements can be implemented in a similar manner (or a different manner) as described with reference to similar elements of computing device 402.

[0059] In some embodiments, representative computing device 602 may correspond to any one or more of the computing devices described herein. For example, computing device 402 may correspond to Figure 3 The one or more computing devices of the computing devices of the monitored environment 301. The representative computing device can be any suitable computing device (e.g., a smart speaker, a mobile phone, a tablet, a smart hub speaker device, a smart media player communicatively connected to a TV, etc.). One or more additional computing devices 406 can correspond to the computing device 402 disclosed herein.

[0060] In some embodiments, one or more networks 408 may include an Internet WAN and a LAN. As described herein, a residential environment may be associated with a LAN, whereby devices present within the monitored environment may communicate with each other via the LAN. As described herein, a WAN may be external to the monitored environment. For example, a router associated with the LAN (and therefore the monitored environment) may enable traffic from the LAN to be sent to the WAN, and vice versa. In some embodiments, a server device 610 may be external to the monitored environment, and therefore communicate with other devices via the WAN.

[0061] As described herein, computing device 402 may represent one or more computing devices connected to one or more of networks 408. Computing device 402 has at least one memory 412, a communication interface 414, one or more processing units (or processors) 416, a storage unit 418, and one or more input / output (I / O) devices 420.

[0062] Turning to each element of the computing device 402 in further detail, the processor 416 may be suitably implemented in hardware, computer executable instructions, firmware, or a combination thereof. The computer executable instructions or firmware implementation of the processor 416 may include computer executable instructions or machine executable instructions written in any suitable programming language to perform the various functions described.

[0063] The memory 412 can store program instructions that can be loaded and executed on the processor 416, as well as data generated during the execution of these programs. Depending on the configuration and type of the computing device 402, the memory 412 can be volatile (such as random access memory ("RAM")) or non-volatile (such as read-only memory ("ROM"), flash memory, etc.). In some specific implementations, the memory 412 may include a variety of different types of memory, such as static random access memory ("SRAM"), dynamic random access memory ("DRAM") or ROM. The computing device 402 may also include additional storage devices 418, such as removable storage devices or non-removable storage devices, including but not limited to magnetic storage devices, optical disks and / or tape storage devices. Disk drives and their associated computer-readable media can provide non-volatile storage devices for computer-readable instructions, data structures, program modules and other data for computing devices. In some embodiments, the storage device 418 can be used to store data content received from one or more other devices (e.g., server devices 410, other computing devices or user devices 404). For example, storage device 418 can store accessory management settings, accessory settings, and user data associated with users affiliated with the monitored environment.

[0064] The computing device 402 may also include a communication interface 414 that allows the computing device 402 to communicate with a stored database, another computing device or server, a user terminal, or other device over a network 408. The computing device 402 may also include an I / O device 420, such as to enable connection to a keyboard, mouse, pen, voice input device, touch input device, display, speaker, printer, etc. In some embodiments, the I / O device 420 may be used to output an audio response or other indication as part of performing a response to a user request. The I / O device may include one or more speakers 446 or one or more microphones 448.

[0065] Memory 412 may include an operating system 422 and one or more applications or services for implementing the features disclosed herein, including a communication module 424, a user interface module 26, a sound processing module 430, an accessory interaction instance 432, and a management module 434. The sound processing module also includes a wake-up word module 436, and the accessory interaction instance 432 also includes a digital assistant 638. The sound processing module also includes a trigger event module 450, which can be configured to detect one or more sounds associated with a trigger event.

[0066] The communication module 424 may include code that causes the processor 416 to generate instructions and messages, send data, or otherwise communicate with other entities. As described herein, the communication module 424 may send messages via one or more network paths of the network 408 (e.g., via a LAN or Internet WAN associated with the monitored environment). The user interface module 426 may include code that causes the processor 416 to present information corresponding to computing devices and user devices present within or associated with the monitored environment.

[0067] The sound processing module 430 may include code that causes the processor 416 to receive and process audio input corresponding to speech or other sounds that can be analyzed by the techniques described herein. The wake-up word module 436 may include code that causes the processor 416 to receive and process a portion of the audio input corresponding to a trigger or wake-up word. For example, the wake-up word module 436 may analyze a portion of the audio input to determine the presence of the wake-up word. In some embodiments, the speech processing module may also determine the language corresponding to the audio input, and use the language to inform the analysis of the wake-up word portion. The trigger event module 450 may include code that causes the processor 416 to receive and process a portion of the audio input corresponding to the trigger event (e.g., an audio clip). For example, the trigger event module 450 may analyze the audio clip to determine the presence of a specific sound associated with the trigger event. In some embodiments, the trigger event module may also determine the trigger event corresponding to the audio input.

[0068] II. Methodology

[0069] Figure 5 A method 500 for event monitoring using a smart speaker according to an embodiment is shown. The method is illustrated as a logic flow chart, each operation of which can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, an operation may represent computer-executable instructions stored on one or more computer-readable storage media, which perform the operations when executed by one or more processors. In general, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the described process or method.

[0070] At box 502, the computing device can monitor the audio stream to find one or more triggering events including a specific sound. The audio stream can be a one-way audio stream. The computing device can be a smart speaker, a smart media player, a tablet computer, or a user device such as a smart phone, etc. The computing device (e.g., computing device 402, sound processing module 430, trigger event module 450, etc.) can use a microphone (e.g., microphone 448, etc.) to monitor the audio stream. One or more triggering events (e.g., specific sounds, triggering events 120, triggering events 314, etc.) may include sounds associated with any event that can be detected by audio. For example, a triggering event may include fire, water damage, forced entry into a monitored environment, animal noise (e.g., barking), people falling, carbon monoxide warnings, broken windows, shootings, etc.

[0071] At box 504, the computing device may detect a specific sound. The sound may be detected by a trigger event module (e.g., sound processing module 430, trigger event module 440) of a computing device (e.g., computing device 110, computing device 210, computing devices 310a to 310c, computing device 402, another computing device 406, etc.). In some cases, more than one computing device may detect a specific sound or trigger event (e.g., computing device 110, computing device 210, computing devices 310a to 310c, computing device 402, another computing device 406, etc.). Each device that detects the sound may use its sound processing module to determine the distance between the separate computing device and the trigger event. The computing device may receive additional distances determined by one or more additional computing devices or server devices (e.g., server device 410). The sound processing module may compare the determined distances to identify the computing device closest to the specific sound (e.g., trigger event 120, trigger event 314, etc.). The computing device determined to be closest to a particular sound may provide a notification to a user device, such as user device 404. The distance may be determined using information about the particular sound, including measured reverberation or arrival time of the sound as described herein.

[0072] At block 506, the computing device may provide an event notification to a user device. The notification may be provided to a user device (e.g., user device 140, user device 250, user device 404, etc.) via a network (e.g., network 150, network 240, network 408, etc.). The notification may be provided by communication module 424 via communication interface 414. The notification may be provided to more than one user device. In some cases, the notification may be provided by another computing device (e.g., computing device 110, computing device 210, computing devices 310a-310c, computing device 402, another computing device 406, etc.).

[0073] In some cases, an event timer may be started in response to transmitting the notification, and the call may be initiated during the event timer. The timer may be started by a communication module (e.g., communication module 424). In some cases, the notification may identify the location where a particular sound was detected, the particular sound detected (e.g., running water), or a triggering event associated with the particular sound (e.g., water damage).

[0074] At block 508, the computing device may receive an indication to initiate a two-way audio stream. The indication may be received from one or more user devices (e.g., computing device 110, computing device 210, computing devices 310a-310c, computing device 402, additional computing device 406, etc.) via one or more networks (e.g., network 150, network 240, network 408). The two-way audio stream may be a multi-directional audio stream between the computing device and two or more user devices.

[0075] At box 510, the computing device may provide a first alert that permission has been granted to start a two-way audio stream. The alert may be provided via a speaker (e.g., speaker 446) of the computing device. The alert may be an audio alert given before the two-way audio stream has been started, during the stream, or after the stream has ended. The alert may be a tone or voice that alerts anyone in the monitored environment that a call has started or will start. The alert may be repeated at periodic intervals during the two-way audio stream. In some cases, the alert may include a visual alert, such as a light on the computing device.

[0076] At block 512, the computing device may initiate a two-way audio stream. The stream may be initiated by a communication module (e.g., communication module 424, etc.), and the stream may be between a user device (e.g., user device 404) and a computing device (e.g., computing device 402, etc.) via one or more networks (e.g., network 408). The computing device may generate a second alarm for the duration of the two-way audio stream. The alarm may be an audio alarm or a visual alarm generated continuously or at regular intervals during the two-way stream.

[0077] The above describes illustrative techniques for initiating a call using a computing device in response to a detected event. Some or all of these techniques may be implemented at least in part through an architecture (such as at least in the above Figures 1 to 5The present invention can be implemented by using the architectures shown in the above (such as those shown in the above), but it is not necessary to implement them by these architectures. Although many embodiments in the embodiments are described above with reference to computing devices and user devices, it should be understood that other types of computing devices may be applicable to executing the technology disclosed herein. In addition, various non-limiting examples are described in the foregoing description. For the purpose of explanation, many specific configurations and details are set forth in order to provide a thorough understanding of the examples. However, it should also be obvious to those skilled in the art that the examples can be implemented without these specific details. In addition, sometimes well-known features are omitted or simplified to prevent confusion with the examples described herein.

[0078] Various embodiments can also be implemented in a variety of operating environments, which in some cases may include one or more user computers, computing devices, or processing devices that can be used to operate any of a plurality of applications. User equipment or client devices may include any of a plurality of general-purpose personal computers, such as desktop computers or laptop computers running standard operating systems, and cellular devices, wireless devices, and handheld devices that run mobile software and can support a plurality of networking protocols and instant messaging protocols. Such systems may also include multiple workstations running a variety of commercially available operating systems and any of other known applications for purposes such as development and database management. These devices may also include other electronic devices, such as virtual terminals, thin clients, gaming systems, and other devices that can communicate via a network.

[0079] Most embodiments utilize at least one network familiar to those skilled in the art to support communications using any of a variety of commercially available protocols such as TCP / IP, OSI, FTP, UPnP, NFS, CIFS, and AppleTalk. The network can be, for example, a local area network, a wide area network, a virtual private network, the Internet, an intranet, an extranet, a public switched telephone network, an infrared network, a wireless network, and any combination thereof.

[0080] In embodiments utilizing a web server, the web server may run any of a variety of server or middle-tier applications, including HTTP servers, FTP servers, CGI servers, data servers, Java servers, and business application servers. The server may also be capable of executing programs or scripts in response to requests from user devices, such as by executing one or more applications that may be implemented in any programming language (such as C, C# or C++) or any scripting language (such as Perl, Python or TCL) and their combinations. The server may also include a database server, including but not limited to and Those obtained commercially.

[0081] Environment can include multiple data storage and other memory and storage media, as discussed above. These can reside in multiple locations, such as on the storage media of one or more computers local in the computer (and / or reside in one or more computers in the computer) or on the storage media of any or all computers in the computer away from the network. In a specific set of embodiments, information can reside in a storage area network (SAN) familiar to those skilled in the art. Similarly, any necessary files for performing functions belonging to computers, servers or other network devices can be appropriately stored locally and / or remotely. When the system includes a computerized device, each such device can include a hardware element that can be electrically coupled via a bus, the element including, for example, at least one central processing unit (CPU), at least one input device (e.g., mouse, keyboard, controller, touch screen or keypad) and at least one output device (e.g., display device, printer or speaker). Such a system can also include one or more storage devices, such as disk drives, optical storage devices and solid-state storage devices (such as RAM or ROM) and removable media devices, memory cards, flash memory cards, etc.

[0082] Such devices may also include a computer-readable storage medium reader, a communication device (e.g., a modem, a network card (wireless or wired), an infrared communication device, etc.) and a working memory as described above. A computer-readable storage medium reader may be connected to or configured to receive a non-transient computer-readable storage medium representing a remote, local, fixed and / or removable storage device, and a storage medium for temporarily and / or more permanently containing, storing, sending and retrieving computer-readable information. Systems and various devices will also typically include a plurality of software applications, modules, services or other elements located in at least one working memory device, including an operating system and an application, such as a client application or a browser. It should be understood that alternative embodiments may have many variations according to those described above. For example, custom hardware may also be used, and / or specific elements may be implemented in hardware, software (including portable software, such as applets), or both. In addition, connections with other computing devices (such as network input / output devices) may be employed.

[0083] Non-transitory storage media and computer-readable storage media for containing code or parts of code may include any suitable media known or used in the art, such as but not limited to volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data), including RAM, ROM, electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, DVD or other optical storage, cassettes, tapes, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by system devices. Based at least in part on the disclosure and teachings provided herein, a person of ordinary skill in the art will recognize other ways and / or methods of implementing various embodiments. However, computer-readable storage media do not include transient media such as carrier waves.

[0084] Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. It will, however, be evident that various modifications and changes may be made thereto without departing from the broader spirit and scope of the present disclosure as set forth in the claims.

[0085] Other variations are within the spirit of the present disclosure. Thus, while the disclosed technology is susceptible to various modifications and alternative constructions, certain illustrative embodiments thereof are shown in the drawings and have been described in detail above. However, it should be understood that it is not intended to limit the present disclosure to the particular forms disclosed, but on the contrary, it is intended to cover all modifications, alternative constructions, and equivalent forms that fall within the spirit and scope of the present disclosure as defined by the appended claims.

[0086] The use of the terms "one", "an" and "the" and similar indicators in the context of describing the disclosed embodiments (particularly in the context of the claims below) will be interpreted as covering the singular and plural, unless otherwise stated or clearly contradicted by the context. Unless otherwise stated, the terms "comprise", "have", "include" and "contain" should be interpreted as open terms (i.e., meaning "including but not limited to"). The term "connected" is interpreted as being partially or completely included, attached or joined together even if there are intervening things. The phrase "at least in part based on" should be understood as open, and is not limited in any way, and is intended to be interpreted or otherwise understood as "at least in part based on" where appropriate. Unless otherwise stated herein, the description of the numerical range herein is intended only to be used as a simple method of individually referring to each individual value falling within the range, and each individual value is incorporated into the specification, as if quoted separately in this article. All methods described herein can be performed in any suitable order, unless otherwise stated herein or otherwise clearly contradicted by the context. Unless otherwise stated, the use of any and all examples or exemplary language (e.g., "such as") provided herein is intended only to better illustrate the embodiments of the present disclosure and will not limit the scope of the present disclosure. No language in the specification should be construed as indicating that any unclaimed element is essential to the practice of the present disclosure.

[0087] Unless otherwise specifically stated, disjunctive language such as the phrase "at least one of X, Y, or Z" is understood in context to be generally used to present an item, term, etc., which can be X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z). Thus, such disjunctive language is generally not intended to, and should not, imply that certain embodiments require that at least one of X, at least one of Y, or at least one of Z each be present. Additionally, conjunctive language such as the phrase "at least one of X, Y, and Z" should also be understood to mean X, Y, Z, or any combination thereof, including "X, Y, and / or Z," unless otherwise specifically stated.

[0088] Preferred embodiments of the present disclosure are described herein, including the best mode known to the inventor for carrying out the present disclosure. After reading the foregoing description, variations of those preferred embodiments may become apparent to those of ordinary skill in the art. The inventor expects that the technical staff will appropriately adopt such variations, and the inventor intends to practice the present disclosure in a manner different from that specifically described herein. Therefore, as permitted by applicable law, the present disclosure includes all modifications and equivalents of the subject matter recorded in the claims attached to the present disclosure. In addition, unless otherwise noted herein or obviously contradictory to the context, the present disclosure includes any combination of all possible variations of the above-mentioned elements.

[0089] All references, including publications, patent applications, and patents, cited herein are hereby incorporated by reference to the same extent as if each reference were individually and specifically indicated to be incorporated by reference and were set forth in its entirety herein.

[0090] As described above, one aspect of the present technology is to collect and use data (e.g., recorded voice) for event detection. The present disclosure contemplates that, in some cases, these collected data may include personally identifiable information (PII) data that uniquely identifies or can be used to contact or locate a specific person. Such personal information data may include an individual's voice characteristics, name, demographic data, location-based data (e.g., GPS coordinates), phone number, email address, Twitter ID, home address, or any other identification or personal information.

[0091] The present disclosure recognizes that the use of such personal information data in the present invention technology can be used to benefit users. For example, personal information data can be used to notify users of events occurring in the user's home.

[0092] The disclosure anticipates that entities responsible for the collection, analysis, disclosure, transmission, storage or other uses of such personal information data will comply with sound privacy policies and / or privacy measures. Specifically, such entities should implement and adhere to the use of privacy policies and practices that are recognized as meeting or exceeding industry or government requirements for maintaining the privacy and security of personal information data. Such policies should be easy for users to access and should be updated as changes in the collection and / or use of data occur. Personal information from users should be collected for legal and reasonable entity purposes and should not be shared or sold outside these legal purposes. In addition, such collection / sharing should be carried out after receiving the user's informed consent. In addition, such entities should consider taking any necessary steps to protect and safeguard access to such personal information data and ensure that other entities with access to personal information data comply with their privacy policies and procedures. In addition, such entities may subject themselves to third-party assessments to prove that they comply with widely accepted privacy policies and practices. In addition, policies and practices should be adapted to specific types of personal information data collected and / or accessed, and to applicable laws and standards including considerations of special jurisdictions. For example, in the United States, the collection or acquisition of certain health data may be governed by federal and / or state laws (such as the Health Insurance Portability and Accountability Act (HIPAA)); while health data in other countries may be subject to other regulations and policies and should be handled accordingly. Therefore, different privacy measures should be advocated for different types of personal data in each country.

[0093] Regardless of the foregoing, the present disclosure also contemplates implementation schemes in which users selectively block the use or access of personal information data. That is, the present disclosure contemplates that hardware elements and / or software elements may be provided to prevent or block access to such personal information data. For example, with respect to services related to performing facial recognition, the technology of the present invention may be configured to allow a user to choose to “opt in” or “opt out” to participate in the collection of personal information data at any time during or after registration for the service. In addition to providing “opt-in” and “opt-out” options, the present disclosure also contemplates providing notifications related to access or use of personal information. For example, a user may be notified that their personal information data will be accessed when downloading an application, and then be reminded again just before the personal information data is accessed by the application.

[0094] In addition, it is the intention of the present disclosure that personal information data should be managed and processed in a manner that minimizes the risk of unintentional or unauthorized access or use. Once the data is no longer needed, the risk can be minimized by limiting the collection of data and deleting the data. In addition, and when applicable, including in certain health-related applications, data de-identification can be used to protect the privacy of users. De-identification can be facilitated by removing specific identifiers (e.g., date of birth, etc.), controlling the amount or specificity of stored data (e.g., collecting location data at the city level rather than at the address level), controlling how the data is stored (e.g., aggregating data between users), and / or other methods when appropriate.

[0095] Therefore, while the present disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, the present disclosure also contemplates that various embodiments may be implemented without access to such personal information data. That is, various embodiments of the present technology will not fail to function properly due to the lack of all or part of such personal information data.

Claims

1. A computer-implemented method comprising: monitoring, by a computing device, a one-way audio stream for one or more triggering events, the one or more triggering events comprising a particular sound identified in the one-way audio stream; detecting, by the computing device, the specific sound in the one-way audio stream; providing, by the computing device, an event notification to a user device via a network connection, the event notification requesting permission to initiate a two-way audio stream between the computing device and the user device; receiving, by the computing device, an indication to initiate the two-way audio stream with the user device based at least in part on the event notification; providing, via a speaker of the computing device, a first alert that permission has been granted to initiate the two-way audio stream; as well as The two-way audio stream with the user device is initiated by the computing device.

2. The method of claim 1, wherein providing event notification further comprises: determining a distance between the computing device and a source of the detected particular sound; transmitting the determined distances to a plurality of monitoring devices associated with the computing device; receiving one or more determined distances from a subset of the plurality of monitoring devices that detected the particular sound; comparing the determined distance to the one or more determined distances to determine whether the computing device is closer to the source than the subset of monitoring devices; as well as Upon determining that the computing device is closer than the subset of monitoring devices, an event notification is provided to the user device. The method of claim 3 , wherein a notification is provided to a second user device.

4. The method according to any one of claims 1 to 3, wherein the bidirectional audio stream is a multi-directional audio stream between the computing device, the user device and the second user device.

5. The method according to any one of claims 1 to 4, wherein providing event notification further comprises: An event timer is initiated by the computing device, wherein the bidirectional audio stream can be initiated during the event timer.

6. The method of any one of claims 1 to 5, wherein the event notification identifies the specific sound recognized in the audio stream.

7. The method according to any one of claims 1 to 6, wherein initiating the two-way audio stream further comprises: A second alert is provided by the computing device at regular intervals during the duration of the two-way audio stream, the second alert announcing that a two-way audio stream has been initiated.

8. A non-transitory computer-readable medium storing an instruction set, the instruction set comprising one or more instructions that, when executed by one or more processors of a computing device, cause the computing device to perform operations comprising: monitoring a one-way audio stream for one or more triggering events, the one or more triggering events comprising a particular sound identified in the one-way audio stream; detecting the specific sound in the audio stream; providing an event notification to a user device via a network connection, the event notification requesting permission to initiate a two-way audio stream between the computing device and the user device; receiving an indication to initiate the two-way audio stream with the user device based at least in part on the event notification; providing, via a speaker of the computing device, a first alert that permission has been granted to initiate the two-way audio stream; as well as Initiate the two-way audio stream with the user device.

9. The non-transitory computer readable medium of claim 8, wherein providing event notifications further comprises: determining a distance between the computing device and a source of the detected particular sound; transmitting the determined distances to a plurality of monitoring devices associated with the computing device; receiving one or more determined distances from a subset of the plurality of monitoring devices that detected the particular sound; comparing the determined distance to the one or more determined distances to determine whether the computing device is closer to the source than the subset of monitoring devices; as well as Upon determining that the computing device is closer than the subset of monitoring devices, an event notification is provided to the user device.

10. The non-transitory computer readable medium of any one of claims 8 to 9, wherein the notification is provided to a second user device.

11. The non-transitory computer-readable medium of claim 10, wherein the bidirectional audio stream is a multi-directional audio stream between the computing device, the user device, and the second user device.

12. The non-transitory computer readable medium of any one of claims 8 to 11, wherein providing event notifications further comprises: An event timer is initiated, wherein the bidirectional audio stream can be initiated during the event timer.

13. The non-transitory computer-readable medium of any one of claims 8 to 12, wherein the event notification identifies the specific sound recognized in the audio stream.

14. The non-transitory computer-readable medium of any one of claims 8 to 13, wherein initiating the two-way audio stream further comprises: A second alert is provided by the computing device at regular intervals during the duration of the two-way audio stream, the second alert announcing that a two-way audio stream has been initiated.

15. A computing device comprising: Storage devices; speaker; and One or more processors configured to execute program instructions stored in the storage device to at least perform the following: monitoring a one-way audio stream for one or more triggering events, the one or more triggering events comprising a particular sound identified in the one-way audio stream; detecting the specific sound in the audio stream; providing an event notification to a user device via a network connection, the event notification requesting permission to initiate a two-way audio stream between the computing device and the user device; receiving an indication to initiate the two-way audio stream with the user device based at least in part on the event notification; providing, via the speaker, a first alert that permission has been granted to initiate the two-way audio stream; as well as Initiate the two-way audio stream with the user device.

16. The computing device of claim 15, wherein providing event notifications further comprises: determining a distance between the computing device and a source of the detected particular sound; transmitting the determined distances to a plurality of monitoring devices associated with the computing device; receiving one or more determined distances from a subset of the plurality of monitoring devices that detected the particular sound; comparing the determined distance to the one or more determined distances to determine whether the computing device is closer to the source than the subset of monitoring devices; as well as Upon determining that the computing device is closer than the subset of monitoring devices, an event notification is provided to the user device.

17. The computing device of any one of claims 15 to 16, wherein the notification is provided to a second user device.

18. The computing device of claim 17, wherein the bidirectional audio stream is a multi-directional audio stream between the computing device, the user device, and the second user device.

19. The computing device of any one of claims 15 to 18, wherein providing event notifications further comprises: An event timer is initiated by the computing device, wherein the bidirectional audio stream can be initiated during the event timer.

20. The computing device of any one of claims 15 to 19, wherein the event notification identifies the particular sound recognized in the audio stream.

Images