Method of generating metadata based on data obtained from one or more connected devices

Abstract

The present disclosure relates to a method for determining a location of a person or object within a space. The method includes obtaining, by one or more computing devices, data from one or more sensor devices associated with one or more connected devices located within the space. The method further includes generating, by the one or more computing devices, metadata indicating a presence of the person or object within the space based at least in part on the data. The method further includes obtaining, by the one or more computing devices, data indicating a user request associated with obtaining a location of the person or object within the space. The method further includes determining, by the one or more computing devices, the location of the person or object based at least in part on the metadata.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Application No. 62 / 909,848, filed October 3, 2019, the disclosure of which is incorporated herein by reference in its entirety and claims priority to that application. Technical Field

[0003] This disclosure generally relates to a method for generating metadata based on data obtained from one or more sensor devices associated with one or more connected devices located throughout the space. Background Technology

[0004] With the advancement of Internet of Things (IoT) technology, spaces (e.g., offices, homes, etc.) can be equipped with various types of connected devices (e.g., thermostats, power switches). These connected devices can include one or more sensor devices (e.g., microphones, motion sensors, etc.) capable of collecting data indicating activity within the space. For example, one or more sensor devices can include one or more motion sensors configured to acquire data indicating the movement of people within the space. Alternatively or additionally, one or more sensor devices can include one or more microphones configured to acquire audio data indicating the presence of people within the space. Summary of the Invention

[0005] Various aspects and advantages of embodiments of this disclosure will be set forth in part in the description which follows, or may be learned from the description or by practice of the embodiments.

[0006] In one aspect, this disclosure relates to a method for determining the location of a person or object within a space. The method may include obtaining data from one or more sensor devices associated with one or more connected devices located within the space by one or more computing devices. Furthermore, the method may include generating metadata indicating the presence of a person or object within the space by one or more computing devices, at least in part, based on the data. The method may also include obtaining data from one or more computing devices indicating a user request associated with determining the location of a person or object within the space. Additionally, the method may include determining the location of a person or object by one or more computing devices, at least in part, based on the metadata. Furthermore, the method may include providing a notification indicating the location of a person or object by one or more computing devices.

[0007] These and other features, aspects, and advantages of the various embodiments will become better understood with reference to the following description and the appended claims. The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments of the present disclosure and, together with the specification, serve to explain the relevant principles. Attached Figure Description

[0008] Referring to the accompanying drawings, a detailed discussion of embodiments for those skilled in the art is set forth in the specification, in which:

[0009] Figure 1 A space having one or more connected devices is depicted according to an example embodiment of this disclosure;

[0010] Figure 2 A block diagram depicts the components of a connection device according to an exemplary embodiment of the present disclosure;

[0011] Figure 3 A perspective view of a connection device according to an example embodiment of the present disclosure is depicted;

[0012] Figure 4 Example embodiments according to this disclosure are depicted. Figure 3 Another block diagram of the components of the connecting device;

[0013] Figure 5 A system for determining the location of a person in a space is described according to an example embodiment of the present disclosure;

[0014] Figure 6 A flowchart is depicted for a method for generating metadata based on data obtained from one or more connected devices in space, according to an example embodiment of the present disclosure;

[0015] Figure 7 A flowchart is depicted illustrating a method for determining the position of a person or object in space according to an exemplary embodiment of the present disclosure; and

[0016] Figure 8 A block diagram depicts the components of a computing system according to an example embodiment of the present disclosure. Detailed Implementation

[0017] Reference will now be made in detail to embodiments, one or more examples of which are illustrated in the accompanying drawings. Each example is provided as an explanation of the embodiments and not as a limitation thereof. In fact, it will be apparent to those skilled in the art that various modifications and changes can be made to the embodiments without departing from the scope or spirit of this disclosure. For example, features shown or described as part of one embodiment may be used with another embodiment to produce further embodiments. Therefore, aspects of this disclosure are intended to cover these modifications and variations.

[0018] An exemplary aspect of this disclosure relates to a method for determining the location of a person or object within a space. In some embodiments, the method according to an exemplary aspect of this disclosure may include obtaining data from one or more sensor devices associated with one or more connected devices located within the space by one or more computing devices. For example, the data obtained from the one or more sensor devices may indicate one or more audible sounds within the space. Alternatively or additionally, the data may indicate the power consumption of a powered load selectively coupled to a power source via one or more connected devices. As will be discussed in more detail below, the data obtained from the one or more sensor devices may be used to generate metadata indicating the presence of a person or object within the space.

[0019] In some implementations, the method may include inputting data obtained from one or more sensing devices into one or more computing devices as input to a classifier (e.g., a machine learning model). The classifier may be configured to detect the presence of people or objects in a space based at least in part on the data obtained from the one or more sensing devices.

[0020] Therefore, the method may also include metadata obtained by one or more computing devices as the output of a classifier. The metadata may indicate the location of a person or object in the space, and in some embodiments, may also include a timestamp to indicate when a person or object was detected in the space.

[0021] In some implementations, the method may include acquiring data from one or more computing devices that indicates a user request to obtain the location of a person or object within a space. For example, the data indicating a user request may be audio data instructing the user to speak one or more voice commands (e.g., “Where was the last time I saw Warren?” or “Show me all the videos of Warren”) to request the location of a person or object within the space. However, it should be understood that the user can provide the data indicating a user request by any suitable method. For example, in some implementations, the data indicating a user request may be obtained through interaction between the user and an application (e.g., a mobile app) running on a mobile computing device (e.g., a smartphone, tablet, etc.) associated with the user. As will be discussed in more detail below, metadata can be used to determine the location of a person or object within the space.

[0022] This method may include determining the location of a person or object within a space by one or more computing devices, at least in part, based on metadata. For example, in some embodiments, the metadata may be associated with video data obtained from one or more image capture devices within the space. More specifically, the metadata may be associated with one or more frames of video data having timestamps corresponding to the timestamps associated with the metadata. In this way, the metadata can be associated with frames of video data depicting a person or object within the space. Thus, in some embodiments, the method may include determining the location of a person or object based on one or more frames associated with the metadata.

[0023] In some implementations, one or more computing devices may be configured to provide a notification indicating the location of a person within a space. For example, the notification may be an auditory notification (e.g., “Warren was last seen in the boardroom at noon”) provided via one or more output devices (e.g., speakers) of one or more connected devices. Alternatively or additionally, the notification may be a visual notification. For example, in some implementations, a visual notification may include one or more videos depicting a person or object displayed by one or more computing devices on one or more output devices. In some implementations, one or more output devices (e.g., displays) may be associated with one or more connected devices. Alternatively or additionally, the notification may be provided to a mobile computing device (e.g., a smartphone, tablet, etc.) associated with the location of the requester. For example, the notification may be a visual notification (e.g., a text message, email). Alternatively or additionally, the notification may be an auditory notification, such as an automated telephone call.

[0024] The computing system according to the exemplary aspects of this disclosure can provide numerous technical benefits, particularly in the field of computing technology. For example, generating metadata indicating the presence of a user can facilitate information retrieval, which can reserve computing resources for more core functions of connected devices, networks, and / or associated systems. As another example, associating metadata with one or more frames of video data reduces the amount of video data that one or more computing devices must search to determine the location of a person in space. In this way, the amount of time that one or more computing devices can use to perform critical processing functions can be increased.

[0025] Now for reference Figure 1An example space 100 is provided according to an exemplary embodiment of the present disclosure. As shown, the interior of space 100 is divided into a plurality of rooms 110 and a corridor 112. Although space 100 is depicted as having only one floor, it should be understood that the present disclosure is not limited to a single-story building. As shown, a plurality of connecting devices 120 are disposed throughout the interior of space 100. For example, at least one of the plurality of connecting devices 120 may be located in each of the plurality of rooms 110. Alternatively or additionally, one or more of the plurality of connecting devices 120 may be located within the corridor 112 of space 100.

[0026] In some embodiments, one or more image capture devices 130 may be located throughout the entire interior of the space. As shown, one or more image capture devices 130 may be independent devices separate from the plurality of connection devices 120. However, in alternative embodiments, one or more image capture devices 130 may be part of (e.g., integrated with) one or more of the plurality of connection devices 120.

[0027] As shown, one or more image capture devices 130 may be located within the corridor 112 of space 100. However, it should be understood that one or more image capture devices 130 may be located at any suitable location within the interior of space 100. For example, in some embodiments, the image capture devices 130 may be located in each of a plurality of rooms 110. In this way, the image capture devices 130 can acquire video data indicating events occurring in corresponding rooms of the plurality of rooms 110.

[0028] Now for reference Figure 2 According to exemplary embodiments of this disclosure, suitable components for a connectivity device 120 are provided. As shown, the connectivity device 120 may include one or more sensor devices. For example, in some embodiments, the one or more sensor devices may include one or more microphones 210. The one or more microphones 210 may be configured to obtain indication space 100 ( Figure 1 Audio data of one or more audible sounds occurring within space 100. For example, one or more microphones 210 may acquire audio data indicating that a person is speaking within space 100. As another example, one or more microphones 210 may acquire audio data indicating that a person is moving around within space 100.

[0029] In some embodiments, one or more sensor devices of the connection device 120 may include one or more motion sensors 212. The one or more motion sensors 212 may be configured to obtain indication space 100 ( Figure 1The motion sensor 212 can detect the movement occurring within space 100. For example, in some embodiments, one or more motion sensors 212 can detect people entering or leaving space 100 or a specific area of ​​space 100. However, it should be understood that any suitable type of motion sensor can be used to obtain data indicating movement occurring within space 100. For example, in some embodiments, one or more motion sensors may include passive infrared (PIR) sensors.

[0030] In some embodiments, the connection device 120 may be configured to selectively couple one or more electrical loads (not shown) to a power source (e.g., AC mains power). In such embodiments, one or more sensing devices of the connection device 120 may include a power metering circuit 214. The power metering circuit 214 may include at least one of a current sensor and a voltage sensor. In this way, when coupled to a power source via the connection device 120, the power metering circuit 214 can obtain data indicating the power consumption of one or more electrical loads.

[0031] In some implementations, the connection device 120 may include one or more output devices. For example, one or more output devices may include one or more speakers 216. In this way, power can be supplied to space 100 via one or more speakers 216. Figure 1 One or more people within the group provide auditory notifications.

[0032] In some embodiments, the connectivity device 120 may include a computing system 220. The computing system 220 may be communicatively coupled to one or more sensor devices (e.g., microphone(s) 210, motion sensors(s) 212, power metering circuitry 214). In this way, the computing system 220 can acquire data from the one or more sensor devices. Furthermore, the computing system 220 may be operatively coupled to one or more output devices (e.g., speaker(s) 216). In this way, the computing system 220 may provide notifications via one or more output devices of the connectivity device 120.

[0033] In some embodiments, the connection device 120 may include communication circuitry 230. Communication circuitry 230 may include associated electronic circuitry systems that can be used to communicatively couple computing system 220 to other devices, such as space 100. Figure 1 The computing device 240 is associated with other connected devices 120 within the system. In some embodiments, communication circuitry 230 may allow the computing system 220 to communicate directly with other connected devices. In alternative embodiments, communication circuitry 230 may provide communication with other connected devices via a network.

[0034] A network can be any suitable type of network, such as a Power over Ethernet (PoE) network, a local area network (e.g., an intranet), a wide area network (e.g., the Internet), a low-power wireless network (e.g., Bluetooth Low Energy (BLE), Zigbee, etc.), or some combination thereof, and can include any number of wired or wireless links. Generally, communication on a network can be implemented using a wide variety of communication protocols, encodings or formats, and / or protection schemes through any type of wired or wireless connection.

[0035] Example communication technologies used in the exemplary aspects of this disclosure may include, for example, Bluetooth Low Energy, Bluetooth Mesh Networking, Near Field Communication, Thread, TLS (Transport Layer Security), Wi-Fi (e.g., IEEE 802.11), Wi-Fi Direct (for peer-to-peer communication), Z-Wave, Zigbee, Halow, cellular communication, LTE, Low Power Wide Area Networking, VSAT, Ethernet, MoCA (Multimedia over Coax Alliance), PLC (Power-line communication), DLT (digital line transmission), Power over Ethernet, etc. Other suitable wired and / or wireless communication technologies may be used without departing from the scope of this disclosure.

[0036] Now for reference Figure 3 According to exemplary embodiments of this disclosure, an example connection device 120 is provided. As shown, in some embodiments, the connection device 120 may be a power switch configured to receive conductors 302, 304, and 306 for delivering power to one or more electrical loads. For example, power may be delivered to the connection device 120 from a power source (e.g., a circuit breaker, panel, etc.) via conductors 304 and 306. In some embodiments, conductor 304 may be a live conductor, and conductor 306 may be a neutral conductor. Additionally, conductor 302 may be a load conductor (e.g., a load line) for delivering power to one or more electrical loads (e.g., lighting fixtures, electronic devices, electrical outlets, appliances, machines, etc.).

[0037] In some embodiments, the connection device 120 may be configured to control power delivery to one or more electrical loads via conductor 302 via a power interrupter. The power interrupter can control whether power is delivered through conductor 302. In some embodiments, the power interrupter may be a thyristor 355 (e.g., a TRIAC). When the thyristor 355 is in a first state, power is conducted to one or more electrical loads via conductor 302. When the thyristor 355 is in a second state, power is not conducted to one or more electrical loads via conductor 302.

[0038] For purposes of illustration and discussion, reference is made to thyristor power terminators to discuss various aspects of this disclosure. Other suitable devices and / or components, such as power semiconductors, relays, contactors, mechanical switches, etc., may be used to control power delivery via conductor 302 without departing from the scope of this disclosure.

[0039] In some implementations, the state of the thyristor 355 can be controlled based on various inputs. For example, the state of the thyristor 355 can be controlled based on user input received at an interface element (such as a joystick button 310 or a switch of the connection device 120). The state of the thyristor 355 can also be controlled based on signals received via a communication link from other devices (e.g., user equipment such as a smartphone, tablet, wearable device, laptop, or display with one or more processors).

[0040] In some embodiments, the front panel 305 of the connecting device 120 may include a joystick button 310, a paddle-shaped housing 308, and a heat sink 312. The joystick button 310 may be received in the paddle-shaped housing 308. The joystick button 310 is rotatable about an axis passing through its center, such that when a user presses the top portion of the joystick button 310, the joystick button 310 can rotate in a first direction, and when a user presses the bottom portion of the joystick button 310, it can rotate in a second direction other than the first direction.

[0041] When the user presses the joystick button 310 to rotate it in a first direction, the thyristor 355 can be controlled to be in a first state to allow power to be delivered to one or more electrical loads via the conductor 302. When the user presses the joystick button 310 to rotate it in a second direction, the thyristor 355 can be controlled to be in a second state to stop power delivery to one or more electrical loads via the conductor 302.

[0042] In some embodiments, the front panel 305 of the connectivity device 120 may include a first button 322 and a second button 324. As shown, in some embodiments, a Fresnel lens 326 may be disposed between the first button 322 and the second button 324. A user can interact with the first button 322 and the second button 324 to control various operations of the connectivity device 120.

[0043] In some implementations, the first button 322 may be a pairing button. More specifically, a user may interact with the first button 322 (e.g., press and / or pull out the first button 322) to initiate a pairing sequence with another device (such as an electrical load, another power switch, or user equipment). The pairing sequence may be used to enable communication between the connected device 120 and the other device. For example, the pairing sequence may be used to allow communication between the connected device 120 and the other device using a direct peer-to-peer communication protocol. Without departing from the scope of this disclosure, any of a variety of suitable interactions (e.g., user interaction sequences) via the first button may be used to initiate a pairing sequence.

[0044] In some embodiments, the second button 324 may be an air gap switch. User interaction with the air gap switch can be used to remove power from connected device 120 and / or one or more electrical loads. In some embodiments, the user can interact with the second button 324 by pulling it away from the front panel 305. The second button 324 may be associated with a long plunger arm such that when the second button 324 is pulled away from the front panel 105, power is removed to connected device 120 and one or more connected loads. In some embodiments, the user can interact with the second button 324 by pushing it towards the front panel 305. For example, the user can push the second button 324 towards the front panel 305 to perform one or more functions. As an example, one or more functions may include activating a digital voice assistant service (e.g., Alexa, Siri, Google, etc.).

[0045] Now for reference Figure 4 According to an exemplary embodiment of this disclosure, a block diagram of an example control system 400 for a connection device 120 is provided. The control system 400 may include one or more processors 440 and one or more memory devices 460. For example, the one or more processors 440 may include dual (e.g., two) processors. Alternatively, the one or more processors 440 may include four (e.g., four) processors.

[0046] One or more processors 440 may be any suitable processing device, such as a microprocessor, integrated circuit (e.g., application-specific integrated circuit), field-programmable gate array, etc., that performs operations to control components (e.g., any of the components described herein). One or more memory devices 460 may be any suitable medium for storing computer-readable instructions and data. For example, one or more memory devices 460 may include random access memory, such as dynamic random access memory (DRAM), static RAM (SRAM), or other volatile memory. Furthermore, and / or alternatively, one or more memory devices may include non-volatile memory, such as ROM, PROM, EEPROM, flash memory, optical storage, magnetic storage, etc.

[0047] One or more memory devices 460 may store computer-readable instructions that, when executed by one or more processors 440, cause the one or more processors 440 to perform operations, such as any of the operations described herein. The instructions may be software written in any suitable programming language or may be implemented in hardware.

[0048] One or more memory devices 460 may also store data that can be acquired, received, accessed, written, manipulated, created, and / or stored. As an example, one or more memory devices 460 may store data associated with one or more classifier models (e.g., machine learning classifier models) that can be used to classify data acquired from one or more sensor devices (e.g., microphones, power metering circuits, motion sensors) of the connected device 120. More specifically, the one or more classifier models may classify data as indicating the presence of a person or object in the space or not indicating the presence of a person or object in the space. Storing the classifier models locally in one or more memory devices 460 allows for local processing of data acquired from one or more sensor devices of the connected device 120.

[0049] Still referencing Figure 4One or more processors 440 may communicate with and / or be configured to control the operation of the audio circuitry system 430. The audio circuitry system 430 may be configured to receive and process audio data received from, for example, a first microphone 442 and a second microphone 444 of the connection device 120. The audio circuitry system 430 may also provide audio output to a speaker 432 of the connection device 120. In some embodiments, the audio circuitry system 430 may include one or more of a digital signal processor (DSP), a codec, an amplifier, etc. For example, the audio circuitry system 430 may be a low-power smart codec with a dual-core audio DSP. In some embodiments, the audio circuitry system 430 may include a CS47L24 smart codec with a dual-core DSP manufactured by Cirrus Logic.

[0050] In some embodiments, one or more processors 440 may communicate with and / or be configured to control the operation of the microcontroller 480. The microcontroller 480 may be configured to control the thyristor 355 and / or provide signals to one or more processors 440 for controlling components based on input received via interface elements of the connection device 120, such as a joystick button 310, a first button 322, a second button 324, or other interface elements. The microcontroller 480 may also receive signals from one or more motion sensors 482 of the connection device 120. In some embodiments, the one or more motion sensors 482 may include PIR sensors. However, it should be understood that the connection device 120 may include any suitable type of motion sensor. In some embodiments, one or more classifier models stored on one or more memory devices 460 of the connection device may be configured to classify data obtained from the one or more motion sensors 482 as indicating the presence of a person or object in the space or not indicating the presence of a person or object in the space.

[0051] In some embodiments, one or more processors 440 may communicate with ambient light sensor 446. Signals from ambient light sensor 446 may be used by processor(s)440, for example, to implement control actions based on ambient light in space (e.g., controlling power delivery to one or more electrical loads). In some embodiments, ambient light sensor 446 may be an LTR-329ALS-01 digital light sensor manufactured by Mouser Electronics.

[0052] One or more processors 440 may communicate with and / or be configured to control the operation of the power metering circuit 448. The power metering circuit 448 may be configured to measure the voltage and / or current flowing through the load line passing through the connection device 120. For example, a sensing resistor may be used to measure the current. For example, a voltage divider may be used to measure the voltage. The power flowing through the load line may be calculated based on the measured current and voltage (e.g., using one or more processors 440 located on and / or away from the connection device 120). In some embodiments, the power metering circuit 448 may be an STPM32 metering circuit system manufactured by STMicroelectronics.

[0053] In some implementations, data obtained from the power metering circuitry 448 of the connection device 120 may be provided as input to one or more classifier models stored on one or more memory devices 460 of the connection device 120. The one or more classifier models may process the data and generate metadata, at least in part, based on the data obtained from the power metering circuitry 448. The metadata may indicate the presence of a person or object in the space. In some implementations, the metadata may include a timestamp indicating when the presence of a person or object in the space was detected.

[0054] One or more processors 440 can communicate with LED driver circuitry 470 and LED board 472 to control the operation of indicators on connected device 120. LED driver circuitry 470 can provide power to LED board 472 to drive multiple LEDs. One or more processors 440 can control light emission from one or more LEDs on LED board 472 to provide various indicators (e.g., light rings, nightlights, etc.). In some embodiments, LED driver circuitry 470 can be an IS31FL3235 LED driver manufactured by Integrated Silicon Solution, Inc.

[0055] One or more processors 440 may communicate with communication interface 492. Communication interface 492 may allow data communication via, for example, one or more wireless links using antenna 495. Communication interface 492 may include any circuitry, components, software, etc., for communication via various communication links (e.g., networks). In some embodiments, communication interface 492 may include one or more of, for example, a communication controller, receiver, transceiver, transmitter, port, conductor, software, and / or hardware for transmitting data. In some embodiments, communication interface 492 may include an SX-SDPAC module manufactured by Silex Technology.

[0056] Example communication technologies and / or protocols may include, for example, Bluetooth Low Energy, Bluetooth Mesh Networking, Near Field Communication, Thread, TLS (Transport Layer Security), Wi-Fi (e.g., IEEE 802.11), Wi-Fi Direct (for peer-to-peer communication), Z-Wave, Zigbee, Halow, cellular communication, LTE, Low Power Wide Area Networking, VSAT, Ethernet, MoCA (Multimedia over Coax Alliance), PLC (Power-line communication), DLT (digital line transmission), etc. Other suitable communication technologies and / or protocols may be used without departing from the scope of this disclosure.

[0057] Now for reference Figure 5 According to an example embodiment of this disclosure, a system 500 for determining the location of a person within a space is provided. As shown, system 500 may include a plurality of connection devices 120 (only one is shown). The plurality of connection devices 120 may be located throughout the space, as referenced above. Figure 1 The discussion space is 100. The connection device 120 can communicate with a variety of devices. For example, in some embodiments, one of the connection devices 120 may be a power switch communicating with an electrical load 510. The electrical load 510 can be any device powered by the power switch, such as one or more lighting fixtures or other light sources, electrical appliances, electronic devices, consumer devices, ceiling fans, machines, systems, or any other suitable type of electrical load. Alternatively or additionally, the connection device 120 may communicate with user equipment 520, 560. For example, user equipment 520, 560 may include one or more smartphones, laptops, desktop computers, tablets, wearable devices, media devices, displays with one or more processors, or other suitable devices.

[0058] In some implementations, one or more of the connected devices 120 may communicate with the electrical load 510, for example, via a direct communication link (e.g., a direct wired or wireless communication link) or via a network such as a local area network 540. For example, a direct communication link may be implemented using Bluetooth Low Energy or other suitable communication protocols. One or more connected devices 120 may control power delivery to the electrical load 510 via a load conductor. In some implementations, one or more connected devices 120 may provide control signals via a direct communication link to control the operation of the electrical load 510 (e.g., fan speed, dimming level, etc.).

[0059] In some implementations, one or more of the connected devices 120 may communicate with user equipment 520, 560, for example, via a direct communication link (e.g., a direct wired or wireless communication link) or via a network such as local area network 540. For example, a direct communication link may be implemented using Bluetooth Low Energy or other suitable communication protocols. In some embodiments, a user may control, view information, and / or specify one or more settings associated with one or more connected devices 120 via a graphical user interface implemented on the display of user equipment 520, 560. For example, a user may access an application implemented on user equipment 520. This application may present a graphical user interface on the display of user equipment 520. The user may interact with the graphical user interface to control the operation of one or more connected devices 120 and / or electrical load 510.

[0060] Local area network 540 can be any suitable type of network or combination of networks that allows communication between devices. In some embodiments, the network(s) may include one or more of a secure network, a Wi-Fi network, an IoT network, a mesh network, one or more peer-to-peer communication links, and / or combinations thereof, and may include any number of wired or wireless links. Communication on local network 340 can be achieved, for example, by using a communication interface with any type of protocol, protection scheme, encoding, format, packetization, etc.

[0061] As shown, system 500 may include gateway 555, which allows access to wide area network 550. Wide area network 550 may be, for example, the Internet, a cellular network, or other networks, and may include any number of wired or wireless links. Communication on wide area network 550 may be implemented, for example, using a communication interface with any type of protocol, protection scheme, encoding, format, packetization, etc. As shown, connection device 120 can transmit information via wide area network 550 through gateway 555 to remote computing systems 580 and 590 and other remote computing devices.

[0062] In some implementations, the remote computing system 580 may be associated with a cloud computing platform to implement one or more services for the connected device 120. Data collected by the cloud computing platform may be processed, stored, and provided to, for example, user device 520 (e.g., for presentation in a graphical user interface).

[0063] In some implementations, the remote computing system 590 may be associated with services accessed by the connection device 120, such as a digital audio assistant service. In some implementations, audio data collected via one or more sensor devices may be transmitted to the remote computing system 590 for processing voice commands. Data in response to voice commands may be transmitted to the connection device 120 for output (e.g., via an output device) and / or transmitted to the user device 520 (e.g., for display in a graphical user interface). In this way, the connection device 120 can act as a source of voice commands for the digital voice assistant service.

[0064] Remote computing systems 580 and 590 may include one or more computing devices. One or more computing devices may include one or more processors and one or more memory devices. Remote computing systems 580 and 590 may be distributed, such that their components are located in different geographical regions. The techniques discussed herein refer to computer-based systems and actions taken by computer-based systems, as well as information sent to and from computer-based systems. Those skilled in the art will recognize that the inherent flexibility of computer-based systems allows for a wide variety of possible configurations, combinations, and partitions of tasks and functions among components. For example, the processes discussed herein may be implemented using a single computing device or multiple computing devices operating in a combined manner. Databases, memory, instructions, and applications may be implemented on a single system or distributed across multiple systems. Distributed components may operate sequentially or in parallel.

[0065] Now for reference Figure 6 According to an example embodiment of this disclosure, a flowchart of a method 600 for generating metadata based on data obtained from one or more connected devices in space is provided. It should be understood that method 600 can use the above-referenced... Figure 5 The system under discussion will be implemented in 500. Figure 6 For purposes of illustration and discussion, the steps performed in a specific order are depicted. Those skilled in the art will understand, using the disclosure provided herein, that the steps of method 600 can be adjusted, modified, rearranged, performed concurrently, or modified in various ways without departing from the scope of this disclosure.

[0066] In (602), method 600 may include obtaining data from one or more sensor devices associated with one or more connected devices located within the space by one or more computing devices of the computing system. For example, in some embodiments, the data may be audio data indicating one or more audible sounds obtained by one or more microphones of one or more connected devices. Alternatively or additionally, the data may indicate one or more electrical loads consuming power supplied from a power source (e.g., AC mains) via one of the connected devices within the space.

[0067] In (604), method 600 includes generating metadata indicating the location of a person or object within a space, at least in part based on the data obtained in (602), by one or more computing devices. In some embodiments, generating metadata indicating the location of a person or object may include, in (606), inputting the data obtained in (602) as input to a classifier model (e.g., a machine learning model) configured to classify the data as indicating the presence of a person or object within the space by one or more computing devices. The classifier model may be implemented using a machine learning model. For example, the machine learning model may include, but is not limited to, convolutional neural networks, decision trees, Bayesian networks, support vector machines, K-means clustering, etc. If the classifier model determines that the data obtained from one or more sensor devices indicates the presence of a person or object within the space, generating metadata may include, in (608), outputting metadata indicating the presence of a person or object as the output of the classifier model by one or more computing devices. Additionally, in some embodiments, the metadata may include a timestamp indicating when the presence of a person or object was detected.

[0068] In some implementations, method 600 may further include one or more computing devices associating metadata with video data obtained from one or more image capture devices in space. For example, the metadata may be associated with one or more frames of video data having timestamps corresponding to the timestamps associated with the metadata. In this way, one or more computing devices may be configured to search only one or more frames of video data associated with the metadata to determine the location of a person or object.

[0069] Now for reference Figure 7 According to an example embodiment of this disclosure, a flowchart of a method 700 for determining the position of a person or object within a space is provided. It should be understood that method 700 can use the above-referenced... Figure 5 The system 500 discussed is used for implementation. It should be understood that, in some embodiments, one or more steps of method 700 may be executed locally by the computing system of one or more connected devices. Alternatively, in some embodiments, one or more steps of method 700 may be executed by the above-referenced... Figure 5 The discussion focuses on remote computing systems for execution. Additionally, although... Figure 7 For purposes of illustration and discussion, the steps performed in a particular order are depicted. Using the disclosure provided herein, those skilled in the art will understand that the steps of method 700 can be adjusted, modified, rearranged, performed concurrently, or modified in various ways without departing from the scope of this disclosure.

[0070] In (702), method 700 may include obtaining data from one or more sensor devices associated with one or more connected devices located within the space by one or more computing devices of the computing system. For example, in some embodiments, the data may be audio data indicating one or more audible sounds obtained by one or more microphones of one or more connected devices. Alternatively or additionally, the data may indicate that one or more electrical loads consume power supplied from a power source (e.g., AC mains) via one of the connected devices within the space.

[0071] In (704), method 700 includes generating metadata indicating the location of a person or object within a space, at least in part based on the data obtained in (702), by one or more computing devices. In some embodiments, generating metadata indicating the location of a person or object may include inputting the data obtained in (702) into one or more computing devices as input to a classifier model (e.g., a machine learning model), which is configured to classify the data as indicating the presence of a person or object within the space. The classifier model may be implemented using a machine learning model. For example, the machine learning model may include, but is not limited to, convolutional neural networks, decision trees, Bayesian networks, support vector machines, K-means clustering, etc. If the classifier model determines that the data obtained from one or more sensor devices indicates the presence of a person or object within the space, generating metadata may include obtaining metadata indicating the presence of a person or object as the output of the classifier model by one or more computing devices. Additionally, in some embodiments, the metadata may include a timestamp indicating when the presence of a person or object was detected.

[0072] In (706), method 700 may include data obtained by one or more computing devices of a computing system indicating a user request to obtain the location of a person or object within a space. For example, the data indicating a user request may include audio data obtained via one or more sensor devices (e.g., a microphone) in one or more connected devices.

[0073] At (708), method 600 may include determining the location of a person or object in the space by one or more computing devices, at least in part, based on metadata generated from data obtained from one or more sensor devices of one or more connected devices in the space. In some embodiments, the metadata obtained at (604) may be associated with one or more frames of video data obtained from one or more image capture devices in the space. For example, the metadata may be associated with one or more frames of video data having timestamps corresponding to timestamps associated with the metadata and indicating when the presence of a person or object was detected. In this way, the amount of time required to determine the location of a person or object in the space can be reduced because the one or more computing devices only search for one or more frames of video data associated with the metadata generated at (704).

[0074] In some embodiments, one or more computing devices may be configured to perform one or more facial recognition techniques on one or more frames of video data to determine whether a person or object depicted in one or more frames of video data corresponds to a person or object associated with a user request obtained at (706). In some embodiments, one or more computing devices may generate metadata while performing one or more facial recognition techniques on one or more frames. For example, in some embodiments, the metadata generated while performing one or more facial recognition techniques may indicate the location of a person or object in space. In some embodiments, the metadata generated at (704) may be combined with the metadata generated while performing one or more facial recognition techniques on one or more frames to determine the location of a person or object in space. In this way, data obtained in multiple domains (e.g., audio, video, etc.) can be used to determine the location of a person or object in space.

[0075] In (710), method 700 may include a notification provided by one or more computing devices indicating the location of a person within a space. For example, the notification may be an auditory notification provided via one or more output devices (e.g., a speaker) of one or more of the connected devices. Alternatively or additionally, the notification may be provided to a user device (e.g., a smartphone, tablet, etc.) associated with the location of the requester. For example, the notification may be a visual notification (e.g., a text message, email). Alternatively or additionally, the notification may be an auditory notification, such as an automated telephone call.

[0076] In some embodiments, method 700 may further include determining one or more patterns indicating the movement of a person or object by one or more computing devices, at least in part, based on data obtained in (702) and metadata generated in (704). In this way, the one or more computing devices can learn the habits of a person or object that can be used to determine the location of the person or object. For example, the one or more computing devices can determine that a person or object is located in a room or area of ​​space on a specific date and / or time. In this way, in some embodiments, the location of a person or object can be determined by relying on one or more patterns determined by the one or more computing devices.

[0077] Figure 8 Suitable components of a computing system 800 according to an example embodiment of this disclosure are shown. It should be understood that the above references... Figure 2 and 5 At least one of the computing systems 220, 580, and 590 discussed can be configured as Figure 8 The computing system 800. As shown, the computing system 800 may include one or more processors 802 configured to perform various computer-implemented functions (e.g., performing the methods, steps, calculations, etc. disclosed herein). As used herein, the term "processor" refers not only to integrated circuits known in the art as contained in a computer, but also to controllers, microcontrollers, microcomputers, programmable logic controllers (PLCs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and other programmable circuits.

[0078] As shown, computing system 800 may include memory device 804. Examples of memory device 804 may include computer-readable media, including but not limited to non-transitory computer-readable media such as RAM, ROM, hard disk drives, flash drives, or other suitable memory devices. Memory device 804 may store information accessible by processor(s) 802, including computer-readable instructions 806 executable by processor(s) 802. Computer-readable instructions 806 may be operations that, when executed by processor(s) 802, cause processor(s) 802 to perform (such as those referenced above). Figure 6 and Figure 7The method discussed refers to any set of instructions. The computer-readable instructions 806 may be software written in any suitable programming language, or may be implemented in hardware. In some implementations, the computer-readable instructions 806 may be executed by processor(s) 802 to perform operations such as generating metadata based at least in part on data obtained from one or more sensing devices among one or more of a plurality of connected devices located in space.

[0079] In some implementations, the computing system 800 may include one or more classifier models 808. For example, the one or more classifier models 808 may include various machine learning models, such as random forest classifiers; logistic regression classifiers; support vector machines; one or more decision trees; neural networks; and / or other types of machine learning models, including both linear and nonlinear models. Example neural networks may include feedforward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), or other forms of neural networks.

[0080] In some implementations, the computing system 800 can train one or more classifier models 808 using a model trainer 810. The model trainer 810 can train one or more classifier models 808 using one or more training or learning algorithms. One example training technique is error backpropagation (“backpropagation”). For example, backpropagation can include Levenberg-Marquardt backpropagation. In some implementations, the model trainer 810 can perform supervised training techniques using a set of labeled training data. In other implementations, the model trainer 810 can perform unsupervised training techniques using a set of unlabeled training data. The model trainer 810 can perform various generalization techniques to improve the generalization ability of the trained model. Generalization techniques include weight decay, dropout, or other techniques.

[0081] Specifically, the model trainer 810 can train one or more classifier models 808 based on a set of training data 812. The training data 812 may include multiple training examples. Each training example may include example features labeled as indicating the presence of a person or object in the space. In some implementations, the features may include raw data from one or more sensor devices from one or more connected devices in the space.

[0082] While the subject matter has been described in detail with reference to specific exemplary embodiments of the invention, it should be understood that those skilled in the art, upon gaining an understanding of the foregoing, can readily generate modifications, variations, and equivalents of these embodiments. Therefore, the scope of this disclosure is illustrative and not restrictive, and the disclosure does not preclude the inclusion of such modifications, variations, and / or additions to the subject matter that would be obvious to those skilled in the art.

Claims

1. A method for determining the position of a person or object within a space, the method comprising: Data is obtained by one or more computing devices from one or more sensor devices, which are associated with one or more connectivity devices located within the space. The data obtained from the one or more sensor devices includes power data indicating the power consumption of electrical loads selectively coupled to a power source via the one or more connectivity devices. Data obtained from the one or more sensor devices is input to the one or more computing devices as input to a machine learning model, which is configured to classify the data input from the one or more computing devices based on whether the data indicates the presence of a person or object in the space; Metadata, obtained as the output of the machine learning model from the one or more computing devices, indicates the presence of a person or object within the space; The one or more computing devices associate the metadata with video data obtained from one or more image capture devices located within the space; The one or more computing devices obtain data indicating a user request, the user request being associated with obtaining the location of a person or object within the space; The location of the person or object is determined by the one or more computing devices by searching portions of the video data associated with the metadata; as well as The one or more computing devices provide a notification indicating the location of the person or object.

2. The method of claim 1, wherein the metadata includes a timestamp indicating when the presence of a person or object in the space was detected.

3. The method according to claim 1 or 2, further comprising: The one or more computing devices associate the metadata with one or more frames of video data obtained from one or more image capture devices located in the space.

4. The method of claim 3, wherein the timestamp associated with the one or more frames associated with the metadata corresponds to the timestamp associated with the metadata.

5. The method according to claim 1 or 2, wherein determining the position of a person or object comprises: The location of the person or object is determined by the one or more computing devices based at least in part on one or more frames of the video data.

6. The method of claim 1 or 2, wherein the data obtained from the one or more sensor devices includes audio data.

7. The method of claim 1 or 2, wherein providing a notification indicating the location of the person or object comprises providing the notification by the one or more computing devices via one or more output devices of the one or more connecting devices.

8. The method according to claim 1 or 2, wherein the notification includes at least one of an auditory notification and a visual notification.

9. The method according to claim 1 or 2, further comprising: The one or more computing devices determine, at least in part, one or more patterns indicating the movement of people or objects within the space based on the metadata.

10. The method of claim 9, wherein determining one or more patterns associated with movement of a person or object within the space comprises: The metadata is input from the one or more computing devices as input to the machine learning model; as well as Data indicating the one or more patterns is obtained by the one or more computing devices as the output of the machine learning model.

11. A system for determining the location of a person within a space, the system comprising: One or more connecting devices located within the space, the one or more connecting devices including one or more sensor devices; as well as One or more computing devices, the one or more computing devices being configured to: Data is obtained from the one or more sensor devices, including power data indicating the power consumption of an electrical load selectively coupled to a power source via the one or more connection devices; Data obtained from the one or more sensor devices is used as input to a machine learning model, which is configured to classify the data input from the one or more computing devices based on whether the data indicates the presence of a person or object in the space. Obtain metadata as the output of the machine learning model, the metadata indicating the presence of people or objects within the space; Associate the metadata with video data obtained from one or more image capture devices located within the space; Obtain data indicating a user request, the user request being associated with obtaining the location of a person or object within the space; The location of the person or object is determined by searching portions of the video data associated with the metadata; and Provide notification indicating the location of the person or object.

12. The system of claim 11, wherein the one or more connection devices include a power switch configured to selectively couple an electrical load to a power source.

13. The system of claim 11 or 12, wherein the one or more sensor devices include a power metering circuit configured to measure the power consumption of the electrical load.

14. The system of claim 11 or 12, wherein the one or more sensor devices include one or more microphones.

15. The system of claim 11 or 12, wherein the one or more sensor devices include one or more motion sensors.

16. The system of claim 11 or 12, wherein the one or more motion sensors comprise passive infrared (PIR) sensors.

17. The system of claim 11 or 12, wherein the one or more connection devices further comprises one or more output devices.

18. The system of claim 17, wherein the one or more computing devices are configured to provide notifications via the one or more output devices.

19. The system of claim 17, wherein the one or more output devices include one or more speakers.

Images