SYSTEMS, METHODS AND DEVICES FOR UTILIZING RADAR IN SMART DEVICES - Patent application

Abstract

To provide methods, devices, and systems that manage radar use and communication in smart devices that collect information, and / or communicate via radar.SOLUTION: A radar-enabled and radio-enabled electronic device with a controller and a memory receives a request to transmit or reflect position information using a radar technique from a remote electronic device by a standard wireless communication protocol, in which the request includes device identification information, determines whether the received device identification information matches device identification information about the electronic device, and transmits a signal indicating a position and / or a speed of the electronic device in accordance with the use of one or more radar techniques, in accordance with a determination that the received device identification information matches the device identification information about the electronic device.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] This application relates generally to computer technology, including, but not limited to, methods and systems for utilizing radar communications with smart devices. [Background technology]

[0002] background Devices in a smart home environment contain numerous circuit components and interfaces to enable communication with other systems, devices and / or servers. Some smart devices contain multiple radios within a compact area to send and receive signals at various wavelengths to and from other devices over a network. For example, some smart devices collect information and / or communicate via radar. Summary of the Invention [Problem to be solved by the invention]

[0003] overview Therefore, there is a need for a method, apparatus and system for managing radar usage and communication. Various implementations of the systems, methods and devices within the scope of the appended claims each have several aspects, none of which are solely responsible for the attributes described herein. Without limiting the scope of the appended claims, after reviewing this disclosure, and especially after reviewing the paragraph entitled "Detailed Description", one will understand how aspects of the various implementations can be used to manage radar in a smart device. In one aspect, an electronic tag includes a first circuit configured to communicate with one or more other devices at a first frequency and a second circuit configured to communicate with the one or more other devices via the radar. [Means for solving the problem]

[0004] In some implementations, a location beacon (sometimes referred to as a location tag) is used to provide a unique signature to the radar subsystem, in some implementations the signature is encoded by transmitting the unique signature in the same frequency band as the radar system.

[0005] Radar systems detect the presence, direction, distance and / or speed of objects. Radar offers advantages over other detection systems such as passive infrared (PIR) motion sensors. Radar systems detect a wide range of speeds, which allows them to distinguish between separate moving objects. For example, a radar system can distinguish between a pet and a person in a home. In addition, a radar system can detect whether an object is moving toward or away from (or tangentially) the radar system. Radar systems can detect subtle human motions such as a heartbeat. Also, it is more difficult to avoid detection with a radar system compared to traditional heat detection systems. For example, a heavy winter coat may be enough to avoid detection with some heat detection systems.

[0006] With respect to PIR sensors, radar systems can generally detect objects at greater distances than PIR sensors. Additionally, radar systems do not require lenses and have a wider field of view than PIR sensors. Radar systems are generally not as sensitive to heat as PIR sensors. For example, PIR sensors can be affected by heat, sunlight, or ambient light. The operation of these components may cause false positives.

[0007] Radar systems can also penetrate walls and other objects. Thus, a radar system can monitor multiple rooms in a residence and detect movement behind various objects in the residence. Thus, a radar system allows communication between rooms through objects. Radar systems can also detect and interpret certain gestures.

[0008] Unlike traditional camera systems, radar systems operate independently of the level of visible light in a room or residence. In addition, radar systems can be configured to consume less power by adjusting the duty cycle and / or by sending out short pulses at set intervals.

[0009] Radar systems can be one-dimensional (1D) or multi-dimensional. A one-dimensional radar consists of one transmitter and one receiver. A multi-dimensional radar contains multiple transmitters and / or multiple receivers (e.g., three transmitters and four receivers).

[0010] Radar systems can be used for multiple purposes including proximity detection, occupancy determination, person count, location determination, single person breath monitoring, motion event classification, multiple person breath monitoring, single person identification, and multiple person identification. For some purposes, such as proximity and / or single person breath monitoring, a 1D radar system can be as effective (or nearly effective) as a multidimensional radar system. For other purposes, such as location determination and multiple person breath monitoring, a multidimensional radar system provides significantly greater precision and / or accuracy. In some implementations, multiple 1D radar systems are networked together to provide precision and accuracy as high or higher than a single multidimensional radar system.

[0011] An electronic device, such as a home electronic device, is provided. The electronic device comprises a communication module including one or more antennas and configured to communicate with one or more remote devices, a radar module configured to generate radar data for an environment, such as a home environment, and one or more processors configured to determine a location and / or movement of an object in the residence by analyzing the radar data. In some examples, anomalies associated with the location and / or movement of the detected object may be determined. These anomalies may include determining that the object is moving in an unusual manner or may include an object that should not be present in the environment or part of the environment. The anomalies may include one or more of an unexpected speed, an unexpected acceleration, and an unexpected location. Determining whether these factors are unexpected may be performed by comparing to stored data, such as historical data, average data, or median data, for objects found in the residence. In some examples, the radar module may include one or more low-power radar transmitters. The term "low" refers to power suitable for home use, details of which are described in the detailed description below. The radar module may additionally include one or more radar receivers. The electronic device further includes one or more circuit boards and a housing that accommodates the one or more circuit boards. The communications module may be coupled to the one or more circuit boards, the radar module may be coupled to the one or more circuit boards, and the one or more processors may be mounted on the one or more circuit boards. The one or more processors may be configured to manage operation of the communications module and the radar module and / or to detect anomalies associated with the detected objects based at least in part on the determined position and / or movement of the detected objects. The electronic device further includes one or more processors coupled to the communications module and the radar module. The power supply may include a

[0012] The radar module may be positioned to have a field of view that extends through the housing, for example through the front face of the housing.

[0013] The electronic device may be a smart home electronic device, such as selected from the group consisting of: a smart thermostat; a smart camera; a hub device; a smart hazard detector; a smart irrigation device; a smart media playback device; a smart hallway interface device; a smart appliance; and a smart security device. The term "smart" refers to a device that is connected to the Internet directly, indirectly, or via the Internet, such as wirelessly, wired, etc. IEEE 802.11b means a device that is capable of communicating with other devices via a wireless LAN.

[0014] Detecting an anomaly associated with the detected object may include determining that the detected object has one or more of an unexpected speed, an unexpected acceleration, and an unexpected position. The unexpected speed, unexpected acceleration, and unexpected position may be determined by comparing with expected position and movement information for objects in the Home environment based on historical data for the Home environment.

[0015] Other aspects, such as those described in the detailed description below, may be added or combined with the above examples.

[0016] BRIEF DESCRIPTION OF THE DRAWINGS For a better understanding of the various implementations described, reference should be made to the following description of implementations in connection with the accompanying drawings, in which like reference characters refer to corresponding parts throughout the drawings. [Brief description of the drawings]

[0017] [Figure 1] FIG. 1 illustrates an exemplary smart home environment, according to some implementations. [Diagram 2] FIG. 1 is a block diagram illustrating an example network architecture including a smart home network, according to some implementations. [Diagram 3]2 is a network-level diagram illustrating an extensible device and service platform into which the smart home environment of FIG. 1 is integrated, according to some implementations. [Figure 4] 4 is an abstracted functional diagram illustrating the extensible device and service platform of FIG. 3 in relation to devices and processing engines in a smart home environment according to some implementations. [Diagram 5] FIG. 1 illustrates a representative operating environment in which a server system interacts with client devices and a hub device communicatively coupled to a local smart device, according to some implementations. [Figure 6] 1 is a block diagram illustrating a representative hub device according to some implementations. [Figure 7A] FIG. 1 is a block diagram illustrating a representative server system according to some implementations. [Figure 7B] FIG. 2 illustrates various data structures used by some implementations. [Figure 8] 1 is a block diagram illustrating a representative client device associated with a user account according to some implementations. [Figure 9A] 1 is a block diagram illustrating a representative smart device according to some implementations. [Figure 9B] 1 is a block diagram illustrating a representative electronic tag, according to some implementations. [Figure 10] FIG. 1 is a block diagram illustrating a representative smart home provider server system according to some implementations. [Figure 11A] FIG. 1 illustrates an exemplary system architecture for video analysis and categorization, according to some implementations. [Figure 11B] FIG. 1 illustrates an exemplary processing pipeline for video analysis and categorization, according to some implementations. [Figure 12A] FIG. 1 illustrates an environment and system for communicating via radar signals, according to some implementations. [Figure 12B] FIG. 1 illustrates an exemplary system and process for utilizing radar tags, according to some implementations. [Figure 12C] FIG. 1 illustrates an exemplary system and process for utilizing radar tags, according to some implementations. [Figure 12D] 1 is a diagram of radar data projections according to some implementations. [Figure 13A] FIG. 1 illustrates an environment and system for utilizing data from multiple radar devices, according to some implementations. [Figure 13B] FIG. 1 illustrates an exemplary system and process for utilizing radar data from multiple radar-equipped devices, according to some implementations. [Figure 14A] FIG. 1 is a block diagram illustrating a representative radar system according to some implementations. [Figure 14B] FIG. 1 is a block diagram illustrating a representative radar system according to some implementations. [Figure 14C] FIG. 1 is a block diagram illustrating a representative radar system according to some implementations. [Figure 14D] FIG. 2 is a block diagram illustrating a representative radar control module, according to some implementations. [Figure 14E] 1 is a block diagram illustrating a representative application processor according to some implementations. [Figure 15A] 1A-1C are various assembly views of a camera device 118 according to some implementations. [Figure 15B] 1A-1C are various assembly views of a camera device 118 according to some implementations. [Figure 15C] 1A-1C are various assembly views of a camera device 118 according to some implementations. [Figure 15D] 1A-1C are various assembly views of a camera device 118 according to some implementations. [Figure 15E] 1A-1C are various assembly views of a camera device 118 according to some implementations. [Figure 15F] 1A-1C are various assembly views of a camera device 118 according to some implementations. [Figure 15G] 1A-1C are various assembly views of a camera device 118 according to some implementations. [Figure 15H] 1A-1C are various assembly views of a camera device 118 according to some implementations. [Figure 15I] 1A-1C are various assembly views of a camera device 118 according to some implementations. [Figure 15J] 1A-1C are various assembly views of a camera device 118 according to some implementations. [Figure 15K] 1A-1C are various assembly views of a camera device 118 according to some implementations. [Figure 15L] 1A-1C are various assembly views of a camera device 118 according to some implementations. [Figure 15M] 1A-1C are various assembly views of a camera device 118 according to some implementations. [Figure 16A] 2 is a diagram of various components of a thermostat according to some implementations. [Figure 16B] 2 is a diagram of various components of a thermostat according to some implementations. [Figure 16C] 2 is a diagram of various components of a thermostat according to some implementations. [Figure 17A] 2 is a diagram of various components of a hazard detector according to some implementations. [Figure 17B] 2 is a diagram of various components of a hazard detector according to some implementations. [Figure 17C] 2 is a diagram of various components of a hazard detector according to some implementations. [Figure 17D] 2 is a diagram of various components of a hazard detector according to some implementations. [Figure 18] FIG. 1 is a block diagram illustrating communication modules of a smart device utilizing bypass coexistence technology, according to some implementations. [Figure 19A] FIG. 1 illustrates a coexistence scheme for smart devices with multiple radios in a smart home environment, according to some implementations. [Figure 19B] FIG. 1 illustrates a coexistence scheme for smart devices with multiple radios in a smart home environment, according to some implementations. [Figure 20A] FIG. 1 is a component diagram illustrating a representative radar device, according to some implementations. [Figure 20B] FIG. 1 illustrates a home environment utilizing a representative radar device, according to some implementations. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Like reference numerals refer to corresponding parts throughout the several views of the accompanying drawings.

[0019] Explanation of the implementation example Reference will now be made in detail to implementations, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various implementations being described. However, it will be apparent to those skilled in the art that the various implementations being described may be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail to avoid unnecessarily obscuring aspects of the implementations.

[0020] 1 is an exemplary smart home environment 100 according to some implementations. The smart home environment 100 includes a structure 150 (e.g., a house, an office building, a garage, or a mobile home) with various integrated devices. It will be appreciated that devices may also be integrated into the smart home environment 100 that does not include the entire structure 150, such as an apartment building, a condominium, or an office space. Additionally, the smart home environment 100 may control and / or be coupled to devices outside the actual structure 150. In fact, some devices in the smart home environment 100 need not be physically located within the structure 150. For example, devices controlling the pool heater 114 or the irrigation system 116 may be located outside the structure 150.

[0021] The illustrated structure 150 includes multiple rooms 152 that are at least partially separated from one another by walls 154. The walls 154 may include interior or exterior walls. Each room may further include a floor 156 and a ceiling 158. The devices may be mounted to, affixed to, integrated with, and / or supported by the walls 154, floor 156, or ceiling 158. In some implementations, electronic tags are affixed to the walls 154, floor 156, ceiling 158, windows, or doors.

[0022] In some implementations, the integrated devices of the smart home environment 100 include intelligent, multi-sensor, network-connected devices seamlessly integrated with each other and / or with a central server or cloud computing system in a smart home network (e.g., 202 (FIG. 2)) to provide a variety of useful smart home functions. The smart home environment 100 may include one or more intelligent, multi-sensor, network-connected thermostats 102 (hereinafter referred to as “smart thermostats 102”), one or more intelligent, network-connected, multi-sensor hazard detection units 104 (hereinafter referred to as “smart hazard detectors 104”), one or more intelligent, multi-sensor, network-connected hallway interface devices 106 and 120 (hereinafter referred to as “smart doorbells 106” and “smart door locks 120”), and one or more intelligent, multi-sensor, network-connected alarm systems 122 (hereinafter referred to as “smart alert systems 122”).

[0023] In some implementations, one or more smart thermostats 102 detect ambient climate characteristics (e.g., temperature and / or humidity) and accordingly control the HVAC system 103. For example, each smart thermostat 102 includes an ambient temperature sensor.

[0024] One or more smart hazard detectors 104 may include a thermal radiation sensor pointed at a respective heat source (e.g., stove, oven, other appliance, fireplace, etc.). For example, the smart hazard detector 104 in the kitchen 153 includes a thermal radiation sensor pointed at the stove / oven 112. The thermal radiation sensor may determine the temperature of the respective heat source (or portion thereof) at which it is pointed and provide corresponding blackbody radiation data as an output.

[0025] The smart doorbell 106 and / or smart door lock 120 may detect the approach and / or departure of a person to a location (e.g., an exterior door), may control doorbell / doorlock functions (e.g., receive user input from the portable electronic device 166-1 to activate the bolt of the smart door lock 120), may announce the approach or departure of a person by audio or visual means, and / or may control settings on a security system (e.g., activate or deactivate the security system when a resident leaves and returns).

[0026] The smart alarm system 122 may detect the presence of an individual in close proximity (e.g., using a built-in IR sensor), sound an alarm (e.g., via a built-in speaker or by sending a command to one or more external speakers), and send notifications to entities or users inside / outside the smart home network 100. In some implementations, the smart alarm system 122 also includes one or more input devices or sensors (e.g., keypad, biometric scanner, NFC transceiver, microphone) for verifying the identity of the user, and one or more power devices (e.g., digital In some implementations, the smart alarm system 122 may also be set to an "armed" mode such that an alarm will sound upon detection of a triggering condition or event unless a disarming action is taken.

[0027] In some implementations, the smart home environment 100 includes one or more intelligent, multi-sensor, network-connected wall switches 108 (hereinafter referred to as "smart wall switches 108") along with one or more intelligent, multi-sensor, network-connected wall plug interfaces 110 (hereinafter referred to as "smart wall plugs 110"). The smart wall switches 108 may detect ambient lighting conditions, detect occupancy, and control the power and / or dimming state of one or more lights. In some cases, the smart wall switches 108 may control the power state or speed of a fan, such as a ceiling fan. The smart wall plugs 110 may detect occupancy in a room or enclosure and control the supply of power to one or more wall plugs (e.g., such that no power is supplied to the plug when no one is in the home).

[0028] In some implementations, the smart home environment 100 of FIG. 1 includes a number of intelligent, multi-sensor, network-connected appliances 112 (hereafter referred to as “smart appliances 112”), such as refrigerators, stoves, ovens, televisions, washers, dryers, lights, stereos, intercom systems, garage door openers, floor fans, ceiling fans, wall mounted air conditioners, pool heaters, irrigation systems, security systems, space heaters, window AC units, powered duct vents, etc. In some implementations, when plugged into an outlet, the appliances can inform the smart home network about themselves, such as by indicating what type of appliance they are, and automatically integrate with the smart home control devices. Such communication by the appliances to the smart home can be facilitated by wired or wireless communication protocols. The smart home may include a variety of non-communicating conventional appliances 140, such as older conventional washers / dryers, refrigerators, etc., that can be controlled by the smart wall plug 110. The smart home environment 100 may further include various partially-communicating conventional appliances 142, such as infrared ("IR") controlled wall mounted air conditioners or other IR controlled devices, that may be controlled by IR signals provided by the smart hazard detector 104 or smart wall switch 108.

[0029] In some implementations, the smart home environment 100 includes one or more network-connected cameras 118 configured to provide video surveillance and security in the smart home environment 100. The cameras 118 may be used to determine occupancy in the structure 150 and / or in a particular room 152 in the structure 150, and thus may function as a presence sensor. For example, video captured by the cameras 118 may be processed to identify the presence of an occupant in the structure 150 (e.g., in a particular room 152). A particular individual may be identified, for example, based on appearance (e.g., height, face) and / or movement (e.g., walking / gait). The cameras 118 optionally include one or more sensors (e.g., IR sensors, radar systems, motion detectors), input devices (e.g., microphones for capturing audio), and power devices (e.g., speakers for outputting audio).

[0030] The smart home environment 100 may additionally or alternatively include one or more other motion sensors (e.g., smart doorbell 106, smart door lock 120, touch screen, IR sensor, microphone, ambient light sensor, motion detector, smart night light 170, etc.). In some implementations, the smart home environment 100 uses radio frequency (RFID) sensors to determine occupancy based on an RFID tag placed on or embedded in the occupant. 10. The smart hazard detector 104 may include a radio-frequency identification (RFID) reader (e.g., in each room 152 or part thereof). For example, the RFID reader may be integrated into the smart hazard detector 104.

[0031] The smart home environment 100 may also include communications with devices outside the physical home but within the immediate geographic range of the home. For example, the smart home environment 100 may include a pool heater monitor 114 that communicates the current pool temperature to other devices in the smart home environment 100 and / or receives commands to control the pool temperature. Similarly, the smart home environment 100 may include an irrigation monitor 116 that communicates information regarding an irrigation system within the smart home environment 100 and / or receives control information for controlling such irrigation system.

[0032] In some implementations, the smart home environment 100 includes one or more electronic tags configured to communicate with one or more smart devices via radar. In some implementations, the electronic tags are affixed to objects, such as windows, doors, or walls, and configured to provide a radar signature for the objects. In some implementations, the electronic tags are affixed to entities, such as pets, and configured to provide a radar signature for the entities. In some implementations, the electronic tags are configured to communicate via multiple wavelengths and / or protocols. For example, certain electronic tags are configured to communicate via RFID as well as radar. In some implementations, a smart device, such as any of the smart devices described above, includes a radar module for detecting the presence, direction, distance, and / or speed of an object by sending out pulses of high frequency electromagnetic waves that reflect off the object and return to the source. In some implementations, the smart device further includes a communication module, distinct from the radar module, for communicating with other smart devices and / or electronic tags (e.g., via RFID, Wi-Fi, Bluetooth, etc.).

[0033] Through network connectivity, one or more of the smart home devices of FIG. 1 may further enable a user to interact with the device even when the user is not in the vicinity of the device. For example, a user may communicate with the device using a computer (e.g., a desktop computer, laptop computer, or tablet) or other portable electronic device 166 (e.g., a mobile phone such as a smartphone). A web page or application may be configured to receive communication from the user, control the device based on the communication, and / or present information regarding the operation of the device to the user. For example, a user may view the current setpoint temperature for a device (e.g., a stove) and adjust it using a computer. The user may be inside or outside the structure during this remote communication.

[0034] As described above, users may control smart devices in the smart home environment 100 using network-connected computers or portable electronic devices 166. In some examples, some or all of the residents (e.g., individuals living in a home) may register their devices 166 with the smart home environment 100. Such registration may be performed at a central server to authenticate the resident and / or device as associated with the home and to grant the resident permission to use the device to control smart devices in the home. A resident may also remotely control smart devices in the home using his / her registered device 166, such as when the resident is at work or on vacation. A resident may also use a registered device to control smart devices when the resident is physically present in the home, such as when the resident is sitting on a couch in the home. Instead of or in addition to registering devices 166, a smart home environment may also be configured to register its devices 166. It should be appreciated that the environment 100 can infer as to which individuals live in the home, and thus are resident, and which devices 166 are associated with those individuals. Thus, the smart home environment "learns" who the residents are and allows the devices 166 associated with each individual to control the smart devices of the home.

[0035] In some implementations, in addition to having processing and sensing capabilities, devices 102, 104, 106, 108, 110, 112, 114, 116, 118, 120, and / or 122 (collectively referred to as "smart devices") can communicate data and share information with other smart devices, a central server or cloud computing system, and / or other networked devices. Data communication can be performed using a variety of custom or standard wireless protocols (e.g., IEEE802.15.4, Wi-Fi, ZigBee, 6LoWPAN, Thread, Z-Wave, Bluetooth Smart, ISA100.11a, WirelessHART, MiWi, etc.) and / or a variety of custom or standard wired protocols (Ethernet, HomePlug, etc.), or other suitable communication protocols, including communication protocols not yet developed as of the filing date of this specification.

[0036] In some implementations, the smart devices communicate via radar. In some implementations, the smart devices communicate with one or more electronic tags via radar. In some implementations, the smart devices communicate with each other and / or with the electronic tags using multiple communication channels, such as RFID and radar. In some implementations, the smart devices communicate via one or more radio frequency bands, such as 3 GHz to 10 GHz, 24 GHz to 24.5 GHz, 57 GHz to 64 GHz, and / or 77 GHz to 81 GHz.

[0037] In some implementations, the smart devices function as wireless or wired routers. In some implementations, a first one of the smart devices communicates with a second one of the smart devices via a wireless router 160. The smart devices may further communicate with each other via a connection to a network (network interface 160), such as the Internet 162. Through the Internet 162, the smart devices may communicate with a smart home provider server system 164 (also referred to herein as a central server system and / or a cloud computing system). The smart home provider server system 164 may be associated with a manufacturer, support entity, or service provider associated with the smart device. In some implementations, a user may contact customer support using the smart device itself, without using other communication means such as a telephone or an Internet-connected computer. In some implementations, software updates may be automatically sent to the smart device from the smart home provider server system 164 (e.g., when available, when purchased, or at regular intervals).

[0038] In some implementations, the network interface 160 includes a conventional network device (e.g., a router), and the smart home environment 100 of FIG. 1 includes a hub device 180 that is communicatively coupled to the network 162 directly or via the network interface 160. The hub device 180 is further communicatively coupled to one or more of the intelligent, multi-sensor, network-connected devices (e.g., smart devices in the smart home environment 100) described above. Each of these smart devices may optionally include at least one of the following devices available in the smart home environment 100: The hub device 180 communicates with the hub device 180 using one or more wireless communication networks (e.g., ZigBee, Z-Wave, Insteon, Bluetooth, Wi-Fi, and other wireless communication networks) that are capable of communicating with the hub device. In some implementations, the hub device 180 and devices coupled / connected to the hub device can be controlled and / or interacted with via applications (sometimes referred to as smart home applications) running on a smartphone, home controller, laptop, tablet computer, game console, or similar electronic device. In some implementations, a user of such controller applications can view the status of the hub device or connected smart devices, configure the hub device to interact with a newly introduced smart device in the home network, authorize the new smart device, adjust or view settings of connected smart devices, etc. In some implementations, the hub device is configured to extend the capabilities of lower performance smart devices to match the capabilities of higher performance smart devices of the same type, integrate the capabilities of multiple different device types even across different communication protocols, and streamline the addition of new devices and the authorizing of the hub device. In some implementations, the hub device 180 further includes a local storage device for storing data related to or output by the smart devices of the smart home environment 100. In some implementations, the data includes one or more of video data output by the camera device, metadata output by the smart devices, setting information for the smart devices, usage logs for the smart devices, etc.

[0039] In some implementations, the smart home environment 100 includes a local storage device for storing data related to or output by smart devices of the smart home environment 100. In some implementations, the data includes one or more of video data output by a camera device (e.g., camera 118), metadata output by a smart device, configuration information for a smart device, usage logs for a smart device, etc. In some implementations, the local storage device is communicatively coupled to one or more smart devices via a smart home network (e.g., smart home network 202 (FIG. 2)). In some implementations, the local storage device is selectively coupled to one or more smart devices via a wired and / or wireless communication network. In some implementations, the local storage device is used to store video data when external network conditions are poor. For example, the local storage device is used when the encoding bit rate of the camera 118 exceeds the available bandwidth of the external network (e.g., network 162). In some implementations, the local storage device temporarily stores video data from one or more cameras (e.g., camera 118) before transferring the video data to a server system (e.g., server system 508 (FIG. 5)). In some implementations, the local storage device is a component of a camera device. In some implementations, each camera device includes local storage. In some implementations, the local storage device performs some or all of the data processing described below in association with server system 508 (FIG. 7A). In some implementations, the local storage device stores some or all of the data described below in association with server system 508, such as data storage database 7160, account database 7162, device information database 7164, and event information database 7166.In some implementations, the local storage device performs some or all of the operations described herein in association with the server system 508 .

[0040] It should be appreciated that a "smart home environment" may refer to a smart environment for a home, such as a single-family home, but the scope of the present teachings is not so limited. The present teachings are applicable to, but not limited to, duplexes, townhomes, multi-unit apartment buildings, hotels, retail stores, office buildings, industrial buildings or other structures, and more generally any living or working space.

[0041] It should also be recognized that terms such as user, customer, installer, homeowner, resident, guest, tenant, renter, renovator, and the like may be used to refer to a person or people performing an action in the context of any particular situation described herein, but that these references do not limit the scope of the present teachings with respect to the person performing such action. Thus, for example, the terms user, customer, purchaser, installer, subscriber, and homeowner may often refer to the same person in the case of a single-family residence, since the head of the household is the person who makes the purchase decision, purchases the unit, installs and configures the unit, and is also one of the users of the unit. However, in other cases, such as a homeowner-tenant environment, the customer may be the homeowner with respect to the purchase of the unit, the installer may be the local apartment block supervisor, the first user may be the tenant, and the second user may be the homeowner with respect to the remote control function as well. Although the identities of the persons performing the operations may be germane to particular advantages provided by one or more implementations, such identities in the following description should not be construed as necessarily limiting the scope of the present teachings to particular individuals having those particular identities.

[0042] 2 is a block diagram illustrating an example network architecture 200 including a smart home network 202 according to some implementations. In some implementations, smart devices 204 (e.g., devices 102, 104, 106, 108, 110, 112, 114, 116, 118, 120, and / or 122) in the smart home environment 100 are combined with a hub device 180 to create a mesh network in the smart home network 202. In some implementations, one or more of the smart devices 204 in the smart home network 202 act as a smart home controller. Additionally and / or alternatively, the hub device 180 acts as the smart home controller. In some implementations, the smart home controller has more computing power than the other smart devices. In some implementations, the smart home controller processes inputs (e.g., from the smart devices 204, the electronic devices 166, and / or the smart home provider server system 164) and sends commands (e.g., to the smart devices 204 in the smart home network 202) to control the operation of the smart home environment 100. In some implementations, some of the smart devices 204 in the smart home network 202 (e.g., in a mesh network) are "spokesman" nodes (e.g., 204-1) and other smart devices 204 are "low power" nodes (e.g., 204-9). Some of the smart devices in the smart home environment 100 are battery powered, while other smart devices have a regular reliable power source, such as by connection to wiring (e.g., 120V line voltage lines) behind the walls 154 of the smart home environment. Smart devices with a regular reliable power source are referred to as "spokesman" nodes. These nodes are typically equipped with the capability to use wireless protocols that facilitate two-way communication with various other devices within the smart home environment 100 as well as with the smart home provider server system 164.In some implementations, one or more "spokesman" nodes act as smart home controllers, while battery-powered devices are "low-power" nodes. These nodes tend to be smaller than spokesman nodes and typically communicate simply using wireless protocols that require very little power, such as Zigbee, 6LoWPAN, or radar. That's all.

[0043] In some implementations, some low power nodes are not capable of two-way communication. These low power nodes transmit messages but cannot "listen." Therefore, other devices in the smart home environment 100, such as spokesman nodes, cannot transmit information to these low power nodes. In some implementations, some low power nodes are only capable of limited two-way communication. For example, other devices can only communicate with the low power nodes during certain periods of time.

[0044] As described above, in some implementations, smart devices function as low power nodes and spokesman nodes to form a mesh network within the smart home environment 100. In some implementations, each low power node in the smart home environment periodically sends out messages about what it senses, and other low power nodes in the smart home environment forward those messages in addition to sending their own messages, thereby transporting messages between nodes (i.e., between devices) throughout the smart home environment 202. In some implementations, a spokesman node in the smart home network 202, which may communicate using a relatively high power communication protocol such as IEEE 802.11, may switch to a relatively low power communication protocol such as IEEE 802.15.4 to receive these messages, convert them to the other communication protocol, and transmit the converted messages (e.g., using the relatively high power communication protocol) to other spokesman nodes and / or the smart home provider server system 164. Thus, low power nodes using the low power communication protocol can send and / or receive messages to the smart home provider server system 164 throughout the smart home environment 202 and even over the Internet 162. In some implementations, the mesh network enables the smart home provider server system 164 to periodically receive data from many or all of the smart devices in the house, make inferences based on the data, facilitate state synchronization across devices inside or outside the smart home network 202, and send commands to one or more of the smart devices to perform tasks in the smart home environment.

[0045] As described above, the spokesman nodes and some of the low power nodes can "listen." Thus, a user, other devices, and / or the smart home provider server system 164 can communicate control commands to the low power nodes. For example, a user may use an electronic device 166 (e.g., a smartphone) to send a command over the Internet to the smart home provider server system 164, which then relays the command to one or more spokesman nodes in the smart home network 202. The spokesman nodes can communicate the command using a low power protocol to the low power nodes throughout the smart home network 202, as well as to other spokesman nodes that did not receive the command directly from the smart home provider server system 164.

[0046] In some implementations, the smart night-light 170 (FIG. 1), an example of a smart device 204, is a low-power node. In addition to housing a light source, the smart night-light 170 houses an occupancy sensor, such as an ultrasonic or passive IR sensor, and an ambient light sensor, such as a photoresistor or a single pixel sensor that measures the light in the room. In some implementations, the smart night-light 170 is configured to activate the light source when the ambient light sensor detects that the room is dark and when the occupancy sensor detects that someone is in the room. In other implementations, the smart night-light 170 simply detects that the room is dark and activates the light source. The smart night-light 170 is configured to activate the light source when an ambient light sensor detects that the smart night-light 170 is in a room. Additionally, in some implementations, the smart night-light 170 includes a low-power wireless communication chip (e.g., a ZigBee chip) that periodically sends out messages regarding occupancy and the amount of light in the room, including instantaneous messages that occur simultaneously when the motion sensor detects that someone is in the room. As described above, these messages can be transmitted wirelessly (e.g., using a mesh network) between nodes (i.e., between smart devices) in the smart home environment 202 and even over the Internet 162 to the smart home provider server system 164.

[0047] Another example of a low power node is a battery operated smart hazard detector 104. These smart hazard detectors 104 are often deployed in areas without access to constant and reliable power and may include any number and type of sensors, such as smoke / flame / heat sensors (e.g., thermal radiation sensors), carbon monoxide / carbon dioxide sensors, occupancy / motion sensors, ambient light sensors, ambient temperature sensors, humidity sensors, etc. Additionally, the smart hazard detectors 104 optionally transmit messages corresponding to each respective sensor to other devices and / or to the smart home provider server system 164, such as by using a mesh network as described above.

[0048] Examples of spokesman nodes include smart doorbell 106, smart thermostat 102, smart wall switch 108, and smart wall plug 110. These devices are often located near and connected to a reliable power source and therefore may include more power-hungry components, such as one or more communication chips capable of bidirectional communication over a variety of protocols.

[0049] In some implementation examples, the smart home environment includes electronic tags 206 such as electronic tag 206-1 and electronic tag 206-2. In some implementation examples, the electronic tag 206 is a low-power node in the smart home network 202. In some implementation examples, the electronic tag 206 is not connected to an external power source. In some implementation examples, the electronic tag 206 is battery-powered. In some implementation examples, an electronic tag (e.g., electronic tag 206-1) can incorporate the energy used when operating the tag. For example, the electronic tag incorporates thermal energy, vibration energy, electromagnetic energy, and / or solar energy received.

[0050] In some implementation examples, the electronic tag 206 can "listen" via a first communication channel (e.g., RFID channel), but cannot send messages. In some implementation examples, the electronic tag 206 is a passive radar device. Passive radar devices include radar devices without a dedicated transmitter.

[0051] Passive radar devices include corner reflector devices and printed radar devices. Corner reflector devices are generally used to generate a strong radar echo from an object with a very low radar cross section (RCS). The corner reflector includes two or more conductive surfaces mounted intersecting (e.g., at exactly a 90-degree angle). The incident electromagnetic wave is backscattered by being reflected multiple times exactly in the direction from which they come. This results in a strong echo even for a small object with a low RCS.

[0052] In some implementation examples, the printed radar reflector includes simple aluminum fibers (e.g., a sheet of paper) that form a half-wave resonator within the object to be tracked. The radar reflection fibers have approximately the same diameter as paper fibers (typically 6.5 mm in length and 1.5 μm in diameter). The randomly oriented radar reflection fibers are readable and future identification Each tagged object has a unique backscatter pattern that can be stored in a database for later use. Ordered patterns can also be designed such that individual resonators are coupled or separated to whatever is likely to give the optimal backscatter pattern. When illuminated by a radar, the backscattered fields of view interact to create a unique interference pattern that allows one tagged object to be identified and distinguished from other tagged objects.

[0053] In some implementations, the electronic tag 206 is an active radar device that can transmit radio frequency tones or pulses independent of any received radio waves. In various implementations, the electronic tag 206 can reflect, amplify and / or modulate received radio waves. Active radar devices include single transistor devices, MEMS-based devices and mechanically gated (closed) devices.

[0054] In some implementations, the electronic tag 206 is configured to communicate via radar in response to an enabling command received via a communication channel (e.g., an RFID channel) from a smart device, such as smart device 204-6 in FIG. 2. In some implementations, the electronic tag 206 is configured to communicate via radar at specific intervals, such as intervals preset by the smart device. For example, the electronic tags 206-1 and 206-2 are configured by device 204-6 such that only one of the tags communicates via radar at any given time. In some implementations, the electronic tag 206 is configured to communicate via radar in response to detecting a change in the environment, such as the movement of an object to which the electronic tag is affixed. For example, in some implementations, the electronic tag 206 includes one or more of a humidity sensor, a temperature sensor, an accelerometer, a gyroscope, and / or an optical sensor. In this example, the tag is configured to communicate via radar in response to a change detected by one or more of the sensors.

[0055] In some implementations, the smart home environment 100 includes a service robot 168 (FIG. 1) configured to perform any of a variety of household chores in an autonomous manner.

[0056] As described above in connection with FIG. 1, in some implementations, the smart home environment 100 of FIG. 1 includes a hub device 180 that is communicatively coupled to the network 162 directly or via the network interface 160. The hub device 180 is further communicatively coupled to one or more of the smart devices using a wireless communication network available at least in the smart home environment 100. The communication protocols used by the wireless communication network include, but are not limited to, ZigBee, Z-Wave, Insteon, EuOcean, Thread, OSIAN, Bluetooth low energy, and the like. In some implementations, the hub device 180 not only converts data received from each smart device to meet the data format requirements of the network interface 160 or the network 162, but also converts information received from the network interface 160 or the network 162 to meet the data format requirements of the respective communication protocol associated with the targeted smart device. In some implementations, in addition to data format conversion, the hub device 180 further processes data received from the smart devices or information previously received from the network interface 160 or the network 162. For example, hub device 180 may aggregate inputs from multiple sensors / connected devices (including sensors / devices of the same and / or different types) and perform higher level processing on those inputs (e.g., overall The hub device 180 may evaluate the overall environment and coordinate actions among various sensors / devices, and / or provide instructions to various devices based on a set of inputs and programmed processing. Note that in some implementations, the network interface 160 and the hub device 180 are integrated into one network device. The functionality described herein represents a specific implementation of a smart device, a control application running on a representative electronic device (such as a smartphone), the hub device 180, and a server coupled to the hub device via the Internet or other wide area network. All or part of the functionality and associated operations may be performed by any of the elements of the above-mentioned system, e.g., all or part of the functionality described herein as being performed by an implementation of the hub device may be performed in whole or in part on a server, on one or more connected smart devices and / or on the control application, or on various combinations thereof, in various system implementations.

[0057] FIG. 3 is a network level diagram illustrating an extensible device and service platform with which the smart home environment of FIG. 1 is integrated, according to some implementations. The extensible device and service platform 300 includes a smart home provider server system 164. Each of the intelligent network-connected devices described in connection with FIG. 1 (e.g., 102, 104, 106, 108, 110, 112, 114, 116, and 118, identified simply as “devices” in FIGS. 2-4) can communicate with the smart home provider server system 164. For example, a connection to the Internet 162 can be established directly (e.g., using a 3G / 4G connection to a wireless carrier), or through a network interface 160 (e.g., a router, switch, gateway, hub device, or an intelligent dedicated whole-home control node), or through any combination thereof.

[0058] In some implementations, the devices and services platform 300 communicates with and collects data from smart devices in the smart home environment 100. Additionally, in some implementations, the devices and services platform 300 communicates with and collects data from multiple smart home environments around the world. For example, the smart home provider server system 164 collects home data 302 from one or more smart home environment 100 devices, which may transmit home data periodically or in specific cases (e.g., when the device queries the home data 302). Examples of collected home data 302 include, but are not limited to, power consumption data, black body radiation data, occupancy data, HVAC settings and utilization data, carbon monoxide level data, carbon dioxide level data, volatile organic compound level data, sleep schedule data, cooking schedule data, interior and exterior temperature and humidity data, television ratings data, interior and exterior sound level data, pressure data, video data, and the like.

[0059] In some implementations, the smart home provider server system 164 provides one or more services 304 to smart homes and / or third parties. Examples of services 304 include, but are not limited to, software updates, customer support, sensor data collection / logging, remote access, remote or distributed control, and / or usage suggestions (based on the collected home data 202) for improving performance, reducing utility bills, improving safety, etc. In some implementations, data associated with the services 304 is stored in the smart home provider server system 164, which retrieves and transmits the data at appropriate times (e.g., periodically, upon receiving a request from a user, etc.).

[0060] In some implementations, the extensible device and services platform 300 includes a processing engine 306. The processing engine 306 may be centralized on a single server or may be distributed among several different computing entities, but is not limited to such. In some implementations, the processing engine 306 includes an engine configured to receive data from devices in the smart home environment 100 (e.g., via the Internet 162 and / or the network interface 160), index the data, analyze the data, and / or generate statistics based on or as part of the analysis. In some implementations, the analyzed data is stored as derived home data 308.

[0061] The results of the analysis or statistics can then be sent back to the device that provided the home data used to derive the results, to other devices, to a server that provides a web page to the user of the device, or to other non-smart device entities. In some implementations, usage statistics, usage statistics regarding the use of other devices, usage patterns, and / or statistics aggregating sensor readings may be generated and transmitted by the processing engine 206. These results or statistics may be provided over the Internet 162. In this manner, the processing engine 306 may be configured and programmed to derive a variety of useful information from the home data 302. A single server may include one or more engines.

[0062] The derived home data 308 can be used at a variety of different granularities for a variety of useful purposes, ranging from explicit programmed control of devices per home, neighborhood, or region (e.g., a demand response program for an electric utility), to generating inference abstractions that can assist on a per-home basis (e.g., an inference can be made that the homeowner has gone on vacation and therefore security detection equipment can be set to maximum sensitivity), to generating statistics and associated inference abstractions that can be used for governmental or charitable purposes. For example, the processing engine 306 can generate statistics regarding device usage among a population of devices and transmit the statistics to device users, service providers, or other entities (e.g., entities that requested the statistics or that offered monetary compensation for the statistics).

[0063] In some implementations, to foster innovation and research and increase the number of products and services available to users, the device and services platform 300 provides a set of application programming interfaces (APIs) 210 to third parties, such as charitable entities 314, government agencies 316 (among others). The home data 302 may be exposed to a wide variety of third parties, such as the smart home provider server system 164, which may include, for example, the Food and Drug Administration or the Environmental Protection Agency, academic institutions 318 (e.g., university researchers), businesses 320 (e.g., providing device warranties or services for related equipment or targeting advertisements based on the home data), gas and power companies 324, and other third parties. The API 310 may be coupled to the third party systems and may allow the third party systems to communicate with the smart home provider server system 164, including the services 304, the processing engine 306, the home data 302, and the derived home data 308. In some implementations, the API 310 may allow applications executed by the third parties to initiate certain data processing tasks performed by the smart home provider server system 164, as well as receive dynamic updates of the home data 302 and the derived home data 308.

[0064] For example, third parties may develop programs and / or applications (e.g., web or mobile applications) to integrate with the smart home provider server system 164 to provide services and information to users. Such programs and applications may, for example, enable users to reduce their energy consumption. The system may be designed to assist in proactively repairing defective equipment, preparing to meet high demand for service, tracking past service performance, etc., or performing other useful functions or tasks.

[0065] 4 is an abstracted functional diagram 400 illustrating the extensible device and services platform 300 of FIG. 3 in relation to the devices and processing engine 306 of a smart home environment, according to some implementations. Although the devices disposed within the smart home environment may have a wide variety of different individual capabilities and limitations, these devices each have a data consumer (DC) 402, a data source (DS) 404, a services consumer (SC) 406, and a data service (DS) 408. They share the common characteristic of being a service source (SS). Advantageously, in addition to providing control information used by devices to achieve local, direct objectives, the extensible device and services platform 300 may be configured to use the large amounts of data generated by these devices. In addition to improving or optimizing the actual operation of the devices themselves with respect to their direct functions, the extensible device and services platform 300 may be directed to "repurpose" that data in a variety of automated, extensible, flexible, and / or scalable ways to achieve a variety of useful objectives. These objectives may be predefined or adaptively identified, for example, based on usage patterns, device efficiency, and / or user input (e.g., requesting a particular function).

[0066] FIG. 4 illustrates the processing engine 306 as including several processing paradigms 410. In some implementations, the processing engine 306 includes a management service paradigm 410a that monitors and manages primary or secondary device functions. The device functions may include ensuring proper operation of the device given user input, inferring (e.g., responding when an intruder is present or attempting to enter the residence), detecting failures of appliances coupled to the device (e.g., a light bulb is out), implementing or responding to an energy demand response event, or alerting the user of current or predicted future events or characteristics. In some implementations, the processing engine 306 includes an advertising / communication paradigm 410b that infers target characteristics (e.g., demographic information), desires, and / or products for the user based on device usage. Services, promotions, products, or upgrades may then be suggested or automatically provided to the user. In some implementations, the processing engine 306 includes a social paradigm 410c. The social paradigm 410c uses information from the social network, provides information to the social network (e.g., based on device usage), and / or processes data associated with the user and / or device interactions with the social network platform. For example, the user's status as reported to trusted contacts on the social network may be updated to indicate when the user is at home based on light detection, security system outage, or device usage detector. As another example, a user may be able to share device usage statistics with other users. In yet another example, a user may share HVAC settings that can reduce their billed electricity, and other users may download the HVAC settings to their smart thermostat 102 to reduce their billed electricity.

[0067] In some implementations, the processing engine 306 includes a dispute / rule / compliance / reward paradigm 410d. The dispute / rule / compliance / reward paradigm 410d allows a user to communicate disputes, conflicts, rules, compliance regulations, and / or rewards. The objection may be to inform the participant of the objection and / or use the operational data to determine whether the objection has been addressed, whether the rule or regulation has been complied with, and / or whether a reward has been earned. The objection, rule and / or regulation may relate to initiatives for energy conservation, safe living (e.g., fewer heat source alarms, reduced exposure to toxic or carcinogenic substances), saving money and / or extending the life of equipment, improving health, etc. For example, one objection may require the participant to turn down the thermostat once a week. Participants who successfully fulfill this objection receive rewards such as coupons, virtual currency, status, etc. With respect to compliance, an example is a rental property owner making a rule that does not allow renters to access a particular owner's room. A device in the room with a motion sensor may send updates to the owner when the room is accessed.

[0068] In some implementations, the processing engine 306 integrates or utilizes exogenous information 412 from exogenous sources to improve the functionality of one or more processing paradigms. The exogenous information 316 may be used to interpret data received from the device, to determine characteristics of the environment near the device (e.g., outside the structure in which the device is enclosed), to determine services or products available to the user, to identify social networks or social network information, to determine contact information for entities near the device (e.g., public service entities such as emergency services, police, or hospitals), to identify statistical or environmental conditions, trends, or other information associated with the home or neighborhood, etc.

[0069] FIG. 5 shows a representative operating environment 500 in which a server system 508 performs data processing for one or more smart devices such as one or more cameras 118. In some implementations, the server system 508 monitors motion events in video streams captured by video cameras 118 to make them easier to review. In some implementations, the server system 508 monitors radar events detected by one or more radar-equipped smart devices to make them easier to review. As shown in FIG. 5, in some implementations, the server system 508 receives video data from video sources 522 (including cameras 118) located at various physical locations (e.g., homes, restaurants, stores, roads, parking lots, and / or within the smart home environment 100 of FIG. 1). Each video source 522 may be associated with one or more reviewer accounts, and the server system 508 provides video monitoring data about the video source 522 to client devices 504 associated with the reviewer accounts. For example, the portable electronic device 166 is an example of a client device 504.

[0070] In some implementations, the smart home provider server system 164 or its components function as the server system 508. In some implementations, the server system 508 includes a dedicated video processing server that provides video processing services to video sources and client devices 504 independent of other services provided by the server system 508. In some implementations, the server system 508 includes a dedicated radar processing server that provides radar processing services for various radar-equipped devices and client devices 504.

[0071] In some implementations, each of the video sources 522 includes one or more video cameras 118 that capture video and transmit the captured video to the server system 508 in substantially real-time. In some implementations, each of the video sources 522 optionally includes a controller device (not shown) that acts as an intermediary between the one or more cameras 118 and the server system 508. The controller device receives video data from the one or more cameras 118 and, optionally, performs some The controller device 504 performs preliminary processing and transmits the video data in substantially real-time to the server system 508 on behalf of one or more cameras 118. In some implementations, each camera has its own built-in processing capability to perform some preliminary processing on the captured video data before transmitting the processed video data (along with metadata obtained by the preliminary processing) to the controller device and / or the server system 508. In some implementations, the captured video is stored in local storage (not shown) before being uploaded to the server system 508.

[0072] 5, according to some implementations, each of the client devices 504 includes a client-side module 502. The client-side module 502 communicates with a server-side module 506 executing on a server system 508 via one or more networks 162. The client-side module 502 comprises client-side functionality for event monitoring and review processing and communicates with the server-side module 506. The server-side module 506 comprises server-side functionality for event monitoring and review processing for the several client-side modules 502 residing on the respective client device 504. The server-side module 506 also comprises server-side functionality for video processing and camera control for any number of control devices and any number of video sources 522, including cameras 118.

[0073] In some implementation examples, the server-side module 506 includes one or more processors 512, a video storage database 514, a device and account database 516, an I / O interface for one or more client devices 518, and an I / O interface for one or more video sources 520. The I / O interface for one or more clients 518 facilitates client-oriented input / output processing for the server-side module 506. The database 516 stores multiple profiles for reviewer accounts registered with the video processing server. In this case, each user profile includes account credentials for each reviewer account and one or more video sources linked to each reviewer account. The I / O interface for one or more video sources 520 facilitates communication with one or more video sources 522 (e.g., a group of one or more cameras 118 and related controller devices). The video storage database 514 stores not only various types of metadata such as motion events, event categories, event category models, event filters, and event masks for use in data processing for event monitoring and review for each reviewer account, but also unprocessed video data received from the video source 522.

[0074] Examples of representative client devices 504 include handheld computers, wearable computing devices, personal digital assistants (PDAs), tablet computers, laptop computers, desktop computers, mobile phones, smartphones, enhanced general packet radio service (EGPRS) mobile phones, media players, navigation devices , game consoles, televisions, remote controls, point-of-sale (POS) store sales information management The data processing devices include, but are not limited to, point-of-sale (POS) terminals, vehicle mounted computers, e-book readers, or any combination of two or more of these or other data processing devices.

[0075] Examples of the one or more networks 162 include local area networks (LANs) and wide area networks (WANs), such as the Internet. The one or more networks 162 may optionally include Ethernet , Universal Serial Bus (USB), FIREWIRE (registered (registered trademark), Long Term Evolution (LTE), Global Global System for Mobile Communications (GSM) ), Enhanced Data GSM Environment (EDGE), Code Division This may be accomplished using any known network protocol, including a variety of wired or wireless protocols, such as code division multiple access (CDMA), time division multiple access (TDMA), Bluetooth, Wi-Fi, voice over Internet Protocol (VoIP), Wi-MAX, or other suitable communication protocols.

[0076] In some implementations, the server system 508 is implemented on one or more stand-alone data processing devices or on a distributed network of computers. In some implementations, the server system 508 also uses various virtual devices and / or services of third party service providers (e.g., third party cloud service providers) to provide the underlying computing and / or infrastructure resources of the server system 508. In some implementations, the server system 508 includes, but is not limited to, a handheld computer, a tablet computer, a laptop computer, a desktop computer, or a combination of any two or more of these or other data processing devices.

[0077] The server-client environment 500 shown in FIG. 1 includes both a client-side portion (e.g., client-side module 502) and a server-side portion (e.g., server-side module 506). The division of functionality between the client and server portions of the operating environment 500 may vary from implementation to implementation. Similarly, the division of functionality between the video sources 522 and the server system 508 may vary from implementation to implementation. For example, in some implementations, the client-side module 502 is a thin client that simply provides input and output processing functions for a user and delegates all other data processing functions to a back-end server (e.g., server system 508). Similarly, in some implementations, each of the video sources 522 is a simple video capture device that continuously captures and feeds video data to the server system 508 without eliminating or limiting local preliminary processing of the video data. While many aspects of the technology are described in terms of the server system 508, the corresponding operations performed by the client device 504 and / or the video sources 522 will be apparent to one of ordinary skill in the art without any creative effort. Similarly, some aspects of the technology may be described in terms of a client device or a video source, and the corresponding operations performed by a video server will be apparent to one of ordinary skill in the art without any creative effort. Additionally, some aspects of the technology may be performed collaboratively by the server system 508, the client device 504, and the video source 522.

[0078] It should be understood that the operating environment 500, including the server system 508, the video source 522, and the video camera 118, is merely one example. Many aspects of the operating environment 500 are generally applicable in other operating environments in which a server system provides data processing to facilitate monitoring and reviewing data captured by other types of electronic devices (e.g., smart thermostats 102, smart hazard detectors 104, smart doorbells 106, smart wall plugs 110, appliances 112, etc.).

[0079] An electronic device, a client device, or a server system communicates with each other using one or more communication networks 162. In an exemplary smart home environment, two or more devices (e.g., network interface device 160, hub device 180, and client device 504-m) are arranged in proximity to each other so as to be communicatively coupled in the same subnetwork 162A via a wired connection, a WLAN, or a Bluetooth Personal Area Network (PAN). The Bluetooth PAN may optionally be established based on conventional Bluetooth technology or Bluetooth Low Energy (BLE) technology. This smart home environment further includes one or more other wireless communication networks 162B. Through these one or more other wireless communication networks 162B, at least some of the electronic devices of the video source 522-n exchange data with the hub device 180. Alternatively, in some situations, some of the electronic devices of the video source 522-n communicate directly with the network interface device 160 via the same subnetwork 162A that couples devices 160, 180, and 504-m. In some implementations (e.g., network 162C), the client devices 504-m and the electronic devices of the video source 522-n communicate directly via the network 162 without passing through the network interface device 160 or the hub device 180. In the same subnetwork 162A via a Bluetooth Personal Area Network (PAN). They are arranged in proximity to each other so as to be communicatively coupled. The Bluetooth PAN may optionally be established based on conventional Bluetooth technology or Bluetooth Low Energy (BLE) technology. This smart home environment further includes one or more other wireless communication networks 162B. Through these one or more other wireless communication networks 162B, at least some of the electronic devices of the video source 522-n exchange data with the hub device 180. Alternatively, in some situations, some of the electronic devices of the video source 522-n communicate directly with the network interface device 160 via the same subnetwork 162A that couples devices 160, 180, and 504-m. In some implementations (e.g., network 162C), the client devices 504-m and the electronic devices of the video source 522-n communicate directly via the network 162 without passing through the network interface device 160 or the hub device 180.

[0080] In some implementations, during normal operation, the network interface device 160 and the hub device 180 communicate with each other to form a network gateway. Through this network gateway, data is exchanged with the electronic devices of the video source 522-n. As described above, the network interface device 160 and the hub device 180 may optionally communicate with each other via the subnetwork 162A.

[0081] FIG. 6 is a block diagram illustrating a representative hub device 180 according to some implementations. In some implementations, the hub device 180 includes one or more processing units (e.g., CPU, ASIC, FPGA, microprocessor, etc.) 602, one or more communication interfaces 604, memory 606, a radio 640, and one or more communication buses 608 for interconnecting these components (sometimes referred to as a chipset). In some implementations, the hub device 180 includes one or more input devices 610, such as one or more buttons for receiving input. In some implementations, the hub device 180 includes one or more output devices 612, such as one or more indicator lights, a sound card, a speaker, a small display for displaying text information and error codes, etc. Additionally, in some implementations, the hub device 180 uses a microphone and voice recognition, or a camera and gesture recognition, to supplement or replace a keyboard. In some implementations, the hub device 180 includes a location detection device 614 , such as a global positioning satellite (GPS) or other geolocation receiver, to determine the location of the hub device 180 .

[0082] Hub device 180 optionally includes one or more built-in sensors (not shown) including, for example, one or more thermal radiation sensors, ambient temperature sensors, humidity sensors, IR sensors, radar, occupancy sensors (e.g., using RFID sensors), ambient light sensors, motion detectors, accelerometers, and / or gyroscopes.

[0083] The radio 640 enables and / or connects to one or more wireless communication networks in a smart home environment, allowing the hub device to communicate with the smart devices 204. In some implementations, the radio 640 supports a variety of custom or standard wireless protocols (e.g., IEEE 802.15.4, Wi-Fi, ZigBee, 6L0WPAN, Thread, Z-Wave, Bluetooth Smart, ISA100.11a, WirelessHART, MiWi, etc.), custom or standard wired protocols (e.g., Ethernet, HomePlug, etc.), and / or communication protocols not yet developed as of the filing date of this specification. The radio 640 may communicate data using any of the following communication protocols: IEEE 802.15.4, ...

[0084] In some implementations, the hub device 180 includes a radar subsystem. In some implementations, the radar subsystem uses radio waves (sometimes referred to as radar signals) to determine the range, angle, position, or speed of an object. In some implementations, the radar subsystem transmits radio waves (or microwaves) that reflect off objects in their path. The radar subsystem further receives and processes the reflected waves to determine the characteristics of the object. In some implementations, the radar subsystem includes memory 606, one or more radios 640, and / or one or more communication modules (e.g., wireless communication module 620) in one or more communication interfaces 604.

[0085] The communications interface 604 includes hardware capable of interfacing one or more radios 640 with the hub device 180 to enable data communications using, for example, any of a variety of custom or standard wireless protocols (e.g., IEEE802.15.4, Wi-Fi, ZigBee, 6LoWPAN, Thread, Z-Wave, Bluetooth Smart, ISA100.11a, WirelessHART, MiWi, etc.) and / or a variety of custom or standard wired protocols (Ethernet, HomePlug, etc.) or other suitable communications protocols, including communications protocols yet to be developed as of the filing date of this specification. In some implementations, the communications interface 604 includes one or more antennas for transmitting and receiving signals managed by the radios 640.

[0086] Memory 606 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid-state memory devices, and optionally includes non-volatile memory, such as one or more magnetic disk storage devices, one or more optical disk storage devices, one or more flash memory devices, or one or more other non-volatile solid-state storage devices. Memory 606, or alternatively, the non-volatile memory in memory 606, includes a non-transitory computer-readable storage medium. In some implementations, memory 606 or the non-transitory computer-readable storage medium of memory 606 stores the following programs, modules and data structures, or a subset or superset thereof:

[0087] • Operating logic 616, which includes procedures for handling various basic system services and for performing hardware dependent tasks.

[0088] Other network devices (e.g., Internet connections, networked storage devices, network routing devices, sub-devices, etc.) connected to one or more networks 162 via one or more communication interfaces 604 (wired or wirelessly) and a hub device communication module 618 for connecting to and communicating with a network interface 160 (such as a router that provides a server system 508 , etc.).

[0089] ● A wireless communication module 620 for connecting the hub device 180 to other devices (e.g., controller devices, smart devices 204 in the smart home environment 100, client devices 504 and / or electronic tags) via one or more wireless communication devices (e.g., radio 640).

[0090] ● A user interface module 622 for providing and displaying a user interface through which settings, captured data and / or other data for one or more devices (e.g., smart devices 204 in the smart home environment 100) may be configured and / or reviewed.

[0091] Hub device database 624. The hub device database 624 includes, but is not limited to, the following:

[0092] · Sensor information 6240 for storing and managing data received, detected and / or transmitted by one or more sensors of the hub device 180 and / or one or more other devices (e.g., smart devices 204 in the smart home environment 100).

[0093] Device settings 6242 for storing operational settings for one or more devices (e.g., smart devices 204 coupled in the smart home environment 100), such as device identification, timing settings, radar settings, operating modes and / or preference settings.

[0094] · Communications protocol information 6244 for storing and managing protocol information for one or more protocols (e.g., standard wireless protocols such as ZigBee, Z-Wave, etc., and / or custom or standard wired protocols such as Ethernet).

[0095] Each of the above identified elements (e.g., modules stored in memory 206 of hub device 180) may be stored in one or more of the memory devices mentioned above (e.g., memory of any of the smart devices in smart home environment 100 (FIG. 1)) and corresponds to a set of instructions for performing the functions described above. The above identified modules or programs (i.e., a set of instructions) need not be implemented as separate software programs, procedures, or modules, and thus various subsets of these modules may be combined or rearranged in various implementations. In some implementations, memory 606 optionally stores a subset of the above identified modules and data structures. Additionally, memory 606 optionally stores additional modules and data structures not described above.

[0096] 7A is a block diagram illustrating a server system 508 according to some implementations. The server system 508 typically includes one or more processing units (C). The system includes a PU 702, one or more network interfaces 704 (including, for example, an I / O interface to one or more client devices and an I / O interface to one or more electronic devices), memory 706, and one or more communication buses 708 for interconnecting these components (sometimes referred to as a chipset). The memory 706 includes high-speed random access memory such as DRAM, SRAM, DDR RAM or other random access solid-state memory devices, and optionally includes non-volatile memory, such as one or more magnetic disk storage devices, one or more optical disk storage devices, one or more flash memory devices, or one or more other non-volatile solid-state storage devices. Memory 706 optionally includes one or more storage devices located remotely from the one or more processing units 702. Memory 706, or alternatively the non-volatile memory within memory 706, includes a non-transitory computer-readable storage medium. In some implementations, memory 706 or the non-transitory computer-readable storage medium of memory 706 stores the following programs, modules and data structures, or a subset or superset thereof:

[0097] • An operating system 710 that contains instructions for handling various basic system services and for performing hardware dependent tasks.

[0098] ● A network communications module 712 for connecting the server system 508 to other systems and devices (e.g., client devices, electronic devices, and systems connected to one or more networks 162 (Figures 1-5)) via one or more network interfaces 704 (wired or wirelessly).

[0099] Server-side modules 714 that provide server-side functionality for device control, data processing and data review. Server-side modules 714 include, but are not limited to:

[0100] · A data reception module 7140 for receiving data from electronic devices (e.g., video data from a camera 118 and / or radar information from a radar-equipped device) and preparing the received data for further processing and storage in a data storage database 7160.

[0101] · A hub and device control module 7142 for generating and sending server-initiated control commands for changing operational modes of electronic devices (e.g., devices of the smart home environment 100) and / or for receiving and forwarding user-initiated control commands (e.g., from the client device 504) for changing operational modes of electronic devices.

[0102] A data processing module 7144 for processing data provided by the electronic device and / or preparing and sending the processed data to a device for review (e.g., the client device 504 for a user to review). The data processing module 7144 may include, but is not limited to:

[0103] - A radar processing module 7145 for processing radar data provided by the radar-equipped device, such as to classify radar events and identify entities detected by the radar.

[0104] - A video processing module 7146 for processing the video data provided by one or more cameras, such as to classify motion events and identify motion entities.

[0105] - A User Interface submodule 7150 for communicating with the user (eg, sending alerts, timeline events, etc., receiving user edits and zone definitions, etc.).

[0106] Server Database 716. Server Database 716 includes, but is not limited to, the following: Not done.

[0107] A data storage database 7160 for storing data associated with each electronic device (e.g., each camera) for each user account, as well as data processing models, processed data results, and other relevant metadata associated with the data (e.g., name of the data result, location of the electronic device, time of creation, duration, electronic device settings, etc.), in which case all or a portion of the data and / or processing associated with the hub device 180 or smart device (optionally) is securely stored.

[0108] · An account database 7162 for storing account information for user accounts including user account information such as user profiles 7163, information and settings for linked hub devices and electronic devices (e.g., hub device identification), hub device specific secrets, associated user and hardware characteristics (e.g., service tier, device model, storage capacity, processing power, etc.), user interface settings, data review preferences, etc. In this case, information about the associated electronic device includes, but is not limited to, one or more device identifiers (e.g., MAC address and UUID), device specific secrets, and displayed titles.

[0109] A device information database 7164 for storing device information about one or more devices, such as a device profile 7165, e.g., a device identifier and a hub device specific secret, regardless of whether the corresponding hub device is associated with any user account.

[0110] · An event information database 7166 for storing event information such as event records 7168, e.g., event log information, event categories, etc.

[0111] · A tag information database 7170 for storing tag information for one or more electronic tags, such as tag identifiers, tag signal timing, tag location information, etc.

[0112] A radar information database 7172 for storing radar information for one or more smart devices, such as radar band and / or mode information, historical radar data, radar target modeling information, etc.

[0113] Device timing information 7174 for storing timing information about one or more smart devices, for example timing synchronization information for synchronizing various smart devices.

[0114] Each of the above identified elements may be stored in one or more of the memory devices described above and corresponds to a set of instructions for performing the functions described above. The above identified modules or programs (i.e., sets of instructions) need not be implemented as separate software programs, procedures, or modules, and thus various subsets of these modules may be combined or rearranged in various implementations. In some implementations, memory 706 optionally stores a subset of the modules and data structures identified above. Additionally, memory 706 optionally stores additional modules and data structures not described above.

[0115] FIG. 7B illustrates various data structures used by some implementations, including an event record 7168-i, a user profile 7163-i, and a device profile 7165-i. An event record 7168-i corresponds to a motion event i and data about the motion event. In some cases, the event record 7168-i corresponds to a motion event i and data about the motion event i. All of the data includes motion start (also called cue point) data 71681, event segment data 71682, raw video data 71683, motion end data 71684, event feature data 71685, scene feature data 71686, associated user information 71687, and associated device information 71688. In some cases, the event record 7168-i includes only a subset of the data described above. In some cases, the event record 7168-i includes additional event data (not shown), such as data related to the event / motion mask.

[0116] Motion start data 71681 includes date and time information such as a timestamp, and optionally includes additional information such as information regarding the amount of motion present and / or the motion start location. Similarly, motion end data 71684 includes date and time information such as a timestamp, and optionally includes additional information such as information regarding the amount of motion present and / or the motion end location.

[0117] The event segments 71682 include information regarding segmentation of the motion event i. In some cases, the event segments are stored separately from the raw video data 71683. In some cases, the event segments are stored at a lower display resolution than the raw video data. For example, the event segments are optionally stored at 480p or 780p, and the raw video data is stored at 1080i or 1080p. By storing the event segments at a lower display resolution, the system can devote less time and resources to retrieving and processing the event segments. In some cases, the event segments are not stored separately, and the segmentation information also includes a reference to the raw video data 71683 along with date and time information for playing the event segments.

[0118] Event feature data 71685 includes information about event features such as event categorization / classification, object masks, motion masks, identified / recognized / tracked motion objects (sometimes referred to as blobs), information about motion object characteristics (e.g., object color, object dimensions, speed, resizing, etc.), information about activity in the zone of interest, etc. Scene feature data 71686 includes information about the scene in which the event takes place, such as depth map information, information about the location of windows, televisions, fans, ceilings / floors, etc., information about whether the scene is indoors or outdoors, information about the zone of interest, etc.

[0119] The associated user information 71687 includes information about a user associated with the event, such as a user identified in the event, a user who receives notification of the event, etc. In some cases, the associated user information 71687 includes a link, pointer, or reference to a user profile 7163 for the user. The associated device information 71688 includes information about the device or devices involved in the event (e.g., the camera 118 that recorded the event). In some cases, the associated device information 71688 includes a link, pointer, or reference to a device profile 7165 for the device.

[0120] The user profile 7163-i corresponds to a user i associated with a smart home network (e.g., smart home network 202), such as a user of the hub device 204, a user identified by the hub device 204, a user receiving notifications from the hub device 204 or the server system 508. In some cases, the user profile 7163-i includes user preferences 71631, user settings 71632, associated device information 71633, and associated event information 71634. In some cases, the user profile 7163-i includes only a subset of the above data. In some cases, Additionally, user profile 7163-i may include additional user information (not shown), such as information about other users associated with user i.

[0121] User preferences 71631 include explicit user preferences entered by the user, as well as implicit and / or inferred user preferences determined by the system (e.g., server system 508 and / or client device 504). In some cases, inferred user preferences are based on historical user activity and / or historical activity of other users. User settings 71632 includes information about settings set by user i, such as notification settings, device settings, etc. In some cases, user settings 71632 includes device settings for a device associated with user i.

[0122] The related device information 71633 includes information about devices associated with user i, such as devices in the user's smart home environment 100 and / or client device 504. In some cases, the related device information 71633 includes links, pointers, or references to corresponding device profiles 7165. The related event information 71634 includes information about events associated with user i, such as events for which user i was identified, events for which user i was notified, events corresponding to user i's smart home environment 100. In some cases, the related event information 71634 includes links, pointers, or references to corresponding event records 7168.

[0123] The device profile 7165-i corresponds to a device i associated with a smart home network (e.g., smart home network 202), such as a hub device 204, a camera 118, a client device 504, etc. In some cases, the device profile 7165-i includes device settings 71651, associated device information 71652, associated user information 71653, associated event information 71654, and environmental data 71655. In some cases, the device profile 7165-i includes only a subset of the above data. In some cases, the device profile 7165-i includes additional device information (not shown), such as information about whether the device is currently active.

[0124] Device settings 71651 include information about the current settings of device i, such as location information, mode of operation information, etc. In some cases, device settings 71651 are user specific and set by the respective user of device i. Related device information 71652 includes information about other devices associated with device i, such as other devices linked to device i and / or other devices in the same smart home network as device i. In some cases, related device information 71652 includes links, pointers or references to respective device profiles 7165 corresponding to the related devices.

[0125] The associated user information 71653 includes information about users associated with the device, such as users who receive notifications from the device, users registered with the device, users associated with the device's smart home network, etc. In some cases, the associated user information 71653 includes links, pointers, or references to user profiles 7163 corresponding to the associated users.

[0126] The related event information 71654 includes information about events associated with device i, such as historical events involving device i. In some cases, the related event information 71654 includes a link, pointer, or reference to an event record 7168 that corresponds to the associated event.

[0127] The environmental data 71655 includes information about the environment of the device i, such as information about whether the device is outdoors or indoors, information about the light levels of the environment, information about the amount of activity expected in the environment (e.g., whether the device is in a private residence or a busy commercial property), information about objects in the environment (e.g., depth mapping information for a camera), etc.

[0128] FIG. 8 is a block diagram illustrating a representative client device 504 associated with a user account, according to some implementations. The client device 504 typically includes one or more processing units (CPUs) 802, one or more network interfaces 804, memory 806, and one or more communication buses 808 for interconnecting these components (sometimes referred to as a chipset). Optionally, the client device also includes a user interface 810 and one or more built-in sensors 890 (e.g., accelerometers and gyroscopes). The user interface 810 includes one or more output devices 812 that enable display of media content, including one or more speakers and / or one or more displays. The user interface 810 also includes one or more input devices 814, including user interface components that facilitate user input (such as a keyboard, a mouse, a voice command input unit or microphone, a touch screen display, a touch sensitive input pad, a gesture capturing camera, or other input buttons or controls). Additionally, some client devices use a microphone and voice recognition or a camera and gesture recognition to supplement or replace a keyboard. In some implementations, the client device includes one or more cameras, scanners, or photosensor units for capturing images (not shown). Optionally, the client device includes a location detection device 816, such as a GPS (Global Positioning Satellite) or other geolocation receiver, for determining the location of the client device.

[0129] The memory 806 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid-state memory devices, and optionally includes non-volatile memory, such as one or more magnetic disk storage devices, one or more optical disk storage devices, one or more flash memory devices, or one or more other non-volatile solid-state storage devices. The memory 806 optionally includes one or more storage devices located remotely from the one or more processing units 802. The memory 806, or alternatively, the non-volatile memory in the memory 806, includes a non-transitory computer-readable storage medium. In some implementations, the memory 806 or the non-transitory computer-readable storage medium of the memory 806 stores the following programs, modules and data structures, or a subset or superset thereof:

[0130] • An operating system 818 that contains instructions for handling various basic system services and for performing hardware-dependent tasks.

[0131] ● A network communications module 820 for connecting the client device 504 to other systems and devices (e.g., client devices, electronic devices, and systems connected to one or more networks 162 (Figures 1-5)) via one or more network interfaces 804 (wired or wirelessly).

[0132] An input processing module 822 for detecting one or more user inputs or interactions from one of the one or more input devices 814 and translating the detected inputs or interactions.

[0133] ● Controlling devices (for example, sending commands to configure settings, etc. to hub devices and One or more applications 824 to be executed by the client device (e.g., games, social network applications, smart home applications, and / or other web or non-web based applications) for transmitting to the client device (and / or other client or electronic devices) and for reviewing data captured by the device (e.g., device status and settings, captured data, or other information regarding the hub device or other connected devices).

[0134] • A user interface module 622 for providing and displaying a user interface through which settings, captured data, and / or other data for one or more devices (e.g., smart devices 204 in the smart home environment 100) may be configured and / or reviewed.

[0135] - Client-side modules 828 that provide client-side functionality for device control, data processing and data review. The client-side modules 828 include, but are not limited to:

[0136] · A hub device and device control module 8280 for generating control commands to change the operating mode of the hub device or the electronic device according to user input.

[0137] A data review module 8282 for providing a user interface for reviewing data processed by the server system 508 .

[0138] - Client Data 830, which stores data associated with user accounts and electronic devices. Client Data 830 includes, but is not limited to:

[0139] Account Data 8300, which stores information about the user account both loaded on the client device (e.g., of the video source 522) and the electronic device associated with the user account. Such information includes cached login credentials, hub device identifiers (e.g., MAC address and UUID), electronic device identifiers (e.g., MAC address and UUID), user interface settings, display preferences, authentication tokens and tags, password keys, etc.

[0140] · A local data storage database 8302 for selectively storing raw or processed data associated with the electronic device (eg, of a video source 522 such as a camera 118).

[0141] Each of the above identified elements may be stored in one or more of the memory devices described above and corresponds to a set of instructions for performing the functions described above. The above identified modules or programs (i.e., sets of instructions) need not be implemented as separate software programs, procedures, modules, or data structures, and thus various subsets of these modules may be combined or rearranged in various implementations. In some implementations, memory 806 optionally stores a subset of the above identified modules and data structures. Additionally, memory 806 optionally stores additional modules and data structures not described above.

[0142] In some implementations, the client device 504 includes one or more graphical user interfaces and / or one or more modules for registering smart devices and / or electronic tags in a smart home environment. In examples, the client device 504 includes an application, such as a smart home application, for interacting with a smart home environment. In some implementations, the smart home application includes one or more user interfaces for one or more of registering smart devices, registering electronic tags, adjusting operation of the smart devices, reviewing data from the smart devices, etc. In some implementations, the smart home application includes a user interface module 826 and a client side module 828.

[0143] 9A is a block diagram illustrating a representative smart device 204 according to some implementations. In some implementations, the smart device 204 (e.g., any device in the smart home environment 100 (FIGS. 1 and 2), such as the camera 118, the smart hazard detector 104, the smart thermostat 102, the hub device 180, etc.) includes one or more processing units (e.g., CPU, ASIC, FPGA, microprocessor, etc.) 902, memory 906, a communication module 942 including one or more radios 940 and 950, a communication interface 904, and one or more communication buses 908 for interconnecting these components (sometimes referred to as a chipset). In some implementations, the user interface 910 includes one or more output devices 912 including one or more speakers and / or one or more displays and enabling display of media content. In some implementations, the user interface 910 also includes one or more input devices 914, including user interface components that facilitate user input, such as a keyboard, mouse, voice command input unit or microphone, touch screen display, touch sensitive input pad, gesture capture camera or other input buttons or controls. Additionally, some smart devices 204 use a microphone and voice recognition or a camera and gesture recognition to supplement or replace a keyboard. In some implementations, the smart device 204 includes one or more image / video capture devices 918 (e.g., a camera, a video camera, a scanner, a photo sensor unit). Optionally, the client device includes a location detection device 916, such as a GPS (Global Positioning Satellite) or other geolocation receiver, to determine the location of the smart device 204.

[0144] The built-in sensors 990 may include, for example, one or more thermal radiation sensors, ambient temperature sensors, humidity sensors, IR sensors, occupancy sensors (eg, using an RFID sensor), ambient light sensors, motion detectors, accelerometers, and / or gyroscopes.

[0145] Radio 940 and radio 950 enable one or more wireless communication networks in a smart home environment, allowing smart device 204 to communicate with other devices. In some implementations, data communication by radio 940 can be performed using a variety of custom or standard wireless protocols (e.g., IEEE802.15.4, Wi-Fi, ZigBee, 6LoWPAN, Thread, Z-Wave, Bluetooth Smart, ISA100.11a, WirelessHART, MiWi, etc.), custom or standard wired protocols (e.g., Ethernet, HomePlug, etc.), and / or any other suitable communication protocols, including communication protocols not yet developed as of the filing date of this specification. In some implementations, radio 940 and / or radio 950 are utilized for radar communication.

[0146] The communication interface 904 may, for example, support a variety of custom or standard wireless protocols (e.g., IEEE802.15.4, Wi-Fi, ZigBee, 6LoWPAN, Thread, Z-Wave, Bluetooth Smart, ISA100.11a, WirelessHART, MiWi, etc.) and / or a variety of color interfaces. The smart device 204 includes hardware capable of interfacing one or more radios 940 and 950 with the smart device 204 to enable data communications using a standard or standard wired protocol (such as Ethernet, HomePlug, etc.), or any other suitable communications protocol, including communications protocols not yet developed as of the filing date of this specification. In some implementations, each radio 940 and radio 950 has a respective communications interface 904 to facilitate and manage data communications for the respective radio, while in other implementations, multiple radios 940 and / or 950 are managed by a single communications interface 904.

[0147] In some implementations, the radio 940 and / or the radio 950 are configured to transmit and receive signals of the same or distinct types in a smart home environment. For example, the radio 940 may include a transceiver configured to transmit data between other devices (e.g., smart devices) in the smart home environment (e.g., IEEE 802.15.4 communication protocol for unidirectional / bidirectional transmission of data between two smart devices and between multiple smart devices). Signals transmitted between devices optionally include signals directed to, for example, important hazard information (e.g., a "ping" indicating smoke has been detected) or device status information (e.g., a "ping" indicating low battery). In contrast, in some implementations, the radio 950 may include a transceiver configured to transmit high bandwidth data over a data network (e.g., IEEE 802.11 Wi-Fi for uploading a video stream to the smart home provider server system 164). In some implementations, the radio 940 and / or the radio 950 include a transceiver configured for short-range communication with devices (e.g., Bluetooth communication protocol for device provisioning). In some implementations, the radio 940 and / or the radio 950 include a transceiver configured to transmit low-power signals (e.g., smart hazard detector 104 that is not connected to a permanent power source). In some implementations, the radio 940 and / or the radio 950 are configured to transmit multiple types of signals in a smart home environment (e.g., Wi-Fi radio 950 uploads video stream data to smart home provider server system 164 in addition to routing received beacons to other nearby smart devices). In some implementations, the radio 940 and / or the radio 950 of each device includes a transceiver for directly and communicatively linking each device to other devices.For example, paired devices communicate directly via Bluetooth rather than through a router by using Wi-Fi. In some implementations, radio 940 and / or radio 950 are configured to convert signals received via the first radio 940 and further configured to retransmit the converted signals using the first radio 940 and / or radio 950 (e.g., a proprietary message format is received and converted via Bluetooth, in which case the converted message is retransmitted to the other device using Wi-Fi).

[0148] In some implementations, radio 940 and / or radio 950 include a transceiver configured to transmit data via RFID (e.g., to be used in identifying electronic tags and / or other devices). In some implementations, radio 940 and / or radio 950 include a transceiver configured for radar operation (e.g., to be used in determining distance, speed, etc.). In some implementations, radio 940 and / or radio 950 are configured for radar operation over one or more radio frequency bands, such as 3 GHz to 10 GHz, 24 GHz to 24.5 GHz, 57 GHz to 64 GHz, and / or 77 GHz to 81 GHz.

[0149] The communications module 942 includes various components for enabling signals to be transmitted and received by each smart device 204, including one or more amplifiers, oscillators, antennas, filters, switches, memory, firmware, and / or any other supporting circuits or circuit components. In some implementations, one or more of the radios 940 and 950 are integrated components of the communications module 942 (e.g., a System on a Chip (SOC)). In some implementations, one or more of the radios 940 and 950 have respective circuit components. Alternatively, one or more of the radios 940 and 950 share one or more circuit components.

[0150] In some implementations, the communications module 942 includes a 1D radar subsystem having one transmitter and one receiver. In some implementations, the communications module 842 includes a multi-dimensional radar subsystem. For example, in some implementations, the communications module 842 includes two radar transmitters and four radar receivers.

[0151] Memory 906 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid-state memory devices, and optionally includes non-volatile memory, such as one or more magnetic disk storage devices, one or more optical disk storage devices, one or more flash memory devices, or one or more other non-volatile solid-state storage devices. Memory 906, or alternatively, the non-volatile memory in memory 906, includes a non-transitory computer-readable storage medium. In some implementations, memory 906 or the non-transitory computer-readable storage medium of memory 906 stores the following programs, modules and data structures, or a subset or superset thereof:

[0152] · Operating logic 920, which includes procedures for handling various basic system services and for performing hardware dependent tasks.

[0153] A device communications module 922 for connecting to and communicating with other network devices (e.g., network interfaces 160 such as routers providing Internet connectivity, networked storage devices, network routing devices, server systems 508, etc.) connected to one or more networks 162 via one or more communications interfaces 904 (wired or wirelessly).

[0154] · A wireless communication module 924 for connecting the smart device 204 to other devices (e.g., a controller device, a smart device 204 in the smart home environment 100, a client device 504) using wireless communication in conjunction with the communication module 942.

[0155] An input processing module 926 for detecting one or more user inputs or interactions from one or more input devices 914 and for interpreting the detected inputs or interactions.

[0156] A user interface module 928 for providing and displaying a user interface through which settings, captured data, and / or other data for one or more devices (e.g., smart devices 204 and / or other devices in the smart home environment 100) may be configured and / or reviewed.

[0157] Controlling devices (e.g., executing commands, sending commands, and / or configuring settings on the smart device 204 and / or other clients / electronic devices) One or more applications 930 (e.g., games, social network applications, smart home applications, and / or other web or non-web based applications) executed by the smart device 930 to monitor (capture) and review data captured by the device (e.g., device status and settings, captured data, or other information regarding the smart device 204 and / or other clients / electronic devices).

[0158] Device-side modules 932 that provide device-side functionality for device control, data processing and data review. Device-side modules 932 include, but are not limited to:

[0159] A command receiving module 9320 for receiving, forwarding, and / or executing instruction and control commands for operating the smart device 204 (e.g., from the client device 504, from the smart home provider server system 164, from user input detected on the user interface 910, etc.).

[0160] A data processing module 9322 for processing data captured or received by one or more inputs, (e.g., input devices 914, image / video capture device 918, location detection device 916), sensors (e.g., built-in sensors 990), interfaces (e.g., communications interface 904, radio 940), and / or other components of the smart device 204, and for preparing and sending the processed data to a device for review (e.g., client device 504 for review by a user). The data processing module 9322 includes, but is not limited to:

[0161] - A radar processing module 9324 for processing radar data captured or received by the smart device 204, e.g., to classify radar events, recognize radar entities, and / or aggregate radar data with data from other sensors (e.g., video data).

[0162] Device Data 934, which stores data associated with a device (e.g., smart device 204). Device Data 934 includes, but is not limited to:

[0163] Account Data 9340, which stores information about user accounts loaded onto the smart device 204. Such information includes cached login credentials, smart device identifiers (e.g., MAC address and UUID), user interface settings, display preferences, authentication tokens and tags, password keys, etc.

[0164] · A local data storage database 9342 for selectively storing raw or processed data (e.g., captured video footage and / or radar data) associated with the smart device 204.

[0165] · A bypass module 936 for detecting whether the radio 940 and / or the radio 950 are transmitting signals via a respective antenna coupled to the radio 940 and / or the radio 950, thereby coupling the radio 940 and / or the radio 950 to the respective antenna via a bypass line or amplifier.

[0166] (e.g., based on detected control signals and requests to transmit) 40 and / or a transmission access module 938 for granting or denying transmission access to the radio 950 .

[0167] A radar module 944 (eg, in conjunction with the communications module 942 and / or communications interface 904) for transmitting, receiving, and / or manipulating radar signals.

[0168] Each of the above identified elements may be stored in one or more of the memory devices described above and corresponds to a set of instructions for performing the functions described above. For example, in some implementations, one or more of the radios 940 and 950 include respective memory and firmware for storing one or more program / executable modules of the memory 906. The above identified modules or programs (i.e., sets of instructions) need not be implemented as separate software programs, procedures, or modules, and thus various subsets of these modules may be combined or rearranged in various implementations. In some implementations, the memory 906 optionally stores a subset of the above identified modules and data structures. Additionally, the memory 906 optionally stores additional modules and data structures not described above, such as a video processing module.

[0169] FIG. 9B is a block diagram illustrating an exemplary electronic tag 206, according to some implementations. FIG. 9B illustrates an electronic tag 206 including a communication circuit 903, a radar circuit 913, a controller 911, an energy storage circuit 923, and a communication line 901. In some implementations, the electronic tag 206 includes one or more additional sensors 925, such as a humidity sensor, an accelerometer, a gyroscope, a temperature sensor, an optical sensor, etc. In some implementations, the controller 911 is a component of the communication circuit 903. The communication circuit 903 includes a receiver 905 for receiving signals, such as an RFID signal. In some implementations, the communication circuit 903 includes a transmitter (not shown) for transmitting signals, such as an RFID signal. In some implementations, the communication circuit 903 includes a decoder 907 for decoding and / or decoding signals received via the receiver 905. In some implementations, the decoder 907 is a component of the controller 911. In some implementations, the controller 911 decodes and / or decodes signals received via the receiver 905. In some implementations, the communications circuitry 903 includes an encoder (not shown) for encoding the outgoing transmission.

[0170] The radar circuitry 913 includes a receiver 915 for receiving radar signals, a transmitter 917 for transmitting radar signals, a controller 919 for managing various components of the radar circuitry 913 (such as the receiver 915 and / or the transmitter 917), and a signal subsystem 921. In some implementations, the controller 919 is a component of the controller 911. In some implementations, the signal subsystem 921 includes components for amplifying, modulating, and / or generating a radar signal. In some implementations, the radar circuitry 913 includes passive radar circuitry, while in other implementations, the radar circuitry 913 includes active radar circuitry. In some implementations, the energy storage circuitry 923 includes an energy storage device, such as a capacitor or a battery. In some implementations, the energy storage circuitry 923 includes circuitry for coupling the electronic tag 206 to an external power source, such as a battery or a wall outlet. In some implementations, the energy storage circuitry 923 includes a power management integrated circuit (IC). In some implementations, the energy storage circuitry 923 includes circuitry for harvesting energy from signals received via the antenna (e.g., receiver 905) of the electronic tag. In some implementations, the energy storage circuitry 923 includes circuitry for harvesting thermal energy, seismic energy, electromagnetic energy, and / or solar energy received by the electronic tag. include.

[0171] In some implementations, the controller 911 is configured to (1) receive a command from another device via one or more antennas, (2) determine whether the command was intended for the electronic tag (e.g., based on the unique ID of the electronic device), (3) act on the command in accordance with a determination that the command was intended for the electronic tag, and (4) ignore the command in accordance with a determination that the command was not intended for the electronic tag.

[0172] In some implementations, the electronic tag 206 is configured with a tunable carrier tone range set to 1 megahertz (MHz). In this case, the electronic tag 206 has an adjustable modulation range of 100 kilohertz (kH z) In some implementations, the electronic tag 206 consumes less than 10 mW of power, such as 5 milliwatts (mW) of power.

[0173] In some implementations, the electronic tag 206 is configured to harvest energy via a radio. In some implementations, the electronic tag 206 is configured to harvest energy via a half-wave dipole antenna.

[0174] FIG. 10 is a block diagram illustrating a smart home provider server system 164 according to some implementations. The smart home provider server system 164 typically includes one or more processing units (CPUs) 1002, one or more network interfaces 1004 (e.g., including I / O interfaces for one or more client devices and I / O interfaces for one or more electronic devices), memory 1006, and one or more communication buses 1008 for interconnecting these components (sometimes referred to as a chipset). The memory 1006 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices, and optionally includes non-volatile memory, such as one or more magnetic disk storage devices, one or more optical disk storage devices, one or more flash memory devices, or one or more other non-volatile solid-state storage devices. The memory 1006 optionally includes one or more storage devices located remotely from the one or more processing units 1002. The memory 1006, or alternatively, the non-volatile memory in the memory 1006, includes a non-transitory computer-readable storage medium. In some implementations, memory 1006 or the non-transitory computer-readable storage medium of memory 1006 stores the following programs, modules and data structures, or a subset or superset thereof:

[0175] · An operating system 1010 that contains instructions for handling various basic system services and for performing hardware-dependent tasks.

[0176] A network communications module 1012 for connecting the smart home provider server system 164 to other systems and devices (e.g., client devices, electronic devices, and systems connected to one or more networks 162 (FIGS. 1-5)) via one or more network interfaces 1004 (wired or wirelessly).

[0177] · Server-side modules 1014 that provide server-side functionality for device control, data processing and data review. Server-side modules 1014 include, but are not limited to:

[0178] · A data reception module 10140 for receiving data from electronic devices (eg, video data from the camera 118 (FIG. 1)) and preparing the received data for further processing and storing in the data storage database 10160.

[0179] · A device control module 10142 for generating and sending server-initiated control commands to change the operational mode of an electronic device (e.g., a device of the smart home environment 100) and / or for receiving and forwarding user-initiated control commands (e.g., from the client device 504) to change the operational mode of the electronic device.

[0180] · A data processing module 10144 for processing data provided by the electronic device and / or preparing and transmitting the processed data to a device for review (e.g., the client device 504 for review by a user).

[0181] Server Database 1016. The server database 1016 includes, but is not limited to:

[0182] A data storage database 10160 for storing data associated with each electronic device (e.g., each camera) for each user account, as well as data processing models, processed data results, and other relevant metadata associated with the data (e.g., name of the data result, location of the electronic device, time of creation, duration, electronic device settings, etc.), where (optionally) all or a portion of the data and / or processing associated with the electronic device is securely stored.

[0183] · An account database 10162 for storing account information for user accounts including user account information, information and settings for linked hub devices and electronic devices (e.g., hub device identification), hub device specific secrets, associated user and hardware characteristics (e.g., service tier, device model, storage capacity, processing power, etc.), user interface settings, data review preferences, etc. In this case, information about the associated electronic device includes, but is not limited to, one or more device identifiers (e.g., MAC address and UUID), device specific secret, and displayed title.

[0184] Each of the above identified elements may be stored in one or more of the memory devices described above and corresponds to a set of instructions for performing the functions described above. The above identified modules or programs (i.e., sets of instructions) need not be implemented as separate software programs, procedures, or modules, and thus various subsets of these modules may be combined or rearranged in various implementations. In some implementations, memory 1006 optionally stores a subset of the above identified modules and data structures. Additionally, memory 1006 optionally stores additional modules and data structures not described above, such as a radar data processing module.

[0185] Additionally, in some implementations, the functionality of any of the devices and systems described herein (e.g., hub device 180, server system 508, client device 504, smart device 204, smart home provider server system 164) may be compatible with one another and performed by other devices or systems, in which case corresponding sub-modules of those functionality may additionally and / or alternatively be located within and / or managed by any of the devices and systems. The devices and systems illustrated in and described in connection with Figures 6-10 are merely exemplary, and various configurations of modules for implementing the functions described herein may be realized in various implementations.

[0186] As described in connection with FIGS. 6-10, devices and tags in a smart home environment (e.g., smart devices 204 in FIG. 2, such as the camera 118, smart thermostat 102, and smart hazard detector 104 in the smart home environment 100 in FIG. 1) include numerous circuit components and interfaces to enable communication with other systems, devices, and / or servers. For example, in some implementations, a smart device includes a communication module (e.g., communication module 942 (FIG. 9A)). The communication module includes one or more radios 940 and / or radios 950 (sometimes generally referred to as "transceivers" or "transceiver devices") for transmitting and receiving signals to other devices over a network. In some implementations, one or more radios 940 and radios 950 are components of a single integrated circuit (e.g., a system-on-a-chip (SOC)). Given the typical physical compactness of a smart device, the components of the communication module 942 and other components of the device are often located in close physical proximity to one another. For example, a typical smart device may include multiple wireless antennas, memory devices, sensors, chips, and other electronic components. However, as a result of their close physical spacing within the smart device, and combined with components in close contact with conductive materials (e.g., metal cases, camera mounts, wires, etc.), device components such as antennas are typically poorly isolated from each other's transmissions. Additionally, because these devices often share the same or similar frequency bands when operating (e.g., IEEE 802.11 (i.e., Wi-Fi) and 802.15.4, which share the 2.4 GHz frequency band), signals transmitted by one component tend to interfere with signals received and / or transmitted by other components. Finally, the components of the communication module 942 typically suffer from poor signal-to-noise ratios. It achieves low SNR, distortion, low analog signal quality, and high bit error rate (BER).

[0187] Furthermore, the insufficient isolation of these devices has an additive effect on the maximum input power of device components, as the transmit power of a signal transmitted by one transceiver unexpectedly adds to the anticipated transmission of signals simultaneously received by other nearby transceivers, thus risking damage to often sensitive device components when these input power thresholds are exceeded.

[0188] System Architecture and Processing Pipeline FIG. 11A illustrates an exemplary system architecture 1100, and FIG. 11B illustrates a corresponding data processing pipeline 1116. In some implementations, the server system 508 includes functional modules for an event processor 7146, an event categorizer 1107, a radar processor 7145, a radar categorizer 1108, and a user-facing front end 7150. The event processor 7146 obtains motion event candidates (e.g., by processing a video stream or by receiving motion onset information from a video source 522). The event categorizer 7148 categorizes the motion event candidates into various event categories. The radar processor 7145 obtains radar information 1106 (including raw and / or preprocessed radar data) from one or more radar sources 1101 and processes the radar information 1106 to obtain radar event candidates (e.g., to identify motion entities and / or stationary objects). In some implementations, the radar processor 7145 receives radar source information 1105, such as device ID, device location information, etc., from the radar source 1101. The radar categorizer 1108 categorizes the radar event candidates into various radar event categories. In some implementations, the radar categorizer 1108 performs machine learning based on the radar event candidates (e.g., to update the radar event categories). In some implementations, the radar categorizer 1108 adjusts one or more radar event categories based on the categorization of one or more radar event candidates and / or feedback (e.g., from a user) on the categorization. In some implementations, the radar categorizer 1108 learns based on one or more motion event candidates and / or motion event categories. In some implementations, the server system 508 includes one or more additional modules for categorizing event candidates based on both video information and radar information (e.g., by aggregating video information and radar information).

[0189] The user-facing front-end 7150 generates event alerts and facilitates review of motion events by reviewers via a review interface on the client device 504. The client-facing front-end also receives user edits for event categories, user preferences for alerts and event filters, and zone definitions for zones of interest. The event categorizer optionally modifies the event categorization model and results based on the user edits received by the user-facing front-end. The server system 508 also includes a video and radar source data database 1109, a categorization model database 1110 (e.g., categorization models for both radar and video events), and an event data and event mask database 1111. In some implementations, each of these databases is part of the server database 716. In some implementations, separate databases are utilized for radar-related information (e.g., radar source data and radar categorization models) and video-related information (e.g., video source data and video categorization models).

[0190] The server system 508 receives the video stream 1104 from the video source 522 and optionally motion event candidate information 1102, such as motion initiation information, such as device settings for the camera 118 (e.g., device profile 7165 for the camera 118), and video source information 1103. In some implementations, the event processor sub-module 7146 communicates with the video source 522. The server system sends alerts about the motion events 1105 and motion event timeline information 1107 to the client device 504. In some implementations, the server system 508 optionally receives user information from the client device 504, such as edits for event categories 1109 and zone definitions 1111.

[0191] The server system 508 receives radar information 1106 for a radar source 1101 (e.g., a radar-equipped smart device) and optionally receives radar source information 1105, such as device settings for the radar source 1101. In some implementations, a radar processor 7145 communicates with the radar source 1101. In some implementations, the server system 508 sends alerts about radar events and / or radar event timeline information to the client device. In some implementations, the server system 508 optionally receives user information from the client device 504, such as edits to radar event categories and / or radar zone definitions.

[0192] The data processing pipeline 1116 processes live-stream video received from video sources 522 (including, for example, cameras 118 and any controller devices) in real-time to identify and categorize motion events in the live-stream video, and provide real-time event alerts and a refreshed event timeline to the video sources 522. to a client device 504 associated with the viewer account. The data processing pipeline 1116 also processes the stored streamed video from the video sources 522 to re-evaluate and / or recategorize the motion events as needed, such as when new information is obtained regarding the motion events and / or when new information is obtained regarding the motion event categories (e.g., when new activity zones are obtained from a user).

[0193] After the video data is captured at the video source 522 (1117), the video data is processed to determine whether any potential motion event candidates are present in the video stream. Potential motion event candidates detected in the video data are sometimes referred to as cue points. Thus, the initial detection of a motion event candidate is referred to as motion onset detection and / or cue point detection. Motion onset detection (1118) triggers the execution of a more complete event identification process on the video segment (sometimes referred to as a "video slice" or "slice") that corresponds to the motion event candidate. In some implementations, the video data is initially processed at the video source 522. Thus, in some implementations, the video source sends motion event candidate information, such as motion onset information, to the server system 508. In some implementations, the video data is processed at the server system 508 for motion onset detection. In some implementations, the video stream is stored at the server system 508 (e.g., in the video and radar source data database 1109). In some implementations, the video stream is stored on a server separate from the server system 508. In some implementations, after a cue point is detected, the relevant portion of the video stream is retrieved from storage (e.g., video and radar source data database 1109).

[0194] In some implementations, the more complete event identification process includes segmenting (1119) the video stream into segments and then categorizing (1120) motion event candidates within each segment. In some implementations, categorizing the motion event candidates includes aggregating background factors, motion entity detection identification, motion vector generation for each motion entity, motion entity features, and scene features to generate motion features (11206) for the motion event candidates. In some implementations, the more complete event identification process further includes categorizing (11207) each segment, generating or updating a motion event log based on the segment categorization (11208), generating an alert for the motion event based on the segment categorization (11209), categorizing (1123) the complete motion event, updating (1124) the motion event log based on the complete motion event, and generating an alert for the motion event based on the complete motion event (1125). In some implementations, the categorization is based on a determination that the candidate motion event is within a particular zone of interest. In some implementations, the categorization is based on a determination that the candidate motion event includes one or more particular zones of interest.

[0195] The event analysis and categorization process may be performed collaboratively by the video sources 522 and the server system 508, and the division of tasks may vary for different implementations, different equipment capacity configurations, and / or different network and / or server load conditions. After the server system 508 categorizes the motion event candidates, the event detection and categorization results may be transmitted to a reviewer associated with the video source 522.

[0196] In some implementations, the server system 508 also determines an event mask for each motion event candidate and caches the event mask to be used later in an event search based on the selected zone of interest.

[0197] In some implementations, the server system 508 stores, for each video source 522, raw or compressed video data (e.g., in a video and radar source data database 1109), event categorization models (e.g., in a categorization model database 1110), and event masks and other event metadata (e.g., in an event data and event mask database 1111). In some implementations, the video data is stored in one or more display resolutions, such as 480p, 780p, 1080i, 1080p, etc.

[0198] The above is an overview of the system architecture 1100 for processing events during video surveillance and the data processing pipeline 1116. Further details of the processing pipeline and processing techniques are provided below.

[0199] 11A, the system architecture 1100 includes a video source 522. The video source 522 transmits live or stored streamed video over one or more networks (e.g., network 162) to a remote server system 508. In some implementations, the video data is transmitted continuously as the video data is captured by the camera 118. In some implementations, the transmission of the video data is independent of the content of the video data, and the video data is uploaded and stored from the video source 522 to the server system 508 regardless of whether any motion events have been captured in the video data. In some implementations, the video data may be stored by default on a local storage device of the video source 522, and only video portions corresponding to potential motion events detected in the video stream are uploaded (e.g., in real time) to the server system 508.

[0200] In some implementations, the video source 522 dynamically determines at what display resolution the video stream should be uploaded to the server system 508. In some implementations, the video source 522 dynamically determines which portion of the video stream should be uploaded to the server system 508. For example, in some implementations, depending on the current server load and network conditions, the video source 522 optionally prioritizes uploading video portions corresponding to newly detected motion event candidates before other portions of the video stream that do not include any motion event candidates, or the video source 522 uploads video portions corresponding to newly detected motion event candidates at a higher display resolution than other portions of the video stream. This upload priority helps ensure that important motion events are detected and reviewers are alerted in real time, even when network conditions and server loads are less than optimal. In some implementations, the video source 522 implements two parallel upload connections. The two parallel upload connections include one upload connection for uploading the continuous video stream captured by the camera 118 and the other upload connection for uploading video portions corresponding to detected candidate motion events. At any given time, the video source 522 determines whether it needs to temporarily stop uploading the continuous video stream to ensure that sufficient bandwidth is provided for uploading video segments corresponding to newly detected candidate motion events.

[0201] In some implementations, video streams uploaded for cloud storage are The stream is of lower quality (eg, lower resolution, lower frame rate, higher compression, etc.) than the video segments uploaded for motion event processing.

[0202] As shown in FIG. 11A, the video source 522 includes a camera 118 and an optional controller device. In some implementations, the camera 118 includes sufficient on-board processing power to perform all necessary local video processing tasks (e.g., cue point detection for motion event candidates, video upload prioritization, network connection management, etc.), and the camera 118 communicates directly with the server system 508 without any controller device acting as an intermediary. In some implementations, the camera 118 captures video data and transmits the video data to the controller device for necessary local video processing tasks. The controller device optionally performs local processing tasks for multiple cameras. For example, there may be multiple cameras in one smart home environment (e.g., smart home environment 100 (FIG. 1)), and a single controller device receives video data from each camera and processes the video data to detect motion event candidates in the video stream from each camera. The controller device is responsible for allocating sufficient outgoing network bandwidth to transmit video segments containing motion event candidates from each camera to the server, and then using the remaining bandwidth to transmit the video streams from each camera to the server system 508. In some implementations, a continuous video stream is transmitted and stored at one server facility, while video segments containing motion event candidates are sent to a different server facility for processing.

[0203] After the video data is captured by the camera 118, the video data is processed locally at the video source 522, optionally in real time, to determine whether there are cue points present in the video data that warrant a more thorough event identification process. Thus, in some implementations, the video source 522 sends motion event candidate information 1102, such as cue point detections, to the server system 508. In some implementations, the video source 522 sends additional metadata, such as the amount of motion between frames, to the server system 508.

[0204] Cue point detection is generally a first tier motion event identification that is intended to be slightly over-inclusive so that actual motion events are a subset of all identified cue points. In some implementations, cue point detection is based on the number of motion pixels in each frame of the video stream. In some implementations, any method of identifying motion pixels in a frame may be used. For example, a Gaussian mixture model is optionally used to determine the number of motion pixels in each frame of the video stream. In some implementations, a cue point is detected when the total number of motion pixels in the current image frame exceeds a predetermined threshold. In some implementations, a cumulative total motion pixel count is calculated for a predetermined number of consecutive frames as each new frame is processed. And, a cue point is detected when the cumulative total exceeds a predetermined threshold. In some implementations, a profile of the total motion pixel count over time is obtained. In some implementations, a cue point is detected when the profile of the total motion pixel count for a current frame sequence of a predetermined length (e.g., 30 seconds) meets a predetermined trigger criterion (e.g., total pixel count under profile>threshold motion pixel count). In some implementations, the cue point detection calculation is based on where in the scene the motion occurs. For example, a lower threshold is required for motion that occurs in or near a pre-defined zone of interest. In some implementations, a higher threshold is required for motion that occurs in or near a pre-defined zone of interest. In some implementations, cue points are required for motion occurring in or near predefined zones (e.g., notification disabled target zones) that are designated as likely to contain undesired motion events. In some implementations, cue points are suppressed for motion occurring within notification disabled target zones.

[0205] In some implementations, the cue point detection is based on one or more additional inputs, such as audio input to an associated microphone, for example, the cue point may be based at least in part on the sound of a glass breaking and / or a human voice.

[0206] In some implementations, the cue point begins when the total motion pixel count meets a predetermined threshold (e.g., 50 motion pixels). In some implementations, the cue point begins when the motion event candidate corresponding to the cue point begins. In some implementations, the beginning of the motion event candidate is a predetermined leading time (e.g., 5 seconds) before the beginning of the cue point. In some implementations, the beginning of the motion event candidate is used to process the video portion corresponding to the motion event candidate for a more complete event identification process.

[0207] In some implementations, the threshold for detecting cue points is adjusted over time based on performance feedback. For example, if too many false positives are detected, the threshold for the motion pixel count is arbitrarily high. If too many motion events are missed, the threshold for the motion pixel count is arbitrarily low. In some implementations, the threshold for detecting cue points is based on where in the scene motion is detected. In some implementations, the threshold is based on whether motion is detected within a particular zone of interest. In some implementations, the threshold is set and / or adjusted by a user (e.g., a user of client device 504 (FIG. 5)). For example, the threshold is adjusted by adjusting a corresponding motion sensitivity slider in a user interface.

[0208] In some implementations, cue point detection occurs at the video source 522, and immediately after a cue point is detected in the live video stream, the video source 522 sends an event alert to the server system 508 to trigger subsequent event processing. In some implementations, the video source 522 includes a video camera equipped with limited processing power but no controller device, and the cue point detection described herein is performed by the server system 508 on a continuous video stream transmitted from the camera to the server system 508.

[0209] In some implementations, the video source 522 sends additional video source information 1103 to the server system 508. This additional video source information 1103 may include information about the camera state (e.g., IR mode, AE mode, DTPZ setting, etc.) and / or information about the environment in which the video source 522 is located (e.g., indoors, outdoors, night, daytime, etc.). In some implementations, the video source information 1103 is used by the server system 508 to perform cue point detection and / or to categorize potential motion events within the video stream 1104.

[0210] In some implementations, after detecting a cue point, the portion of the video after the detected cue point is divided into multiple segments. In some implementations, end-of-motion information (sometimes referred to as an "end-of-motion signal") is obtained. Segmentation continues until a desired event is obtained. In some implementations, segmentation is performed within the server system 508 (eg, by the event processor module 7146).

[0211] As shown in FIG. 11B, in some implementations, a video stream is captured (111) and motion onset information corresponding to candidate motion events is obtained (1118). After obtaining the motion onset information, the video stream is segmented (1119) as described above. Each segment is then processed and categorized (1120). As described in more detail below, this process includes obtaining information about the background (e.g., background factors) in the scene (11201), identifying motion entities (11202), and obtaining motion vectors (11203). In some implementations, the process also includes identifying additional features of each motion entity (motion entity features) (11204), such as the amount of a particular color in the motion entity and / or the height-to-width ratio of the motion entity. In some implementations, the motion features include information about what zone of interest, if any, is associated with the motion entity. In some implementations, the process also includes identifying additional features of the scene (11205), such as the proportion of a particular color in the scene, audio information corresponding to the scene, and / or the total amount of motion in the scene. In some implementations, the scene features include information about zones of interest in the scene. The background factors, motion entities, motion vectors, and any additional motion entity and / or scene features are then aggregated to generate a resulting motion feature (11206). The resulting motion feature is categorized and a category is assigned to the motion event candidate (11207). In some implementations, a log entry is generated for the motion event candidate (11208) and the assigned category is stored internally. In some implementations, an alert is generated and sent to the client device 504 (11209). Once the motion end information is obtained, the final segment is processed and categorized (1121).In some implementations, after all segments have been categorized, the multi-segment features are processed (1122). These multi-segment features optionally include features generated by comparing motion event categories, event masks, motion entity features, etc. from various segments that contain the event. For example, the motion event masks for individual segments are combined to form a single motion event mask across all segments. In some implementations, after the multi-segment features are processed, an event category is assigned based on the multi-segment features (1123). In some implementations, the event category is assigned based on the multi-segment features and the categories assigned to the individual segments. In some implementations, an event log corresponding to the motion event candidates is updated (1124). In some implementations, an alert is generated based on the event category (1125).

[0212] In some implementations, after a motion event candidate is detected in a video stream, the video portion corresponding to the motion event candidate, or a particular segment within the video portion, is used to identify a motion track of a motion entity in the video segment. Identification of the motion track is optionally performed locally at the video source 522 or remotely at the server system 508. In some implementations, the motion track information is included in the motion event candidate information 1102 transmitted from the video source 522 to the server system 508. In some implementations, motion track information based on the video segments corresponding to the motion event candidates is determined. Identification of the cue points is performed at the server system 508 by an event processor module. In some implementations, the event processor module receives an alert about a cue point detected in the video stream and retrieves the video portion corresponding to the cue point from cloud storage (e.g., radar source data database 1109 (FIG. 11A)) or from the video source 522. In some implementations, the video portion used to identify the motion track may be of higher quality than the uploaded video for cloud storage, and the video portion is retrieved from the video source 522 separately from the continuous feed video uploaded from the video source 522.

[0213] In some implementations, after the event processor module acquires the video portion corresponding to the motion event candidate, the event processor module 7146 acquires background factors and performs motion entity detection identification, motion vector generation and feature identification for each motion entity. Once the event processor module 7146 completes these tasks, the event categorizer module 7148 collects all information and performs categorization for the motion event candidate. In some implementations, false positive suppression is optionally performed to reject some motion event candidates before the motion event candidates are event categorized. In some implementations, determining whether the motion event candidate is a false positive includes determining whether the motion event candidate occurred in a particular zone. In some implementations, determining whether the motion event candidate is a false positive includes analyzing an importance score for the motion event candidate. The importance score for the motion event candidate is optionally the same as or incorporates an importance score for the corresponding cue point. The importance score for the motion event candidate is optionally based on a zone of interest associated with the motion event candidate, background features, motion vectors, scene features, entity features, motion features, motion tracks, etc.

[0214] In some implementations, the video source 522 has sufficient processing power to perform background estimation, motion entity identification, motion vector generation, and / or feature identification.

[0215] In some embodiments, the motion vector representing the motion event candidate is a simple two-dimensional linear vector defined by the start coordinates and the end coordinates of a motion entity (sometimes referred to as a "motion object") in the scene shown in the video portion, and the motion event categorization is based on the motion vector. In some embodiments, the motion vector for the motion event candidate is generated independently for each segment. In some embodiments, a single motion vector is used for all segments and the motion vector is modified as each segment is processed. The advantage of using a simple two-dimensional linear motion vector for event categorization is that the event data is very compact and can be calculated and transmitted quickly over the network. When the network bandwidth and / or server load are constrained, simplifying the representative motion vector and offloading the motion vector generation from the event processor module of the video server system 508 to the video source 522 can assist in realizing real-time alert generation and event categorization for many video sources in parallel.

[0216] In some embodiments, after the motion track in the video segment corresponding to the motion event candidate is determined, the track length of the motion track is determined. In some implementations, the track length is determined for each segment independently. In some implementations, the track length is modified as each subsequent segment is processed. In some implementations, "short tracks" having a track length less than a predefined threshold (e.g., 8 frames) are suppressed, presumably due to inconsequential motion such as leaves blowing in the wind or lights flickering on a pond. In some implementations, short pairs of tracks that are roughly opposite in direction are suppressed as "noisy tracks". In some implementations, if after track suppression, there is no motion track remaining for the video segment, the cue point is determined to be a false positive, and the motion event candidate is not sent to the event categorizer for event categorization. In some implementations, if after track suppression, there is no motion track remaining, the motion event candidate is classified as a non-event. If at least one motion track remains after false positive suppression is performed, a motion vector is generated for each remaining motion track. In other words, multiple motion entities may be identified within a particular video segment. False positive suppression, which occurs after cue point detection and before motion vector generation, is a second layer of false positive suppression that eliminates false positives based on the characteristics of the motion tracks.

[0217] In some implementations, the radar information 1106 is processed independently of the video stream 1104. In some implementations, the video stream 1104 is processed on a first server of the server system 508, and the radar information 1106 is processed on a second server of the server system 508. In some implementations, the radar information received via the wireless receiver is processed at the radar source 1101. In some implementations, the radar information 1106 includes raw radar information from the wireless receiver and / or processed radar information, such as radar cue point information. In some implementations, similar to the video processing, the radar processing includes cue point detection. The radar cue points are optionally based on total detected motion, speed of detected motion, angle or position of motion, or a combination thereof. In some implementations, the radar data is processed without performing cue point detection. The radar data corresponds to an outgoing wireless transmission and one or more corresponding received wireless transmissions.

[0218] In some implementations, radar processing of the radar processor 7145 identifies the position and velocity of moving objects. In some implementations, the radar categorizer 1108 obtains the position and velocity data and assigns radar event categories based on the position and velocity data. In some implementations, additional information is obtained and utilized by the radar categorizer 1108 to categorize events, such as information about the dimensions of the motion object, information about the acceleration of the motion object, information from one or more radar tags, and background information received from other sources (e.g., identification information about one or more detected objects). In some implementations, the radar processor 7145 and / or the radar categorizer 1108 receive radar information from multiple radar sources and process the information to assign radar event categories. For example, the server system 508 receives radar data from multiple radar sources (e.g., 1D radar sources) in a particular room and aggregates the radar data to categorize events occurring in the particular room. In some implementations, the server system 508 receives radar data from one or more radar-equipped smart devices and from one or more radar tags, processes the radar tags, and categorizes radar events.

[0219] In some implementations, a composite event category may be a combination of a radar category assigned by the radar categorizer 1108 and a motion event category assigned by the motion event categorizer 1107. The composite event category is assigned (e.g., by a composite categorizer) based on the motion event category assigned by the motion event segment and the motion event category assigned by the motion event segment. For example, the motion event category may indicate a person with an active flashlight and a ski mask moving in a room. The radar category may indicate a particular person (John) walking toward the back of the room. In this example, the composite event category may indicate John walking toward the back of the room while wearing a ski mask and carrying an active flashlight. In some implementations, the assigned radar category is used as an input to the processing pipeline 1116. For example, the assigned radar category may be obtained and used to categorize 11207 the motion event segments and / or to categorize 1123 the entire motion event. In some implementations, the assigned motion event category is used to categorize the radar event.

[0220] Radar tag operation FIG. 12A illustrates an environment and system for communicating via radar signals according to some implementations. FIG. 12A illustrates a control device 1200 in a residence communicating with an electronic tag 1202 via a signal 1206. In some implementations, the control device 1200 is a smart device such as a camera, a thermostat, a hazard detector, a hub device, etc. In some implementations, the control device 1200 includes a smart device 204 (e.g., FIG. 9A) according to any of the implementations disclosed in FIGs. 1-9 (e.g., devices 102, 104, 106, 108, 110, 112, 114, 116, 118, 120, and / or 122 (FIG. 1) such as a camera 118, a smart hazard detector 104, a smart thermostat 102, etc.). In some implementations, the electronic tag 1202 includes an electronic tag 206 (FIG. 9B). In some implementations, the signal 1206 includes an RFID signal. In some implementations, the signals 1206 include signals transmitted at the same frequency, while in other implementations, at least a subset of the signals 1206 are transmitted at distinct frequencies (e.g., distinct frequencies within a frequency band). In some implementations, the signals 1206 each include device identification information. For example, each of the electronic tags 1202 receives the signal 1206-1 and analyzes the included device identification information to determine whether the signal 1206-1 was intended for the particular tag. Thus, according to some implementations, the signal 1206-1 includes a device identification for the electronic tag 1202-1, the signal 1206-2 includes a device identification for the electronic tag 1202-2, the signal 1206-3 includes a device identification for the tag 1202-3, and the signal 1206-4 includes a device identification for the tag 1202-4. The control device 1200 further transmits and receives radar signals. In some implementations, the radar signals are used to determine the distance, velocity, acceleration, position and / or direction of movement of multiple objects within the signal range. The electronic tag 1202 communicates with the control device 1200 via the radar signals. In some implementations, the electronic tag 1202 communicates via the radar signals in response to an enabling signal (e.g., signal 1206-1).In some implementations, communicating via radar signals includes reflecting, modifying, and / or amplifying received radar signals. In some implementations, communicating via radar signals includes generating a radar signal in the electronic tag independent of any radar signals received by the tag. As shown in FIG. 12A, the electronic tag 1202 is optionally configured to be affixed to inanimate objects such as walls and ceilings, moving objects such as doors and windows, and / or moving entities such as people and pets.

[0221] FIG. 12B illustrates an exemplary system and process for utilizing radar tags, according to some implementations. At a first time, the radar circuitry of electronic tags 1202-1 and 1202-2 is disabled, as indicated by 1208 and 1209. In some implementations, disabling the radar circuitry includes disconnecting the radar circuitry from a power source. In some implementations, disabling the radar circuitry includes preventing the radar circuitry from transmitting radio signals.

[0222] At a second time, the control device 1200 (e.g., smart device 204) transmits a first enabling signal 1210 via sideband (e.g., Wi-Fi). The electronic tag 1202-1 receives the first enabling signal 1210. In some implementations, the electronic tag 1202-1 determines that the first enabling signal 1210 is intended for the electronic tag 1202-1. In some implementations, determining that the first enabling signal 1210 is intended for the electronic tag 1202-1 includes comparing a device identification in the signal 1210 to a device identification of the electronic tag 1202-1. In response to receiving the first enabling signal 1210 or determining that the signal 1210 was intended for the electronic tag 1202-1, the electronic tag 1202-1 enables (1212) its radar circuitry.

[0223] In some implementations, electronic tag 1202-2 receives first enabling signal 1210 and determines that the first enabling signal is not intended for electronic tag 1202-2.

[0224] At a third time, the control device 1200 transmits a radio signal 1214 using its radar circuitry. The electronic tag 1202-1 receives the radio signal 1214. In some implementations, the electronic tag 1202-1 modifies the received radio signal 1214. In some implementations, modifying the radio signal 1214 includes amplifying and / or modulating the radio signal 1214. In some implementations, the electronic tag 1202-1 transmits a modified radio signal 1216. In some implementations, the electronic tag 1202-1 transmits a radio pulse or tone in response to receiving the radio signal 1214. After transmitting the radio signal 1216, the electronic tag 1202-1 disables (1217) its radar circuitry.

[0225] The control device 1200 receives signals 1216 from the electronic tag 1202-1. In some implementations, the control device 1200 processes the received signals 1216 to determine the location and / or motion of the electronic tag 1202-1. In some implementations, the control device 1200 transmits information about the wireless signals 1214 and 1216 to another device (e.g., a device in the server system 508) for processing.

[0226] At a fourth time, the control device 1200 transmits a second enabling signal 1218 via sideband. The electronic tag 1202-2 receives the second enabling signal 1218. In some implementations, the electronic tag 1202-2 determines that the second enabling signal 1218 was intended for the electronic tag 1202-2. In response to receiving the second enabling signal 1218, or in response to determining that the signal 1218 was intended for the electronic tag 1202-2, the electronic tag 1202-2 enables (1220) its radar circuitry.

[0227] At a fifth time, the control device 1200 transmits a radio signal 1222 using its radar circuitry. The electronic tag 1202-2 receives the radio signal 1222. In some implementations, the electronic tag 1202-2 modifies the received radio signal 1222. In some implementations, the electronic tag 1202-2 transmits the modified radio signal 1224. In some implementations, the electronic tag 1202-2 transmits a radio pulse or tone signal 1224 in response to receiving the radio signal 1222. After transmitting the radio signal 1224, the electronic tag 1202-2 disables (1225) its radar circuitry.

[0228] The control device 1200 receives signals 1224 from the electronic tag 1202-2. In some implementations, the control device 1200 processes the received signals 1224 to determine the location and / or motion of the electronic tag 1202-2. In some implementations, the control device 1200 transmits information about the wireless signals 1222 and 1224 to another device (e.g., a device in the server system 508) for processing.

[0229] 12C illustrates an exemplary system and process for utilizing radar tags, according to some implementations. The control device 1200 obtains 1232 an identification for the electronic tag 1202-1. In some implementations, the identification is obtained via an application, such as a smart home application, on the client device 504. In some implementations, the client device 504 scans a barcode or QR code on the electronic tag 1202-1 and transmits the scanned information to the control device 1200.

[0230] The control device 1200 transmits (1234) a request to enable the electronic tag 1202-1 along with the identification information. In some implementations, the request is transmitted over a first communication channel (e.g., RFID). In some implementations, the control device 1200 encrypts (1236) the request before transmission.

[0231] The electronic tag 1202-1 receives (1238) a request to enable the electronic tag 1202-1. The electronic tag determines (1240) that the enablement request was intended for the electronic tag 1202-1. In some implementations, the electronic tag 1202-1 decodes (1242) the enablement request before determining whether the enablement request was intended for the electronic tag 1202-1. In some implementations, determining that the enablement request was intended for the electronic tag 1202-1 includes comparing identification information in the request to a device identification stored in the electronic tag 1202-1.

[0232] Pursuant to a determination that the enablement request was intended for electronic tag 1202-1, electronic tag 1202-1 enables its radar circuitry (1244). In some implementations, enabling the radar circuitry includes providing power to one or more components in the radar circuitry. In some implementations, enabling the radar circuitry includes configuring the radar circuitry to respond to received radio signals (e.g., received radio signals having a particular frequency or within a particular frequency range).

[0233] The control device 1200 transmits 1246 a wireless signal. In some implementations, the control device 1200 utilizes radar circuitry to transmit the wireless signal.

[0234] The electronic tag 1202-1 receives (1248) a radio signal. In some implementations, the radio signal is received via radar circuitry in the electronic tag 1202-1. In some implementations, the electronic tag 1202-2 modifies (1250) the received radio signal. In some implementations, modifying the received radio signal includes amplifying and / or modulating the received radio signal. The electronic tag 1202-1 transmits (1252) a radio signal corresponding to the radio signal transmitted by the control device 1200. In some implementations, the transmitted signal includes the modified signal. In some implementations, the transmitted signal includes radio tones and / or pulses generated by the electronic tag 1202-1.

[0235] The control device 1200 receives (1254) the radio signal transmitted by the electronic tag 1202-1. In some implementations, the control device 1200 The signals are processed to determine the location and motion of the electronic tag 1202-1. In some implementations, the control device 1200 transmits information about the transmitted wireless signals received from the electronic tag 1202-1 to a second device or server system (e.g., server system 508) for processing.

[0236] In some implementations, the control device 1200 obtains information (e.g., during the registration process) regarding the object to which each electronic tag 1202 is attached. For example, the control device 1200 obtains information that electronic tag 1202-1 is attached to a wall, that electronic tag 1202-2 is attached to a window, and that electronic tag 1202-3 is attached to a door. In some implementations, the control device 1200 uses the object information when processing the received radar signal. For example, the control device 1200 ignores radar information regarding the object behind the wall to which electronic tag 1202-1 is attached.

[0237] In some implementations, the control device 1200 is configured to enable each of the electronic tags 1202 at a particular time (or at particular time intervals) to reduce interference between electronic tag communications. For example, the control device 1200 is configured so that only one electronic tag communicates via radar at a given time. In some implementations, the control device 1200 transmits an enable signal to enable radar communications for a particular electronic tag. In some implementations, the control device 1200 transmits a disable signal to disable radar communications for a particular electronic tag.

[0238] In some implementations, each of the electronic tags 1202 is configured to communicate via radar at set intervals. In some implementations, the intervals are set by the control device 1200 such that only one electronic tag 1202 transmits at a given time. In some implementations, these intervals are based at least in part on the object to which the electronic tag is affixed. For example, according to some implementations, an electronic tag affixed to a wall communicates once a day, while an electronic tag affixed to a window communicates once a minute.

[0239] In some implementations, the electronic tag 1202 communicates via radar in response to a stimulus received at a sensor of the electronic tag. For example, the electronic tag 1202-3 communicates via radar in response to detecting (e.g., by an accelerometer) the movement of a door to which it is affixed. In some implementations, the electronic tag 1202 detects the movement of an object to which it is affixed and in response, transmits a notification to the control device 1200 (e.g., by sideband). In some implementations, the control device 1200 receives the notification and in response transmits a radio signal used in radar analysis.

[0240] In some implementations, the control device 1200 uses radar communications from the electronic tag 1202 along with other radar signals to improve the accuracy and precision of the radar system. For example, the control device 1200 detects slight movement of a door to which tag 1202-3 is affixed during a particular time frame. In this example, during the same time frame, tag 1202-3 transmits a signal, and the control device 1200 uses the detected movement of the door and the signal from tag 1202-3 to map the movement of the door (e.g., with greater precision / accuracy than a mapping based on the detected movement of the door).

[0241] In some implementations, an electronic tag such as electronic tag 1202-3 is (e.g., temporarily) affixed to an object and a registration signal is sent to control device 1200. In some implementations, the registration signal is sent via an application (e.g., a smart home application) on a mobile device such as client device 504. In some implementations, the registration signal includes information about the object to which the electronic tag is attached (e.g., identifying the object as a door, window, or wall). In some implementations, the control device 1200 completes the registration process by processing the registration information and radar information received from the electronic tag to classify the object to which the electronic tag is attached and / or to determine the object's location relative to the control device 1200. In some implementations, the classification and / or location information is used for future radar processing. In some implementations, after the registration process is completed, the electronic tag is not used in future radar processing of objects. In some implementations, after the registration process is completed, the electronic tag is used to register one or more additional objects.

[0242] In some implementations, an electronic tag (e.g., electronic tag 1202-1) for affixing to an object to provide a radar signature for the object includes a first circuit (e.g., communication circuit 903 (FIG. 9B)) configured to communicate with one or more other devices at a first frequency. In some implementations, the first circuit includes (a) one or more antennas (e.g., receiver 905) configured to communicate at the first frequency, and (b) a first controller (e.g., controller 911) coupled to the one or more antennas and configured to manage the one or more antennas. In some implementations, the electronic tag includes a second circuit configured to communicate with one or more other devices via radar (e.g., radar circuit 913). In some implementations, the second circuit includes (a) one or more second antennas (e.g., receiver 915 and / or transmitter 917) configured to communicate via the radar, and (b) a second controller coupled to the one or more second antennas and the first controller (e.g., controller 919) and configured to communicate with the first controller and manage the one or more second antennas.

[0243] In some implementations, the first controller is further configured to (1) receive a command from another device via one or more antennas, (2) determine whether the command was intended for the electronic tag (e.g., based on a unique ID of the electronic device), (3) act on the command in accordance with a determination that the command was intended for the electronic tag, and (4) ignore the command in accordance with a determination that the command was not intended for the electronic tag.

[0244] In some implementations, the command includes a command to enable radar communications. In this case, the first controller acting on the command includes the first controller communicating the enable command to the second controller. In some implementations, the command includes a recipient identification. In this case, the first controller determining whether the command was intended for the electronic tag includes comparing the recipient identification to an identification of the electronic tag. In some implementations, the command is encrypted. In this case, the first controller is further configured to decrypt the command.

[0245] In some implementations, the first circuit further includes a decoder configured to decode signals received via the one or more antennas. In some implementations, the decoder is further configured to decode the command. In some implementations, the decoder is further configured to determine whether the command was intended for the electronic device. In some implementations, the decoder is a component of the first controller. In some implementations, the second controller is further configured to encrypt data transmitted via the one or more second antennas.

[0246] In some implementations, the electronic tag includes an energy storage circuit coupled to the first circuit and the second circuit, the energy storage circuit configured to provide power to the first circuit and the second circuit.

[0247] In some implementations, the energy storage circuit includes at least one of: (i) one or more capacitors; (ii) one or more batteries; (iii) circuitry configured to harvest energy from a signal received via the electronic tag's antenna (e.g., one of the one or more antennas, one of the one or more second antennas, or a third antenna dedicated to harvesting); and (iv) circuitry configured to harvest thermal energy, vibration energy, electromagnetic energy, and / or solar energy received by the electronic tag.

[0248] In some implementations, the second controller is further configured to selectively enable radar communications via the one or more second antennas, including at least one of: (i) reflecting received radio waves (e.g., at a given wavelength), (ii) amplifying the received radio waves, (iii) modulating the received radio waves, and (iv) generating radio waves. In some implementations, the second controller is configured to enable radar communications in response to receiving an enable command from the first controller.

[0249] In some implementations, the second circuit further includes a modulator configured to modulate radio waves received via the one or more second antennas, hi some implementations, the second circuit further includes a signal generator configured to generate radio waves at a particular frequency.

[0250] In some implementations, the electronic tag further includes one or more additional sensors coupled to the first controller, the one or more additional sensors including at least one of a humidity sensor, a temperature sensor, an accelerometer, a gyroscope, and an optical sensor.

[0251] In some implementations, the first circuit is configured to transmit tag information (e.g., tag status information such as battery life, sensor readings, etc.) via one or more antennas. In some implementations, the second circuit is configured to transmit tag information (e.g., tag status information such as battery life, sensor readings, etc.) via one or more second antennas.

[0252] In some implementations, the electronic tag includes a passive tag, such as a corner reflector or a printed radar tag. In some implementations, the passive tag modulates an incoming radio signal. The modulated incoming signal is reflected back toward a transmitting device (e.g., the control device 1200). In some implementations, the signal is modulated such that when a radar system analyzes the reflected signal, the tag appears to be moving at a particular speed.

[0253] FIG. 12D is a forecast of radar data according to some implementations. FIG. 12D shows detected velocities for various objects based on received radar data. In FIG. 12D, most of the detected objects are stationary or have very little velocity. Two objects (object 1260 and object 1262) with high velocity are detected. The detected object 1260 has a high negative velocity indicating that it is moving away from the radar device, and a distance d 1 The detected object 1262 is detected at distance d with a high positive velocity indicating that it is approaching the radar device. 2 It is detected in.

[0254] According to some implementations, the detected objects 1260 and 1262 represent passive radar tags. Passive radar tags are configured to modulate incoming radio waves so that they appear to be moving at high speeds even when the tags themselves are stationary. The apparent speed of a radar tag due to modulation is often referred to as a phantom speed. Each radar tag is optionally configured such that its modulation results in a distinct phantom speed. In some implementations, the radar device determines that the speed of the objects 1260 and 1262 is a phantom speed based on the value of each speed. For example, a radar-equipped device installed in a smart home environment detects an object with a speed of 30 miles per hour. Since objects in a smart home generally do not move at such high speeds, the radar-equipped device determines that the speed is a phantom speed and that the object is a radar tag. In some implementations, the radar-equipped device determines that the speed of the objects 1260 and 1262 is a phantom speed based on the fixed position of the object. For example, a radar-equipped device may perform a first scan and measure the distance d 1 The radar-equipped device later performs a second scan and detects the object 1260 at distance d 1 1260. Because object 1260 is at the same range during both scans, the velocity is determined to be phantom velocity. In some implementations, the radar-equipped device stores a list of radar tag information along with the expected range and / or expected phantom velocity of each tag (e.g., obtained during the registration process for the radar tag). In some implementations, the radar-equipped device compares the list of radar tag information to the detection results to identify any detected tags.

[0255] In some implementations, the radar tag is affixed to an object such as a door or window. In some cases, the object to which the device is affixed is moving during the radar scan. In these environments, the detected velocity of the radar tag includes both a phantom velocity due to the modulation of the radio signal and the actual velocity of the object to which the radar tag is attached. In some implementations, the radar-equipped device identifies a component of the actual velocity and associates it with the object to which the radar tag is affixed. For example, the radar tag is affixed to a door, and the radar-equipped device determines that the radar tag is moving one inch per second. In this example, the radar-equipped device associates the inch-per-second movement with the door and optionally alerts a user of the smart home environment that the door movement has been detected. In some implementations, the radar-equipped device identifies a component of the actual velocity based on the detected distance of the radar tag (e.g., the detected distance over a period of time). In some implementations, the radar-equipped device identifies an absolute velocity component based on an expected phantom velocity for the radar tag.

[0256] In some implementations, the radar tag is configured to optionally selectively modulate the incoming radio waves so that it exhibits one of a number of different phantom velocities. For example, a particular radar tag may be configured to operate in a first state in which it exhibits a phantom velocity of 10 miles per hour, in a second state in which it exhibits a phantom velocity of 15 miles per hour, or in a third state in which it exhibits a phantom velocity of 20 miles per hour. In some implementations, the operational state of the radar tag is managed by a control device, such as control device 1200. In some implementations, the operational state of the radar tag is configured during a registration process.

[0257] In some implementations, the method is performed in a computing system having one or more processors and memory (e.g., control device 1200). The method includes (1) transmitting a request to enable a remote radar device (e.g., electronic tag 1202-1) via a first transmitter, and (2) receiving, in response to the request, wireless communication from the radar device via a radar receiver.

[0258] In some implementations, the method further includes (1) obtaining an identification for the remote radar device and (2) encrypting the identification, wherein transmitting the request includes transmitting the encrypted identification.

[0259] In some implementations, the method further includes transmitting, via the first transmitter, a request to enable a second remote radar device after receiving wireless communication from the radar device.

[0260] In some implementations, the method further includes, after receiving wireless communication from the radar device, transmitting a request via the first transmitter to disable wireless communication by the radar device (e.g., to prevent interference with communication by a second radar device).

[0261] In some implementations, the method further includes (1) receiving a registration request for the remote radar device before transmitting the request, and (2) determining a location of the remote radar device in response to receiving the registration request.

[0262] In some implementations, determining the location of the remote radar device includes determining the location of the remote radar device based on received wireless communications.

[0263] In some implementations, the registration request is received via an application (eg, an application on a mobile device).

[0264] In some implementations, the registration request includes information about the object to which the remote radar device is affixed (eg, mounted on a door, window, wall, etc.).

[0265] In some implementations, the computing system includes a smart device (eg, smart device 204).

[0266] In some implementations, the smart device includes one of a thermostat, a camera, a hub device, a hazard detector, an irrigation device, a media playback device, an aisle interface device, an appliance, or a security device.

[0267] In some implementations, the wireless communication includes at least one of: (i) reflection of a radio signal transmitted by the computing device; (ii) modulation of a radio signal transmitted by the computing device; and (iii) a radio signal (e.g., a tone or pulse) generated at a remote radar device.

[0268] In some implementations, the wireless communication includes information regarding the operation of the remote radar device (e.g., sensor readout information).

[0269] In some implementations, the method further includes receiving a communication (eg, status information) from the radar device via the first receiver.

[0270] In some implementations, a method performed in an electronic device (e.g., electronic tag 1202-1) includes the steps of: (1) receiving an enablement request from a remote device (e.g., control device 1200) via a first receiver; (2) determining whether the enablement request was intended for the electronic device; (3) ignoring the enablement request in accordance with a determination that the enablement request was not intended for the electronic device; and (4) ignoring the enablement request in accordance with a determination that the enablement request was not intended for the electronic device. and communicating with the remote device via the radar in accordance with the determination.

[0271] In some implementations, the enablement request is encrypted and the method further includes decrypting the enablement request.

[0272] In some implementations, determining whether the enablement request was intended for the electronic device includes comparing an identification of the electronic device to an identification in the enablement request.

[0273] In some implementations, the enablement request includes timing information for communicating with the remote device via the radar, and the method further includes communicating with the remote device via the radar according to the received timing information (e.g., transmitting a pulse every minute, every hour, or every day).

[0274] In some implementations, the method further includes encrypting information to be transmitted to the remote device, in which case communicating with the remote device via the radar includes communicating the encrypted information.

[0275] In some implementations, the method further includes (1) storing energy received via one or more antennas of the electronic device, and (2) using the stored energy to communicate with a remote device via the radar.

[0276] In some implementations, communicating with a remote device via the radar includes at least one of reflecting the received radio waves, amplifying the received radio waves, modulating the received radio waves, and generating radio waves.

[0277] Synchronization of multiple radar-based devices FIG. 13A illustrates an environment and system for utilizing data from multiple radar devices, according to some implementations. FIG. 13A illustrates a first radar-equipped device 1300 (e.g., camera 118) communicating with a second radar-equipped device 1302 (e.g., smart hazard detector 104) via a signal 1304. In some implementations, the signal 1304 includes a wireless signal utilizing a precision timing protocol, such as Wi-Fi. For example, according to some implementations, the device 1300 and the device 1302 communicate using the IEEE 1588 Precision Time Protocol (PTP). In some implementations, the signal 1304 allows the first device 1300 and the second device 1302 to be precisely time-synchronized (e.g., synchronized to within microseconds or nanoseconds of each other).

[0278] FIG. 13A further illustrates a first radar-equipped device 1300 utilizing radar via wireless signal 1306-1 and a second radar-equipped device 1302 utilizing radar via wireless signal 1306-2. In some implementations, synchronization between the first device 1300 and the second device 1302 is utilized to combine radar data from the first device 1300 and the second device 1302. Combining the radar data can provide greater accuracy and precision in determining the location and motion of objects. For example, according to some implementations, the first device 1300 and the second device 1302 each include a one-dimensional radar system. In this example, combining the radar data collected at each device produces results that are comparable to or better than a single multi-dimensional radar system located at either device in terms of accuracy and / or precision. Thus, synchronizing multiple radar-equipped devices can allow the system to use lower dimensional radar data at each device. This allows for the use of radar systems (e.g., 1D radar systems) that are capable of producing accurate and precise results. 1D radar systems generally require a smaller physical footprint (e.g., smaller antennas), consume less operating power, and require less processing power from the device than higher dimensional radar systems (e.g., 2D radar systems).

[0279] In some implementations, combining radar data between the devices includes analyzing a radio signal received by the first device that corresponds to a radio signal transmitted by the second device. For example, the second device emits a short duration radio transmission (sometimes referred to as a chirp). The first device receives a short duration radio transmission (e.g., a radio signal The radar information from both devices may include analyzing a first wireless signal received by the first device that corresponds to the wireless signal transmitted by the first device and analyzing a second wireless signal received by the second device that corresponds to the wireless signal transmitted by the second device. For example, at a first time, the first device transmits a short-term wireless transmission and receives a corresponding wireless signal. Then, at a second time, the second device transmits its own short-term wireless transmission (e.g., utilizing the same or substantially the same wireless frequency) and receives a corresponding wireless signal. In this example, radar information from both devices is combined and analyzed.

[0280] In some implementations, multiple radar-equipped devices are synchronized to enable multi-dimensional radar processing. For example, a radio signal transmitted by a first device (e.g., using a first transmitter) is received by multiple receivers, including a receiver at the first device and a receiver at a second device. By utilizing multiple receivers at separate locations, the system can determine the location of an object with greater accuracy and precision. For example, the angle between the transmitter and the object can be determined based on scattering patterns received across multiple receivers.

[0281] In some implementations, the system includes multiple synchronized radar-equipped devices (e.g., device 1300 and device 1302). The system determines which device should transmit a wireless signal. In some implementations, the system determines which device should transmit based on a pre-established schedule. In some implementations, the system determines which device should transmit based on a particular object of interest. For example, the system determines that a first device (e.g., device 1302) should transmit because it is the radar-equipped device closest to the object of interest. In another example, the system determines that a second device (e.g., device 1300) should transmit because it has a clear view of the object. In some implementations, the system includes a hub device (e.g., hub device 180), which manages the radar operations of the multiple devices (e.g., makes the decision as to which device should transmit at a given time). In some implementations, the system includes a server (e.g., a server of server system 508). The server manages the radar operations of the multiple devices (e.g., makes decisions regarding which device should transmit at a given time). In some implementations, the multiple synchronized devices include one device designated as a master device. In some implementations, the master device manages the radar operations of the multiple devices. In some implementations, processing and analysis of radar data from the multiple devices is performed at the master device.

[0282] In some implementations, the distance between synchronized devices is determined by radio signals. For example, the distance between device 1300 and device 1302 is determined using signal 1304. In some implementations, the distance between synchronized devices is determined based on the travel time of the wireless signal.

[0283] In some implementations, in a first mode, the system cycles between multiple synchronized radar-equipped devices, with each device taking turns transmitting. For example, a first device transmits at time 0. A second device transmits at time 5. A third device transmits at time 10. In some implementations, the system analyzes the wireless signal received with each transmission to determine if the transmission includes an object of interest. For example, the system determines that a detected object is an object of interest based on the object's location or motion, such as the object's position, velocity, direction of movement, and / or acceleration. In this example, the system optionally determines that (1) the location of the object is within a zone of interest or meets one or more other predefined criteria (e.g., within 20 feet, 10 feet, or 5 feet of the transmitting device), (2) the velocity of the object meets one or more predefined criteria (e.g., above a predefined threshold such as 1 meter per second, 1 foot per second, or 1 inch per second), (3) the direction of movement of the object meets one or more predefined criteria (e.g., toward the zone of interest or toward the transmitting device), or (4) the acceleration of the object meets one or more predefined criteria (e.g., above a predefined threshold). In some implementations, following a determination that the detected object is an object of interest, the system switches to a second mode. In some implementations, in the second mode, the system determines that a first device of the multiple devices should transmit based on the relative positions and field of view of the multiple devices. For example, the system determines that the first device is closest to the object of interest and / or has a clear view of the object of interest. In some implementations, the first device transmits a wireless signal that is received by at least a subset of the plurality of devices.In some implementations, the received signals are analyzed to determine the location and movement of the object with greater precision and / or accuracy than when the object of interest was initially detected and / or determined to be an object of interest. In some implementations, the system utilizes multiple-input and multiple-output (MIMO) methods, such as phase shifting in the transmit antennas, to increase resolution.

[0284] 13B illustrates an exemplary system and process for utilizing radar data from multiple radar-mounted devices, according to some implementations. At a first time, a radar-mounted device 1300 is synchronized with a radar-mounted device 1302 using a signal 1304. In some implementations, the radar-mounted device 1300 is synchronized with the radar-mounted device 1302 using the IEEE 1588 precision time protocol in conjunction with the signal 1304.

[0285] At a second time, the device 1300 transmits (1314) a radio signal 1306-1. In some implementations, the device 1300 transmits the radio signal 1306-1 using radar circuitry. The device 1300 receives (1318) a radio signal corresponding to the transmitted radio signal 1306-1. For example, the device 1300 receives a radio signal including radio waves from the transmitted radio signal that bounces off various objects and back to the device 1300. The device 1300 generates (1320) first radio information about the received radio signal. In some implementations, the device 1300 analyzes a relationship between the transmitted radio signal 1306-1 and the received radio signal to determine the location and / or movement of the various detected objects. In some implementations, the device 1300 generates timing and phase information for the received wireless signal.

[0286] Device 1302 receives the transmitted wireless signal 1306-1 (1316). In some implementations, device 1302 receives the transmitted radio wave signal 1306-1 via its radar circuit. Device 1302 generates and transmits second wireless information about the received wireless signal (1322). In some implementations, the generated second wireless information includes the timing and / or phase information of the received wireless signal. In some implementations, the generated second wireless information includes information about the positions and / or movements of various detected objects. In some implementations, device 1302 transmits the second wireless information via a wireless communication channel such as Wi-Fi or Bluetooth.

[0287] Device 1300 receives the transmitted second wireless information (1324). In some implementations, device 1300 receives the transmitted second wireless information via a wireless communication channel such as Wi-Fi or Bluetooth. Device 1300 processes the first and second wireless information to determine the positions and movements of various objects within the signal range (1326). For example, device 1300 compares the first and second wireless information with the transmitted wireless signal 1306-1 to detect objects and determine their positions and movements.

[0288] Radar-equipped smart device Figures 14A-14C show block diagrams of a radar system according to some implementations. In some implementations, the radar systems shown in Figures 14A-14C are some of the components of smart device 204. For example, according to some implementations, components of smart device 204 illustrated in Figure 9A, such as communication module 942, radar module 944, and / or radar processing module 9324, include one of the radar systems shown in Figures 14A-14C.

[0289] FIG. 14A shows a block diagram of a one-dimensional radar system according to some implementations. The one-dimensional radar system in FIG. 14A includes a radar control module 1400, a wireless transmitter 1402, and a wireless receiver 1404. In some implementations, the one-dimensional radar system is configured such that the transmitter 1402 and the receiver 1404 are spaced apart and spaced apart at a known distance on a single plane. In some implementations, the transmitter 1402 includes an antenna, such as a stamped metal antenna, an adhesive antenna (e.g., a stick-on antenna or a tape antenna), a trace antenna on the surface of a printed circuit board (also referred to as a PCB antenna or a substrate antenna), a chip antenna, or a ceramic antenna. In some implementations, the receiver 1404 includes an antenna, such as a stamped metal antenna, an adhesive antenna, a trace antenna, a chip antenna, or a ceramic antenna.

[0290] 14B and 14C show block diagrams of a multi-dimensional radar system. FIG. 14B shows a multi-dimensional radar system with a radar control module 1410, a wireless transmitter 1412, and wireless receivers 1414 and 1416. In some implementations, the wireless transmitter 1412 and the wireless receivers 1414 and 1416 each include an antenna configured for one or more radio frequency bands. For example, the transmitter 1412 emits a wireless signal, such as a tone or pulse (e.g., a 77 GHz tone). The receivers 1414 and 1416 receive radio waves corresponding to the emitted radio signals. The control module 1410 compares the timing and phase of the received radio waves with the emitted radio signals to determine the location and / or motion of various detected objects. By analyzing the difference between the radio waves received at the receiver 1414 and the radio waves received at the receiver 1416, the detection The position / movement of the detected object is determined with greater accuracy and / or precision.

[0291] 14C shows a multi-dimensional radar system with a radar control module 1420, wireless transmitters 1422, 1424, and 1426, and wireless receivers 1428, 1430, 1432, and 1434. In some implementations, the wireless transmitters 1422, 1424, and 1426, and the wireless receivers 1428, 1430, 1432, and 1434 are all on the same plane perpendicular to the dimension of interest. In some implementations, the wireless transmitters 1422, 1424, and 1426, and the wireless receivers 1428, 1430, 1432, and 1434 each include an antenna configured for one or more radio frequency bands. For example, first, the transmitter 1422 emits a wireless signal. The receivers 1428, 1430, 1432, and 1434 receive radio waves corresponding to the emitted wireless signal. The control module 1420 compares the timing and phase of the received radio waves with the emitted radio signals to determine the location and / or movement of the various detected objects. By analyzing the differences between the radio waves received at each receiver, the location / movement of the detected objects is determined with greater accuracy and / or precision. The transmitter 1424 then emits a radio signal and the process is repeated. The transmitter 1426 then emits a radio signal and the process is repeated again. By using multiple transmitters and multiple receivers at known distances from each other, more accurate and more precise results can be obtained. By using multiple receivers, it is also possible to track multiple objects moving simultaneously. For example, to accurately track N objects, generally at least N+1 receivers are required.

[0292] In some implementations, the radar systems described herein utilize continuous wave radar. In some implementations, the radar system utilizes unmodulated continuous wave radar to detect moving objects based on the Doppler effect. In some implementations, the radar system utilizes frequency-modulated continuous wave radar (FMCW), such as sawtooth frequency modulation or sinusoidal frequency modulation. The radar system utilizes FMCW to determine the distance and movement of objects. For example, according to some implementations, the frequency is modulated across frequency bands such as 3 GHz to 10 GHz, 24 GHz to 24.5 GHz, 57 GHz to 64 GHz, and / or 77 GHz to 81 GHz. In some implementations, the radar system utilizes FMCW to determine the speed of various objects based on the phase shift or phase shift ratio in the radar signal reflected from the objects. In some implementations, the radar system utilizes phase shifting modulation continuous wave radar (PMCW). In one implementation, a radar system uses PMCW to detect the position and movement of an object by generating pseudo-random phase shift sequences.

[0293] In some implementations, the radar system described herein uses a relatively low power radar transmitter because the radar transmitter is deployed in or in close proximity to a smart home environment (e.g., smart home environment 100 as shown in FIG. 1 ). In this case, long-range radar detection is not required. For example, according to some implementations, the radar transmitter consumes less than 1 watt of power during idle or periodic scanning modes. In some implementations, the radar transmitter includes a low power radar transmitter configured to consume less than a threshold amount of power (e.g., less than 1 W, 100 mW, or 10 mW). In some implementations, the operation of the radar transmitter is constrained to a limited range based on a characteristic size (e.g., area, volume, width, depth, height, etc.) of the smart home environment 100 as a whole or a space within the smart home environment 100 (even compared to the native range of the radar transmitter). In some implementations, the radar transmitter range is suppressed by using a radar tag as described herein (e.g., radar tag 1202 as described in connection with FIGS. 12A-12D). In some implementations, the radar transmitter range is suppressed by a server. The radar signal processing operation performed by the system 508 or the electronic tag 206 acts to prevent signals from returning from objects that are spaced greater than a predefined distance from the transmitter. In some implementations, the operation of the radar transmitter is suppressed to identify stationary or slowly moving objects when such transmitters are used in a smart home environment to identify people (adults and children) and pets that generally move relatively slowly. In some implementations, the radar system described herein is used to detect anomalies of objects, people and / or animals within the smart home environment 100. For example, anomalies detectable by the radar system in some implementations include abnormal behavior of people within the smart home environment, such as an individual who has collapsed, an individual whose breathing is unstable, an individual who is in the home after the smoke alarm has been activated, an unrecognized individual (e.g., an intruder), or a child who is in a restricted area of ​​the home, such as near the entrance to the garage or pool.

[0294] Various types of internal antennas are optionally used with various devices disclosed herein. An internal antenna may also be referred to as an embedded antenna. An internal antenna as used herein includes any antenna located within a device casing. An internal antenna is generally not implemented within a metal case as such may reduce efficiency. The internal antenna is coupled to a communication control module (e.g., control module 1400, 1410, or 1420). The communication control module may also be referred to as a communication circuit, a radio module, or a radio. In some implementations, the communication control module includes a communication chip. In some implementations, the internal antenna is coupled to the communication control module via one or more controlled impedance structures, such as coaxial cables, controlled impedance circuit board traces (e.g., microstrip or stripline), spring-loaded pogo pins or spring fingers, flex circuits with controlled impedance lines, etc. In some implementations, the internal antenna is coupled to the communication control module via one or more filters, amplifiers, and / or switches. Cables and traces can introduce losses and should be considered carefully, for example, they provide opportunities for noise to enter the receiver system.

[0295] Internal antennas are generally susceptible to interference with other device components (including other internal antennas). In some cases, the primary noise source is digital circuitry such as the processor and memory. For example, in some cases, the processor clock, high speed memory, display and graphics processors are the highest noise sources and generate the widest range of frequencies. In some implementations, the digital electronics are shielded with a shielding can on which the board is mounted. In some implementations, the antenna is placed as far away as possible from the source of the greatest noise. In some implementations, the antenna interconnect is routed away from the source of the greatest noise. In some implementations, the unshielded antenna interconnect (e.g., spring fingers or pogo pins) is positioned to limit exposure to the source of the greatest noise.

[0296] The sheet metal antenna may also be referred to as a stamped metal antenna. In some implementations, the sheet metal antenna is mounted to a circuit board and coupled to a communication module. In some implementations, the sheet metal antenna is mounted perpendicular to the surface of the circuit board. In some implementations, the sheet metal antenna is mounted parallel to the surface of the circuit board. In some implementations, the sheet metal antenna includes an inverted-F antenna. In some implementations, the sheet metal antenna includes a patch antenna. In some implementations, the patch antenna is a printed patch antenna. In some implementations, the patch antenna is printed on a surface of a multi-layer circuit board. In some implementations, The patch antenna comprises a directional antenna with its main lobe of radiation directed away from the ground plane of the device.

[0297] The size and shape of the local ground plane, and the relatively close spacing of the ground plane to the antenna element each affect the antenna design. Sheet metal antennas are optimally placed on the edge of a ground plane, such as the edge of a circuit board, or on a flat ground plane. Thus, sheet metal antennas are optimally not surrounded by ground planes and / or other conductive surfaces.

[0298] A substrate antenna is also called a printed circuit board (PCB) antenna. or PCB trace antenna. In some implementations, the substrate antenna is mounted on a circuit board and coupled to a communication module. The substrate antenna is generally influenced by the substrate properties of the circuit board, such as the dielectric constant and dissipation factor. In some implementations, the substrate antenna includes a single-ended antenna. In some implementations, the substrate antenna includes a differential antenna. In some implementations, the substrate antenna includes a Yagi antenna. In some implementations, the substrate antenna includes an F antenna. In some implementations, the substrate antenna includes an inverted-F antenna. In some implementations, the substrate antenna is laminated on a surface of the circuit board. In some implementations, the substrate antenna occupies one or more layers on the circuit board.

[0299] Chip antennas are typically mounted on printed circuit boards. In some implementations, chip antennas are placed on circuit boards similar to standard circuit components, but these antennas typically have reduced efficiency. Chip antennas are typically affected by the substrate properties of the circuit board, such as dielectric constant and dissipation factor. The substrate material of the circuit board shortens the resonant length of the antenna, which in turn reduces the available bandwidth. The circuit board also introduces loss mechanisms that reduce the efficiency of the antenna. In some cases where the substrate space available for the antenna is limited, the chip antenna is the antenna type of choice. In some implementations, the chip antenna includes a ceramic chip antenna. In some implementations, the chip antenna includes an F antenna. In some implementations, the chip antenna includes an inverted F antenna. In some implementations, the chip antenna includes a monopole antenna.

[0300] In some implementations, the adhesive antenna is attached to a cover (sometimes referred to as a casing or housing). In some implementations, the adhesive antenna includes a tape antenna. In some implementations, the adhesive antenna includes a stick-on antenna. In some implementations, the adhesive antenna includes a conductive paint antenna. In some implementations, the adhesive antenna includes a wire antenna.

[0301] To reduce noise while optimizing performance, the radar system (e.g., radar antenna and circuitry) is typically positioned such that the radar system is not obscured by other components. For example, in a device with multiple circuit boards, the radar system is typically positioned on the circuit board closest to the cover in the direction the radar system is intended to monitor. Metal casings and other similar materials that generate electromagnetic fields are typically not positioned in front of the radar system as they can introduce noise and / or reduce detection range. In some implementations, the radar system is positioned within the smart device to minimize impedance mismatch for transmitted waves leaving / entering the device. In some implementations, the radar system is positioned within the smart device to minimize interference to other communication systems by positioning antennas remote from each other (e.g., by placing antennas on both ends of a circuit board). ... the surface of interest (e.g., by placing antennas on both ends of a circuit board). It is positioned parallel to the device cover facing a particular area (or dimension of interest) of interest to optimize resolution in that area (or dimension of interest).

[0302] In a device with both Wi-Fi and radar systems, such as radar-equipped smart device 204, interference between Wi-Fi and radar signals is optionally minimized by configuring the device such that radar frequencies are not harmonics of Wi-Fi frequencies, or vice versa.

[0303] In devices with both a radar system and a temperature sensor, such as the radar-equipped thermostat 102, interference between the temperature sensor and the radar system is optionally minimized by configuring the device so that the temperature sensor does not operate simultaneously with the radar or immediately after radar operation. In some cases, operation of the radar system generates enough heat to cause inaccurate temperature sensor readings in a smart home environment. In some implementations, the duty cycle for the radar system is configured so that the radar does not transmit radio waves just before the temperature sensor performs a temperature reading.

[0304] 14D is a block diagram illustrating a radar control module 1450 according to some implementations. The radar control module 1450 is coupled to a number of antennas 1448 for transmitting and receiving radio waves via a receive line 1452 and a transmit line 1454. In some implementations, the radar control module 1450 includes an analog front end 1452 coupled to the antennas 1448.

[0305] In some implementations, the analog front end 1452 is coupled to a number of analog-to-digital converters (ADCs) 1454 for converting analog signals received from the antennas to digital signals. In this example, the plurality of ADCs 1454 includes a first plurality of ADCs for calculating an imaginary component of the received signal and a second plurality of ADCs for calculating a real component of the received signal.

[0306] In some implementations, the analog front end 1452 is coupled to a transmit module 1456. In some implementations, the transmit module 1456 is configured to generate various wireless signals to be transmitted by the antenna 1448. In some implementations, the transmit module 1456 is configured to sample signals received from the analog front end that are used to generate wireless signals to be transmitted in the future. In some implementations, the transmit module 1456 includes a digital-to-analog converter (DAC) 1457 and a power amplifier 1459. In some implementations, the transmit module 1456 includes a signal mixer and / or an oscillator. In some implementations, the transmit module 1456 is configured to construct a programmable radar signal to be transmitted by the antenna 1448.

[0307] In some implementations, the ADC 1454 is coupled to a controller 1458 for processing of the digital signal. In some implementations, the controller 1458 performs digital signal processing (DSP), such as a fast Fourier transform (FFT), on the digital signal. For information about the 1458 controller, see the ARM Limited website. This includes multi-core processors such as ARM® processors.

[0308] In some implementations, the radar control module 1450 includes a storage 1462. In some implementations, the storage is for storing data from the controller 1458 and / or the transmission module 1456. In the embodiment, the storage is used to store data received from the application processor 1464. In some implementations, the storage includes a number of registers. In some implementations, the storage includes volatile and / or non-volatile memory.

[0309] In some implementations, the radar control module 1450 includes a data merger 1460 coupled to the ADCs 1454. In some implementations, the data merger 1460 is configured to aggregate the data output by the individual ADCs for subsequent processing.

[0310] In some implementations, the radar control module 1450 includes a power sub-module (not shown) for providing power to various components of the radar control module 1450.

[0311] In some implementations, the radar control module 1450 is coupled to an application processor 1464 and outputs data from the data merger 1460 and / or the controller 1458 for further processing by the application processor 1464. In some implementations, the control module 1400, the control module 1410 and / or the control module 1420 include the radar control module 1450.

[0312] FIG. 14E is a block diagram illustrating an application processor 1464 according to some implementations. FIG. 14E illustrates an application processor 1464 coupled to a radar control module 1450, a video source 1466, storage 1468, and a categorizer 1480. In some implementations, the application processor 1464 is a component of a radar-equipped device, such as the smart device 204. In some implementations, the application processor 1464 is a component of a server system, such as the server system 508. In some implementations, the application processor 1464 includes an event processor 7146 and a radar processor 7145. In some implementations, the categorizer 1480 includes an event categorizer 1107 and / or a radar categorizer 1108. In some implementations, the video source 1466 includes the video source 1101. In some implementations, the radar control module 1450 is a component of the radar source 1101.

[0313] In some implementations, application processor 1464 includes an image signal processor (ISP) 1472 for processing images received from video source 1466. In some implementations, processing the image includes determining pixel color, adjusting exposure, adjusting focus, correcting lens imperfections, reducing noise, etc.

[0314] In some implementations, the application processor 1464 includes a signal processing module 1470 for processing data received from the radar control module 1450 and / or the image processor 1472. In some implementations, the signal processing module 1470 performs transformations of the data, such as a Fast Fourier Transform, to make the data easier to collect and analyze.

[0315] In some implementations, the application processor 1464 includes a number of controllers 1474 for analyzing the radar and / or image data. In some implementations, the controller 1474 identifies objects of interest in the radar and / or image data. In some implementations, the controller 1474 processes the identified objects, categories assigned to the identified objects, and / or Machine learning is performed by analyzing clusters of identified objects.

[0316] In some implementations, the categorizer 1480 categorizes objects of interest identified by the controller 1474. In some implementations, the categorizer 1480 categorizes radar and / or motion events that include one or more objects of interest identified by the controller 1474.

[0317] In some implementations, the application processor 1464 is coupled to the storage 1468. In some implementations, the storage 1464 includes volatile and / or non-volatile memory. In some implementations, the storage 1464 includes DDR memory. In some implementations, the storage 1464 stores data output by the ISP 1472, the controller 1474, and / or the categorizer 1480. For example, the storage 1464 stores objects of interest identified by the controller 1474. In some implementations, the storage 1464 includes the video and radar source data database 1109, the categorization module database 1110, and / or the event data and event mask database 1111.

[0318] 15A-15M show various assembly views of a camera device 118 according to several implementations. In particular, these implementations show the arrangement of components as part of a component layout strategy to improve coexistence. The camera 118 (e.g., when the camera is a smart device 204 (FIG. 9A)) and components of various implementations are described in more detail throughout.

[0319] 15A and 15B show perspective views of an assembled camera device 118. The camera device 118 captures multimedia data (image data, video data, and audio data) in real time and communicates raw or processed multimedia data to its user via a remote server. The captured raw multimedia data is optionally processed locally to the camera device 118 or remotely in a remote server (e.g., server system 508). In some implementations, the camera device 118 is radar-mounted and includes a radar system, such as one of the radar systems of FIGS. 14A-14C. In some implementations, the radar-mounted camera device 118 captures radar data and communicates raw or processed radar data to a remote server system (e.g., server system 508) for analysis and / or storage. 15A illustrates a camera device 118 with one or more radios 1597 that support the Wi-Fi communication band and one or more radios 1599 that support the 15.4 communication band. In some implementations, the camera device 118 includes radios, circuitry, and / or modules for implementing radar. In some implementations, the camera device 118 includes radios, circuitry, and / or modules for implementing RFID communications.

[0320] FIG. 15C shows a view of the camera device 118 with the cover 1502 removed from the camera device 118 assembly. FIG. 15C shows a Wi-Fi antenna 1595 and a 15.4 antenna 1593 mounted inside the cover 1502. FIG. 15C also shows an area 1501 on the front side of the interior of the cover. In some implementations, a radar system, such as one of the radar systems of FIGS. 14A-14C, is positioned at least partially within the area 1401 (e.g., a radar antenna is positioned). FIGS. 15D and 15E further show the camera device 118 and the position where one or more antennas (e.g., 1504-1 and 1504-2) are positioned relative to the cover 1502 of the camera device 118 assembly. As shown in FIG. 15D, cables (such as For example, an RF coaxial cable) connects antenna 1504-2 (e.g., a 15.4 antenna) to the main circuit board of camera device 118 (illustrated in more detail in FIGS. 15H and 15I). Although not shown, in some implementations antenna 1504-1 (e.g., a Wi-Fi antenna) is also connected to the main circuit board using a similar cable. In some implementations, camera 118 includes one or more additional antennas for communicating via other wavelengths and / or protocols, such as RFID or radar. For example, one or more radar antennas are mounted to cover 1502 and / or to a circuit board near cover 1502.

[0321] The antennas (e.g., antenna 1504 and radar antenna) in the camera device 118 assembly are positioned so that interference from conductive and transmitting components is minimized. The placement of the antennas is based at least in part on the type of antenna used. In some implementations, the antennas are optionally integrated into or secured by adhesive or tape (or alternatively secured using a non-conductive adhesive). By selecting a less conductive (or no conductive) material, such as plastic, as the material for the closed housing (e.g., cover 1502) of the camera assembly, the antennas are optionally adhesively secured along the interior of the closed housing, thereby reducing the effects of interference that they would otherwise suffer. In some implementations, some or all of the antennas are stamped sheet metal antennas. The antennas are optionally configured as three-dimensional sheet metal antennas mounted on a circuit board near the location of the corresponding transceiver. As an example, the stamped Planar Inverted-F Antenna (PIFA) In some implementations, some or all of the antennas are metal antennas embedded in / on a printed circuit board (PCB). In some implementations, the traces are laminated on the surface of the PCB. In some implementations, the traces occupy several layers of a multi-layer board. In some implementations, some or all of the antennas are chip antennas, passive surface mounted antenna components, optionally constructed from ceramic. In some implementations, some or all of the antennas are outside the closed housing of the camera assembly.

[0322] FIG. 15F is an exploded view of camera device 118 according to some implementations. This view of camera device 118 shows camera lens 1508, multiple LEDs 1506, and one or more antennas 1504. In some implementations, camera device 118 includes a radar system (e.g., the radar system of FIGS. 14A-14C) positioned at least partially on circuit board 1507, such as in region 1503. For example, the radar system has one or more antennas positioned above or below LEDs 1506 (e.g., in region 1503) on circuit board 1507. In some implementations, camera device 118 includes a radar system positioned at least partially on circuit board 1509. For example, the radar system is positioned such that one or more antennas are located near the periphery of circuit board 1509 (e.g., in region 1511 or region 1513) to have a field of view that is unobstructed by circuit board 1507. In some implementations, the camera device 118 includes a radar system mounted at least partially on the cover 1502. For example, the radar system is mounted such that one or more antennas are located on a forward-facing portion of the cover 1502 (e.g., in the area 1501).

[0323] According to the periodic monitoring mode, the camera device 118 is configured to provide video surveillance and security in a smart home environment illuminated by a visible light source (e.g., the sun or a light bulb). In some implementations, the camera device 118 is configured to provide video surveillance and security in a smart home environment illuminated by a visible light source (e.g., the sun or a light bulb). The camera device 118 includes alternative operating modes, such as a vision mode and a depth imaging mode. Each of the alternative operating modes is associated with a respective lighting condition. For example, in the night vision mode, the camera device 118 is configured to capture activities in the smart home environment at night when there is no or limited visible light illumination. In the depth imaging mode, the camera device 118 is configured to create a depth map or image for a corresponding field of view in the smart home environment. The depth map can then be used in a periodic monitoring mode to accurately identify objects in the smart home environment. In some implementations, the depth image is created based on one or more images captured when a portion of the field of view is selectively illuminated. Thus, in some implementations, the camera device 118 is configured to include an LED lighting system and use it as an internal light source to provide illumination in the smart home environment according to the respective lighting conditions associated with each of the alternative operating modes of the camera device 118. In some implementations, the camera device 118 is configured to perform radar surveillance in the smart home environment. In some implementations, camera device 118 is configured to perform radar monitoring regardless of the illumination-based operational mode of the camera device.

[0324] Specifically, in some implementations, the LEDs 1506 include infrared LEDs 1506. The infrared LEDs 1506 are enclosed within a dark, infrared-transparent plastic cover 1502 of the camera device 118, making them invisible from outside the camera device 118. Assuming that the plastic cover 1502 allows infrared light to pass through, the camera device 118 can rely on the infrared LEDs 1506 to provide illumination at night. In a night vision mode, the LEDs are powered on to illuminate the field of view with infrared light at night. The camera device 118 includes an infrared image sensor that captures infrared images or video clips of the field of view. In a depth imaging mode, the LEDs 1506 are grouped into several LED sets, each of which is selectively powered to illuminate a respective portion of the field of view associated with the scene in which the camera device 118 is located. The images captured associated with these LED sets are combined to generate a depth map of the entire field of view at the scene.

[0325] Alternatively, in some implementations, the plurality of LEDs 1506 is a combination of infrared and visible LEDs including at least one infrared LED and at least one visible LED. In night vision mode, at least one infrared LED of the plurality of LEDs is turned on to illuminate the field of view with infrared light.

[0326] In some implementations, the LEDs 1506 are arranged in an internal assembly structure of the camera device 118 and are configured to surround the camera lens 1508 of the camera device 118. In some implementations, as shown in the inset of FIG. 15F, each LED is optionally tilted at an angle with respect to an optical axis passing through the center 340 of the camera lens 1508, where the optical axis is perpendicular to the lens surface at the center of the camera lens 1508. In some implementations, each LED is tilted away from the optical axis of the camera at an angle ranging from 20 degrees to 40 degrees. In this example, the LEDs 1506 include eight LEDs grouped into four pairs of LEDs. The four pairs of LEDs are symmetrically arranged in four quadrants surrounding the camera lens 1508. In some implementations, the mechanical or electrical components 314 are arranged between two LED pairs or between two LEDs in an LED pair. It is noted from the above examples that the camera lens 1508 can be surrounded by several LEDs having different physical arrangements (e.g., the camera lens 1508 is surrounded by 100 LEDs uniformly distributed in three layers surrounding the camera lens 1508).

[0327] In some implementations, the antennas (e.g., antenna 1504 and / or radar antenna) are mounted on the inner wall of the camera cover 1502 according to the form factor of the camera device 118. Each of the antennas is positioned in a respective preferred position with a respective preferred orientation relative to the other antennas to reduce or eliminate interference from the presence and interference of the other antennas. In some situations, the receiving element and base assembly are made of a metallic material, and each of the antennas is positioned in a respective preferred position with a respective preferred orientation relative to the receiving element and base assembly to reduce or eliminate interference from their presence. To achieve the preferred position and orientation, each of the antennas is marked with a respective alignment mark configured to guide the assembly of the respective antenna on the inner wall of the camera cover 1502.

[0328] In some implementations, the one or more antennas include at least a first antenna and a second antenna. The first antenna transmits data captured by the camera device 118 over a wireless local area network (WLAN). N). The second antenna is configured to communicate configuration data associated with the WLAN over the first wireless network for the purpose of delegating the camera device 118 to the WLAN. In some implementations, the camera device 118 further includes a third antenna. The third antenna enables the camera device 118 to communicate with a local hub device (e.g., hub device 180 shown in FIG. 1) over a second wireless network different from the first wireless network. In some implementations, the camera device 118 further includes a third antenna to enable the camera device 118 to perform radar operations. In some implementations, the WLAN and the first and second wireless networks are associated with custom or standard wireless protocols (e.g., IEEE802.15.4, Wi-Fi, ZigBee, 6L0WPAN, Thread, Z-Wave, Bluetooth Smart, ISA100.11a, WirelessHART, MiWi, etc.). In a specific example, the first antenna, the second antenna, and the third antenna are configured to wirelessly transfer respective data according to the IEEE 802.11 standard, the IEEE 802.15.4 standard, and the ZigBee standard, respectively. In another example, the first antenna, the second antenna, and the third antenna are configured to wirelessly transfer respective data via IEEE 802.11, RFID, and radar, respectively.

[0329] In some implementations, the antennas are electrically coupled to separate radio transmitter circuits that operate in separate frequency bands and simultaneously transmit radio signals in the separate frequency bands, hi some implementations, the antennas are electrically coupled to a duplex filter or switch that controls their connection to a single radio receiver circuit.

[0330] 15G-15M show additional views of camera device 118. In particular, FIG. 15G shows an exploded view of camera device 118. FIGs. 15H and 15I show top and bottom views, respectively, of the main circuit board of camera device 118. FIGs. 15J and 15K show top and bottom views, respectively, of the sensor board of camera device 118. FIGs. 15L and 15M show top and bottom views, respectively, of the IR LED circuit board of camera device 118.

[0331] In some implementations, the camera device 118 includes a front cover 1573, a back cover 1591, a sensor assembly 1571, a sensor board 1585, an LED board 1507, a main board 1509, a speaker 1581, a microphone 1577, and mounting screws 1565. In some implementations, the main board 1509 includes a board-to-board connector 1589 for coupling the main board to the sensor board 1585, a memory 1555 (e.g., NAND memory) and , a processor 1557 (e.g., an Amberella A5S CPU) for image processing, motion detection, compression, etc. In some implementations, the sensor board 1585 includes a heat sink 15 53. In some implementations, the LED controller 1567 is coupled (e.g., mounted) to the LED substrate 1507. In some implementations, the back cover 1591 includes a plurality of ridges 1559 to reduce thermal touch sensation. In some implementations, the back cover 1591 includes a metal cover.

[0332] In some implementations, the camera assembly includes a closed housing having a back surface 1591, a front surface 1573, and a periphery (e.g., camera 118 (FIGS. 15A-15M)). In some implementations, the camera assembly further includes a lens module 1571. The lens module 1571 is located within the housing and configured to receive light via a front surface (e.g., lens 1508 (FIG. 15F)). In some implementations, video data captured by the lens module is communicated between the camera assembly and a server using a wireless local area network.

[0333] In some implementations, the camera assembly further includes a communication circuit. The communication circuit is located within the housing and is configured to communicate wirelessly via a plurality of different communication protocols (e.g., a main circuit board 1509 of the camera device 118 including a plurality of transceivers). In some implementations, the communication circuit includes a first circuit 1579 configured to communicate via a first communication protocol (e.g., a Wi-Fi transceiver) and a second circuit 1587 configured to communicate via a second communication protocol (e.g., an 802.15.4 transceiver). In this case, the first circuit and the second circuit are separate. In some implementations, the communication circuit is an integrated circuit configured to communicate via both the first and second communication protocols (e.g., a radio 950 including a radio 950-1 for Bluetooth LE and a radio 950-2 for Wi-Fi). In some implementations, the communication circuit is configured to communicate wirelessly via at least three or more different communication protocols (e.g., Wi-Fi, 802.15.4, and Bluetooth LE). In some implementations, the second antenna is further configured to communicate via a third one of the communication protocols. In some implementations, the communication circuitry is further configured to provide transmission access for communication via the first communication protocol and deny transmission access for communication via the second communication protocol while detecting an activated priority control signal. In some implementations, the communication circuitry includes a first circuitry configured to communicate via the first communication protocol (e.g., RFID) and a second circuitry configured to communicate via radar. In some implementations, the communication circuitry is at least partially located on multiple circuit boards (e.g., the main board 1509 and the LED board 1507).

[0334] In some implementations, the camera assembly further includes a first antenna (e.g., antenna 1504-1) disposed at a first location on the inner surface of the periphery and configured to communicate via a first one of the communication protocols (e.g., Wi-Fi), and a second antenna (e.g., antenna 1504-2) disposed at a second location on the inner surface of the periphery different from the first location and configured to communicate via a second one of the communication protocols (e.g., 802.15.4). In some implementations, the first antenna is disposed at the first location on the inner surface of the periphery and / or the second antenna is disposed at the second location on the inner surface of the periphery by adhesive (e.g., FIG. 15E). In some implementations, the first antenna has a first orientation and the second antenna has a second orientation to reduce interference from each other's antennas (e.g., antennas 1504-1 and 1504-2 are disposed relative to each other and to other antennas within the camera assembly). (orienting the first antenna and the second antenna at different angles relative to the component). In some implementations, the first antenna and the second antenna are marked with respective alignment marks configured to guide assembly of the respective antennas on the inner wall of the camera lid.

[0335] In some implementations, the first antenna and the second antenna are configured to operate at the same frequency (e.g., 2.4 GHz). In some implementations, the first antenna and the second antenna are configured to operate at separate frequencies (e.g., 2.4 GHz and 5 GHz). In some implementations, the first antenna is configured to transmit and receive signals including one or more of warnings, control signals, and status information to and from other smart home devices via a first communication protocol (e.g., emergency warnings from the hazard detector 104 using 802.15.4, or any signal transmitted / received by any of the smart devices 204). Furthermore, the second antenna is configured to transmit and receive signals for configuring the smart home devices via a second communication protocol (e.g., Bluetooth LE for provisioning and setting up the camera 118), and transmit and receive data corresponding to video captured by the smart home devices via a third communication protocol (e.g., Wi-Fi for streaming video data). In some implementations, the third antenna is configured to transmit and receive wireless signals used in radar.

[0336] In some implementations, the first antenna or the second antenna is configured to transmit and receive radio signals in a wireless local area network according to the IEEE 802.11 standard. In some implementations, the first antenna or the second antenna is configured to transmit and receive radio signals in a wireless personal network according to the IEEE 802.15.4 standard. In some implementations, the first antenna or the second antenna is configured to transmit and receive radio signals according to the Bluetooth Low Energy (BLE) standard. In some implementations, the first antenna or the second antenna is configured to transmit and receive radio signals used in radar. In some implementations, the first antenna and the second antenna are electrically coupled to a duplex that controls their connection to a single radio receiver circuit.

[0337] In some implementations, the camera assembly further includes a loop arm configured to hold the closed housing when the closed housing is inserted into a notched opening in the loop arm, and in some implementations, the loop arm is made of a metallic material, and the first position of the first antenna and the second position of the second antenna are configured to reduce the effect of the presence of the metallic loop arm (e.g., located at a position furthest from a position where the loop arm contacts the closed housing).

[0338] In some implementations, the radar system included in the camera device 118 is located on top of the main board 1509, such as one of the radar systems of FIGS. 14A-14C. For example, in some implementations, the radar system is located in a lower region on the top side of the main board (e.g., below the sensor board connector). In some implementations, the radar system is located in an upper region on the top side of the main board (e.g., region 1511 or region 1513). In some implementations, the radar system is located in several regions. For example, a control module (e.g., control module 1420) is located in a first region (e.g., region 1511) and a transmitter and / or receiver is located in a second region (e.g., region 1513).

[0339] In some implementations, the camera device 118 includes a radar system, such as one of the radar systems of FIGS. 14A-14C, located on top of a sensor substrate. For example, in some implementations, the radar system is located in an upper region on the top side of the sensor substrate (e.g., above the image sensor). As another example, the radar system is disposed at least partially on the sensor substrate in region 1515 and / or region 1517.

[0340] In some implementations, camera device 118 includes a radar system, such as one of the radar systems of Figures 14A-14C, and is located on top of LED substrate 1507. For example, in some implementations, the radar system is located in the upper left region of the LED substrate (e.g., to the left of the LEDs).

[0341] In some implementations, the camera device 118 includes a radar system located on a circuit board (e.g., a flex circuit board) connected to the main board 1509, the LED board 1507, and / or the sensor board.

[0342] In some implementations, the radar system is positioned such that the control module (e.g., control module 1400) is located on a first component of the camera device 118 and the antenna (e.g., transmitter 1402 and / or receiver 1404) is located on a second component of the camera device 118. In some implementations, the radar system is positioned such that the control module (e.g., control module 1400) is located on a first circuit board (e.g., in area 1511 on main board 1509) and the antenna is located on a second circuit board (e.g., in area 1504 on LED board 1507). In some implementations, one or more of the antennas are located on a flex circuit board or are mounted to cover 1502 (e.g., in area 1501) of the camera device 118.

[0343] 15H illustrates a top view of the main board 1509 according to some implementations. In some implementations, the main board 1509 includes an antenna connector 1583 (e.g., a Wi-Fi or radar antenna connector), a communication module 1579 (e.g., to operate with the antenna connector 1583), a sensor board connector 1589, a PMIC 1551, a communication module 1587 (e.g., a 15.4 or radar communication module), and an antenna connector 1549 (e.g., a 15.4 or radar antenna connector). In some implementations, the main board 1509 includes a radar module mounted in area 1511 or area 1513. In some implementations, the main board 1509 includes a radar antenna mounted in area 1511 or area 1513.

[0344] 15I shows a bottom view of the main board 1509 according to some implementations. In some implementations, the main board 1509 includes a processor 1557, memory 1555, a speaker cavity 1547, a reset button 1545, a microphone connector 1543, a USB connector 1541, and an antenna connector 1563 (e.g., a radar or 15.4 antenna connector).

[0345] 15J shows a top view of a sensor board 1585 according to some implementations. In some implementations, the sensor board 1585 includes RGB status LEDs 1539, an inductor 1535, a boost 1533, an integrated circuit (IC) 1531, and an image sensor 1537. In some implementations, the sensor board 1585 further includes a radar module mounted in region 1515 or region 1517. In some implementations, the sensor board 1585 further includes a radar antenna mounted in region 1515 or region 1517.

[0346] FIG. 15L shows a top view of an LED substrate 1507 according to some implementations. In some implementations, the LED board 1507 includes a current sinking operational amplifier (OpAmp) 1529. further includes a radar module mounted in region 1503. In some implementations, sensor board 1585 further includes a radar antenna mounted in region 1503.

[0347] 15M shows a bottom view of the LED board 1507 according to some implementations. In some implementations, the LED board 1507 includes a depth mode controller integrated circuit 1527.

[0348] 16A-16C show various views of a smart thermostat (eg, smart thermostat 102 (FIG. 1)) optionally used as part of a smart home environment 100, as previously described.

[0349] Specifically, FIG. 16A shows an exploded perspective view of the smart thermostat 102 into two components: head unit 1600-A and backplate 1600-B. Head unit 1600-A includes head unit circuit board 1650 (described in further detail in connection with FIG. 16B), and backplate 1600-B includes backplate circuit board 1660 (described in further detail in connection with FIG. 16C). Further technical and / or functional descriptions of the various electrical and mechanical components illustrated herein below may be found in one or more of commonly-assigned incorporated applications, such as U.S. Application Serial No. 13 / 199,108. In the figures shown, the "z" direction is outward from the wall, the "y" direction is head-to-toe up-down for a walking user, and the "x" direction is side-to-side for the user.

[0350] FIG. 16B shows a front view of a head unit circuit board 1650 for a smart thermostat that includes a head unit microprocessor 1602 (such as a Texas Instruments AM3703 chip) and associated oscillator 1604, along with memory 1606 (e.g., DDR SDRAM) and storage 1608 (e.g., mass NAND storage). For wireless communication capabilities (e.g., Wi-Fi), a wireless communication module 1612 is provided (e.g., in a separate section of a radio frequency (RF) shield 1610), such as a Murata Wireless Solutions LBWA19XSLZ module based on the Texas Instruments WL1270 chipset supporting the 802.11b / g / n WLAN standard. Support circuitry 1614 is provided for the wireless communication module 1612, including an oscillator 1616. According to some implementations, for ZigBee functionality, a ZigBee module 1612 is provided (e.g., a ZigBee module such as a C2530F256 module by Texas Instruments). 18 is provided (e.g., in a separately shielded RF section). Support circuitry 1620 is provided for the ZigBee module 1618, including an oscillator 1622 and a low noise amplifier 1624. According to some implementations, the smart thermostat also includes a display backlight voltage conversion circuit 1626, a piezoelectric drive circuit 1628, and / or a power management circuit 1630 (such as a local power rail). In some implementations, a proximity and ambient light sensor (PROX / ALS), such as a Silicon Labs SI1142 proximity / ambient light sensor with an I2C interface, is provided on a circuit board 1632 (e.g., a flex circuit board) attached to the backside of the head unit circuit board by a flex circuit connector 1634. In some implementations, the smart thermostat includes one or more of a battery charging-monitoring-disconnect circuit 2432 and a spring / RF antenna 1638. In some implementations, the smart thermostat also includes one or more of a temperature sensor 1644 (standing perpendicular to the circuit board in the +z direction, housing two separate temperature sensing elements at different distances from the circuit board) and a PIR motion sensor 1646. In some implementations, the PIR motion sensor 1646 (and associated circuitry) is similar to the radar of FIGS. 14A-14C. In some implementations, the radar system may be replaced with a radar system such as any of the systems. In some implementations, the smart thermostat includes a radar system in addition to the PIR motion sensor 1646. In some implementations, the radar system is located at least partially on the head unit circuit board 1650. For example, according to some implementations, the radar system is located on the top bottom region of the head unit circuit board 1650 (e.g., next to or in place of the PIR motion sensor 1646). In some implementations, the radar system is located at least partially on a flex circuit board coupled to the head unit circuit board 1650 (e.g., in addition to or in place of the PROX / ALS circuitry).

[0351] In some implementations, even though the PROX / ALS and temperature sensor 1644 and the PIR motion sensor 1646 are physically located on the head unit circuit board 1940, all of these sensors are polled and controlled by a low-power backplate microcontroller on the backplate circuit board to which they are electrically connected. In some implementations, the head unit circuit board includes a Bluetooth module 1648 and additional circuitry (not shown) including one or more oscillators, amplifiers, and / or other support circuitry. In some implementations, the head unit circuit board includes one or more integrated circuits including a combination radio and transceiver. For example, in some implementations, the wireless communication module 1612 and the Bluetooth module 1648 include a single chip. In this case, the wireless communication module 1612 and the Bluetooth module 1648 transmit and receive signals using a single antenna 1638. Various implementations of the transceivers (e.g., radio 940 and radio 950) are described in more detail in connection with FIG. 9A.

[0352] FIG. 16C shows a rear view of a backplate circuit board 1660 including a backplate processor / microcontroller 1662, such as a Texas Instruments MSP430F system-on-chip microcontroller, including on-chip memory 1664. The backplate circuit board 1660 further includes a power circuit 1666, optionally including a power steal circuit, and a switch circuit 1668 for switching between the respective HVAC functions. In some implementations, for each HVAC function, the switch circuit 1668 includes an isolation transformer 1670 and a back-to-back NFET package 1672. The use of FETs in the switching circuit allows for "active power steal," i.e., drawing power during an HVAC "ON" cycle, by temporarily diverting power from the HVAC relay circuit to a storage capacitor for a very short interval, such as 100 microseconds. This time is too short to trip the HVAC relay to the OFF state, but is sufficient to charge the storage capacitor. The use of FETs allows for such fast switching times (100s of microseconds), which would typically be difficult to achieve using relays (which last for tens of milliseconds). Such relays also typically degrade this type of fast switching implementation and may generate audible noise. In contrast, according to some implementations, FETs operate without or essentially without audible noise. In some implementations, the backplate circuit board 1660 also includes a combination temperature / humidity sensor module 1274, such as a Sensirion SHT21 module. In some implementations, the backplate microcontroller 1662 performs one or more of the following functions: polling various sensors, detecting mechanical wire insertion during installation, alerting the head unit of current vs. set point temperature conditions and activating switches accordingly, and other functions such as searching for the appropriate signal on the inserted wire during installation.

[0353] Commonly owned U.S. serial application Ser. No. 13 / 269,501, commonly owned U.S. serial application Ser. No. 13 / 275,307, and other applications incorporated by reference in their entirety by the same applicant. In accordance with the teachings therein, the smart thermostat 102 (FIG. 1) represents an advanced, multi-sensor, microprocessor-controlled, intelligent or "learning" thermostat that provides a rich combination of processing power, an intuitive and visually pleasing user interface, network connectivity, and energy saving capabilities (including the auto-away / auto-arrival algorithms described herein) while simultaneously not requiring line power from the so-called "C-wire" from the HVAC system or from the household wall plug. In some cases, such advanced functionality may require a greater instantaneous power draw than a "power steal" option (i.e., drawing a smaller amount of power from one or more HVAC call relays) can safely provide. As an example, in some environments the head unit microprocessor 1602 draws on the order of 250 mW while powered up and processing, and the LCD module (not shown) draws on the order of about 250 mW during operation. Additionally, the Wi-Fi module 1612 needs to draw 250mW during operation and operate in a consistent manner, such as at a constant 2% duty cycle, in common scenarios. However, to avoid falsely tripping HVAC relays for many commercially used HVAC systems, the power steal circuitry is limited to a power supply capacity on the order of 100mW to 200mW, which may not be sufficient to provide the power needed for many common scenarios.

[0354] In some implementations, the smart thermostat 102 solves such problems by using at least a rechargeable battery (or an on-board power storage medium of equivalent capacity) that will recharge during time intervals when hardware power usage is below what can be safely provided by power theft and will discharge to provide the required extra power during time intervals when hardware power usage exceeds what can be safely provided by power theft. To operate in a battery-conscious manner that promotes reduced power consumption and extended service life of the rechargeable battery, the thermostat 1800 is provided with both (i) a relatively powerful and relatively power-intensive first processor (such as a Texas Instruments AM3703 microprocessor) that can quickly perform more complex functions such as driving a visually pleasing user interface display and performing various mathematical learning calculations, and (ii) a relatively less powerful and less power-intensive second processor (such as a Texas Instruments MSP430 microcontroller) for performing less intensive tasks, including driving and controlling the occupancy sensor. In some implementations, to conserve power, the first processor is kept in a "sleep" state for long periods of time and "woke up" only when its performance is needed, while the second processor is kept on more or less continuously to perform its relatively low power work (although preferably slowed down or has certain internal clocks disabled for short periodic intervals to conserve power). The first and second processors are mutually configured to be able to "wake" the first processor upon the occurrence of some event, which may be referred to as a "wake on" mechanism. In some implementations, these wake on mechanisms may be turned on and off as part of various functional and / or power saving goals to be achieved.For example, a "wake on PROX" mechanism may be provided whereby the second processor will "wake up" the first processor with an active proximity sensor (such as PROX provided by a Silicon Labs SI1142 Proximity / Ambient Light Sensor with I2C Interface) when it detects a user's hand approaching the thermostat dial so that it can provide a visual indication to an approaching user and be ready to react more quickly when the user's hand touches the dial. As another example, a "wake on PIR" mechanism may be provided whereby the second processor will wake up the PerkinElmer DigiPyr when it detects motion in the general vicinity of the thermostat. o A passive infrared (PIR) motion sensor such as provided by the PYD 1998 Dual Element High Temperature Detector will wake up the first processor. Notably, wake-on-PIR is not synonymous with auto-arrival as there may need to be N consecutive buckets of sensed PIR activity to invoke auto-arrival, whereas only a single sufficient motion event can cause a wake-on-PIR wakeup.

[0355] In some implementations, the smart thermostat 102 utilizes a radar system, such as one of the radar systems in FIGS. 14A-14C, to detect an approaching user and / or hand contact and wakes up the first processor based on the radar system detection.

[0356] In some implementations, smart thermostat 102 includes a radar system, such as one of the radar systems of Figures 14A-14C, located on top of head unit 1600-A. For example, in some implementations, the radar system is located at least partially within region 1601 inside the head unit cover. In some implementations, the radar system is located in an area above the head unit cover.

[0357] In some implementations, the smart thermostat 102 includes a radar system, such as one of the radar systems of Figures 14A-14C, located on the top of the head unit circuit board 1650. For example, in some implementations, the radar system replaces the PIR sensor 1646 on the bottom of the head unit circuit board 1650. As another example, the radar system is located at least partially adjacent to the PIR sensor 1646 (e.g., to the left or right of the PIR sensor 1646).

[0358] In some implementations, the smart thermostat 102 includes a radar system that is located at least partially on a circuit board (e.g., a flex circuit board), such as a flex circuit board 1632 connected to the head unit circuit board 1650 or the backplate circuit board 1660.

[0359] In some implementations, the radar system is positioned such that the control module (e.g., control module 1400) is located on a first component of the smart thermostat 102 and the antenna (e.g., transmitter 1402 and / or receiver 1404) is located on a second component of the smart thermostat 102. In some implementations, the radar system is positioned such that the control module (e.g., control module 1400) is located on a first circuit board (e.g., in area 1677 on the backplate circuit board 1660) and the antenna is located on a second circuit board (e.g., in place of PIR 1646 on the head unit circuit board 1650). In some implementations, one or more of the antennas are located on a flex circuit board (e.g., flex circuit board 1632) or are mounted to a cover of the smart thermostat 102 (e.g., in area 1601).

[0360] 17A-17D show various views of a smart hazard detector (e.g., smart hazard detector 104 (FIG. 1)) that may be used as part of a smart home environment 100, as previously described. According to some implementations, the views also show a number of holes or openings in a cover plate 1728 that form a visually pleasing design that is visible to an occupant of the space in which the smart hazard detector 104 is mounted.

[0361] Specifically, Figure 17A shows an assembly diagram of the smart hazard detector 104. 1 shows the mounting plate 1724, front casing 1726 and cover plate 1728 in an assembled configuration along with various other components housed within the interior space of the smart hazard detector 104. According to some implementations, a lens button 1732 is shown attached to the smart hazard detector 104 such that it is centrally positioned relative to the cover plate 1728. As outlined, a light ring 1742 is optionally used to create a halo of light around and behind the lens button 1732. The assembled smart hazard detector 104 comprises a compact yet multi-function device.

[0362] FIG. 17B shows an exploded perspective view of the smart hazard detector 104. As shown in FIG. 17B, the smart hazard detector 104 includes a mounting plate 1724. The mounting plate 1724 is attached to a wall or ceiling of a structure to secure the smart hazard detector 104 thereto. The smart hazard detector 104 also includes a backplate 1730 attached to the mounting plate 1724 and a front casing 1726. The front casing 1726 is coupled to or secured to the backplate 1730 to define a housing having an interior region in which the components of the smart hazard detector 104 are housed. In some implementations, a circuit board 1700 is coupled to or attached to the backplate 1730. Various components are optionally mounted on the circuit board 1700. For example, according to some implementations, a smoke chamber 1734 is coupled to or mounted on the circuit board 1700 and configured to detect the presence of smoke. In some implementations, the smoke chamber 1734 is mounted near the center with respect to the circuit board 1700 such that air flows into the smoke chamber 1734 from above and below the circuit board 1700. In some implementations, a speaker 1736 and / or alarm is mounted on the circuit board 1700 to audibly alert occupants of a potential fire hazard when the presence of smoke is detected by the smoke chamber 1734. Other components such as motion sensors, carbon monoxide sensors, microprocessors, etc. are optionally mounted on the circuit board 1700 as described herein. In some implementations, the smart hazard detector 104 includes a radar system or module, such as one of the radar systems of FIGS. 14A-14C, for transmitting and receiving radar signals. In some implementations, the smart hazard detector 104 includes a communication module for communicating with other smart devices and / or electronic tags (e.g., via RFID or Wi-Fi).

[0363] In some implementations, the smart hazard detector 104 includes a radar system at least partially mounted on the circuit board 1700. For example, one or more radar antennas are located on the circuit board 1700. In some implementations, the smart hazard detector 104 includes a radar system at least partially mounted on the front casing 1726. For example, one or more radar antennas are located on or attached to the front casing 1726. In some implementations, the smart hazard detector 104 includes a radar system at least partially mounted on the cover plate 1728. For example, one or more radar antennas are located on or attached to the cover plate 1728.

[0364] In some implementations, a protective plate 1738 is attached to or coupled with the circuit board 1700 to provide a visually pleasing appearance to the internal components of the smart hazard detector 104 and / or to direct or direct airflow into the smoke chamber 1734. For example, when a user views the internal components of the smart hazard detector 104 through a vent in the backplate 1730 or the like, 17. In this embodiment, the protective plate 1738 presents the appearance of a relatively smooth surface that would otherwise conceal the components or circuitry of the circuit board 1700. The protective plate 1738 optionally functions to direct airflow from the vents in the backplate 1730 to the smoke chamber 1734 to facilitate the flow of air into and out of the smoke chamber 1734.

[0365] In some implementations, the smart hazard detector 104 includes a battery pack 1740 configured to power various components of the smart hazard detector 104. For example, the smart hazard detector 104 provides power when not coupled to an external power source, such as a 120V power source of the home or structure. In some implementations, a cover plate 1728 is coupled to the front casing 1726 to provide a visually pleasing appearance to the smart hazard detector 104 and / or for other functional purposes. In some implementations, the cover plate 1728 includes a number of holes or openings. The number of holes or openings allows one or more sensors coupled to the circuit board 1700 to observe or view through a surface of the cover plate 1728 to detect objects external to the smart hazard detector 104. In some implementations, the number of openings in the cover plate 1728 are positioned to provide a visually pleasing appearance when viewed by an occupant of the home or structure. In some implementations, the multiple openings in the cover plate 1728 are arranged according to a repeating pattern, such as a Fibonacci or other sequence.

[0366] In some implementations, the lens button 1732 is coupled to or attached to the cover plate 1728. The lens button 1732 optionally allows one or more sensors to be viewed through the lens button 1732 for various purposes. For example, in some implementations, a passive IR sensor (not shown) is positioned behind the lens button 1732 and configured to be viewed through the lens button 1332 to detect the presence of an occupant or occupants in the home or structure. In some implementations, a radar system is positioned behind the lens button 1732 and configured to be viewed through the lens button 1332 to detect the presence of an occupant or occupants in the home or structure. In some implementations, the lens button 1732 also functions as a button operable by a user to input various commands to the smart hazard detector 104, such as to silence alerts triggered in response to false or harmless conditions. In some implementations, a light ring 1742 is positioned distally behind the lens button 1732 and configured to receive light from an LED or the like and disperse the light within the ring 1742 to create a desired visual phenomenon, such as a halo, behind the lens button 1732. In some implementations, a flexible circuit board 1744 is positioned distally behind the light ring 1742 and includes one or more electrical components, such as a passive IR sensor, an LED, a radar circuit, etc. The flexible circuit board 1744 (sometimes referred to as a flex ring 1744) may be electrically coupled to the circuit board 1700 for communicating instructions to and / or receiving instructions from one or more microprocessors mounted on the circuit board (not shown) during operation of the smart hazard detector 104.

[0367] 17C and 17D show front and rear perspective views of a circuit board 1700 of a smart hazard detector (e.g., smart hazard detector 104 (FIG. 1)). The circuit board 1700 includes a body 1702 having a front or front side or a rear or back side. As described herein, various electrical components are mounted on the circuit board 1700. In some implementations, these components are mounted on the front side of the circuit board 1700, on the back side opposite the front side of the circuit board 1700, or on both sides of the circuit board 1700. For example, according to some implementations, one or more microprocessors and / or other processor-related components are mounted on a back side of the circuit board 1700 facing the protective plate, while one or more functional components (e.g., an alarm, a CO detector, a speaker, a motion sensor, a Wi-Fi device, a Zigbee device, a Bluetooth device, an RFID device, a radar device, etc.) are mounted on a front side of the circuit board 1700 facing the space of the home or building in which the smart hazard detector 104 is positioned. In some implementations, other components are mounted near the center of the circuit board 1700 such that opposing sides are positioned on either side of the circuit board 1700, as described herein. In some implementations, the circuit board 1700 includes one or more integrated circuits. Each of the one or more integrated circuits includes a combination radio and transceiver. For example, in some implementations, the Wi-Fi device and the Bluetooth device of the smart hazard detector 104 include a single chip, and the Wi-Fi device and the Bluetooth device use a single antenna to transmit and receive signals. Various implementations of a transceiver (eg, radio 940, radio 950) are described in more detail in connection with FIG. 9A.

[0368] As shown in FIG. 17C, in some implementations, the front side of the circuit board 1700 includes a CO detector 1704. The CO detector 1704 is configured to detect the presence of carbon monoxide gas and trigger an alarm 1706 when the carbon monoxide gas level is too high to be determined (e.g., above a predefined threshold). The alarm 1706, optionally including a piezoelectric buzzer that emits a deliberately sharp or harsh sound, is mounted on the front side of the circuit board 1700 to face the occupants of the space in which the smart hazard detector 104 is located to alert the occupants of the potential danger. In some implementations, the alarm 1706 is configured to generate one or more sounds and / or signals to alert the occupants of the potential danger. In some implementations, the front side of the circuit board 1700 includes an area 1708 in which a speaker (not shown) is positioned. In some implementations, the speaker is configured to give an audible warning and / or message to the occupants of the room. For example, the speaker is configured to warn the occupant of potential danger and instruct the occupant to leave the room. In some implementations, the speaker provides the occupant with specific instructions, such as an exit route to use when leaving the room and / or the home or building. In some implementations, other messages are communicated to the occupant to, for example, warn the occupant of a low battery, relatively high CO levels in the room, that the smart hazard detector 104 needs periodic cleaning, and / or to warn the occupant of any other anomaly or problem with the hazard detector 104 or its components.

[0369] In some implementations, the circuit board 1700 includes one or more motion sensors mounted on its front surface. The motion sensors are optionally used to determine the presence of an individual in the room or surrounding area of ​​the smart hazard detector 104. In some implementations, the circuit board 1700 includes a radar system or components thereof mounted on its front surface to determine the presence of an individual in the room or surrounding area. In some implementations, this information is used to modify the functionality of the smart hazard detector 104 and / or one or more other devices connected in a common network, as described above. For example, according to some implementations, this information is relayed to a smart thermostat to notify the thermostat that an occupant of the home or structure is present so that the smart thermostat may adjust the home or structure according to one or more learned or programmed settings. In some implementations, the smart hazard detector 104 may be configured to generate an alarm (e.g., in response to a gesture for silence) as described herein or for various other reasons. This information is used for one or more purposes, such as shutting down and quieting the device.

[0370] In some implementations, a first ultrasonic sensor 1720 and a second ultrasonic sensor 1722 are mounted on the front side of the circuit board 1700. The two ultrasonic sensors 1720 and 1722 are optionally axially offset to face slightly different directions. In this orientation, each ultrasonic sensor is optionally used to detect the movement of an individual based on the orientation of the smart hazard detector 104 relative to the room and / or occupant. In some implementations, the detection of the movement of the individual is used to deactivate and silence an alarm as described herein or for other reasons. In some implementations, the axis of the first ultrasonic sensor 1720 is oriented substantially outward relative to the smart hazard detector 104, and the axis of the second ultrasonic sensor 1722 is oriented at an angle relative to the axis of the first ultrasonic sensor 1720. In some implementations, the first ultrasonic sensor 1720 is configured to detect the movement of an individual when the smart hazard detector 104 is mounted on the ceiling of a home or building. Because the first ultrasonic sensor 1720 is oriented substantially outward relative to the smart hazard detector 104, the first ultrasonic sensor 1720 essentially looks directly down at an individual beneath the smart hazard detector 104. In some implementations, the second ultrasonic sensor 1722 is configured to detect movement of an individual when the smart hazard detector 104 is mounted on a wall of a home or building. Because the second ultrasonic sensor 1722 is oriented at an angle relative to the first ultrasonic sensor 1720 and the smart hazard detector 104, the second ultrasonic sensor essentially looks down toward the floor when the smart hazard detector 104 is mounted on a wall of a home or building, rather than looking directly outward as the first ultrasonic sensor 1720. In some implementations, the offset angle between the two ultrasonic sensors is approximately 30°.

[0371] In some implementations, the two ultrasonic sensors 1720 and 1722 are replaced with a single ultrasonic sensor that is configured to rotate within the smart hazard detector 104 so that it can point straight out like the first ultrasonic sensor 1720 and point downward like the second ultrasonic sensor 1722. In some implementations, the single ultrasonic sensor is coupled to the circuit board 1700 via a hinge that allows the ultrasonic sensor to rotate based on the orientation of the smart hazard detector 104. For example, if the smart hazard detector 104 is mounted to the ceiling of a home or building, gravity will orient the ultrasonic sensor to point straight down, whereas if the smart hazard detector 104 is coupled to a wall of a home or building, gravity will rotate the ultrasonic sensor via the hinge to point down at the floor relative to the smart hazard detector 104. In some implementations, a motor is coupled to the single ultrasonic sensor to rotate the ultrasonic sensor based on the orientation of the smart hazard detector 104. In this embodiment, the ultrasonic sensor is pointed in a direction that is likely to detect movement of individuals within the room or space surrounding the smart hazard detector 104, regardless of the orientation of the smart hazard detector. In some implementations, a single ultrasonic sensor has a wide field of view that can accommodate both mounting positions of the smart hazard detector 104.

[0372] 17C and 17D, the body 1710 of the circuit board 1700 also includes a substantially centrally located aperture 1712 through which a smoke chamber is inserted to centrally mount the smoke chamber relative to the circuit board 1700. The aperture 1712 optionally includes a pair of notches 1714 through which wires are inserted to electrically couple the smoke chamber to the circuit board 1700. As previously described, centrally mounting the smoke chamber through the aperture 1712 allows smoke and air to pass through the front of the circuit board 1700, and the smoke chamber is then electrically connected to the circuit board 1700. The smoke chamber can be accessed from the side and the rear side of the circuit board 1700.

[0373] Various aspects of the electrical components on the circuit board 1700 are described herein, although the location of many of them will be apparent to one of ordinary skill in the art in light of the description herein and Figures 17C and 17D. In some implementations, the circuit board 1700 includes one or more of a system processor, a relatively high power wireless communication circuit and antenna, a relatively low power wireless communication circuit and antenna, a non-volatile memory, an audio speaker, one or more interface sensors, a safety processor, a safety sensor, an alarm device 1706, a power source, and a power supply circuit. These components are operable to provide fail-safe safety detection features and user interface features using circuit topologies and power budget organization methods that minimize power consumption. According to some implementations, a branched or hybrid processor circuit topology is used to process various features of the smart hazard detector 104. In this case, the safety processor is a relatively small, relatively thin processor dedicated to core safety sensor management and core alarm functions, such as provided on a traditional smoke / CO alarm, and the system processor is a relatively larger, relatively higher power processor dedicated to more advanced features such as cloud communications, user interface features, occupancy and other advanced environmental tracking features, and more generally other tasks that would not be considered "core" or "traditional" safety detection and alarm tasks.

[0374] As an example, the safety processor is optionally a Freescale KL15 microcontroller, while the system processor is optionally a Freescale K60 microcontroller. Preferably, the safety processor is programmed and configured to operate and perform the safety-related duties of its cores regardless of the status or state of the system processor. Thus, for example, even if the system processor is not available or cannot perform any functions, the safety processor continues to perform the safety-related tasks of its cores, so that the smart hazard detector 104 meets all necessary industry and / or government safety standards for smoke, CO and / or other safety-related monitoring provided for the smart hazard detector 104 (provided there is sufficient power to enable the operation of the safety processor). The system processor, on the other hand, performs "optional" and / or "advanced" functions that overlap with the functions of the safety processor. In this case, "optional" or "advanced" refers to tasks that are not specifically required for compliance with industry and / or government safety standards. Thus, the system processor is designed to interoperate with the safety processor in a manner that may improve the overall performance, features, configuration and / or functionality of the smart hazard detector 104, but that is not necessary for the smart hazard detector 104 to meet core safety-related industry and / or government safety standards. The system processor, being generally a larger and more powerful processor than the safety processor, will generally consume more power than the safety processor when both the system processor and the safety processor are in operation.

[0375] Similarly, if both processors are stopped, the system processor will generally still consume more power than the safety processor. In some implementations, the system processor is operable to manage user interface features and monitor interface sensors (such as motion sensors, audio sensors, cameras, etc. that are not directly related to the core safety detection). For example, according to some implementations, the system processor directs wireless data traffic for both high and low power wireless communication circuitry, accesses non-volatile memory, communicates with the safety processor, and issues audio through a speaker. As another example, according to some implementations, The system processor monitors the interface sensors to determine whether any action needs to be taken (e.g., silencing a sounding alarm in response to a detected user action to silence the alarm). In some implementations, the safety processor is operable to handle the core safety-related tasks of the smart hazard detector 104. In some implementations, the safety processor polls the safety sensors (e.g., smoke, CO) and activates the alarm 1706 if one or more of the safety sensors indicate the detection of a hazard event. In some implementations, the safety processor operates independently of the system processor and activates the alarm 1706 regardless of the state of the system processor. For example, if the system processor is performing an active function (e.g., performing a Wi-Fi update) or is shut down due to power constraints, the safety processor still activates the alarm 1706 upon detection of a hazard event.

[0376] In some implementations, the software running on the safety processor is permanently fixed and prohibited from being updated by software or firmware updates after the smart hazard detector 104 leaves the factory. The safety processor is generally a lower power processor compared to the system processor.

[0377] In some implementations, the relatively high power wireless communication circuitry includes, for example, a Wi-Fi module capable of communicating according to any of the 802.11 protocols. As an example, the relatively high power wireless communication circuitry is optionally implemented using a Broadcom BCM43362 Wi-Fi module. In some implementations, the relatively low power wireless communication circuitry includes a low power wireless personal area network (6L0WPAN) module or a ZigBee module capable of communicating according to the IEEE 802.15.4 protocol. For example, in some implementations, the relatively low power wireless communication circuitry is implemented using an EmberEM357 6L0WPAN module. In some implementations, the non-volatile memory is any suitable permanent memory storage, such as NAND flash, hard disk drive, NOR, ROM, or phase change memory. In some implementations, the non-volatile memory stores an audio clip that is played using the speaker 1736. The audio clip optionally includes installation instructions and / or warnings in one or more languages. In some implementations, the interface sensors include sensors monitored by the system processor, while the safety sensors include sensors monitored by the safety processor. The sensors are optionally mounted on a printed circuit board, a flexible printed circuit board, the housing of the system, or a combination thereof.

[0378] In some implementations, the interface sensor includes one or more of an ambient light sensor (ALS) (such as may be implemented using a discrete photodiode), a passive infrared (PIR) motion sensor (such as may be implemented using an Excelitas PYQ1348 module), a radar system (such as the radar system of FIGS. 14A-14C), and one or more ultrasonic sensors (such as may be implemented using one or more Manorshi MS-P1640H12TR modules). In some implementations, the safety sensor includes one or more of a smoke detection chamber 1734 (e.g., including an Excelitas IR module), a CO detection module 1704 (e.g., including a Figaro TGS5342 sensor), and a temperature and humidity sensor (e.g., including a Sensirion SHT20 module). The power source provides power to enable operation of the smart hazard detector and optionally includes any suitable source of energy. Implementations described herein may be implemented using a combination of AC line powered, battery powered, AC line powered with battery backup, and / or externally powered. Implementations that use AC line power, AC line power with battery backup, or externally supplied DC power are optionally subject to different power conservation constraints than battery-only implementations.

[0379] Preferably, a battery-only powered implementation is configured to manage the consumption of its finite energy supply so that the smart hazard detector 104 operates for a minimum period of time, such as at least 7, 8, 9 or 10 years. A line-powered implementation is less constrained. A battery-backup powered line implementation optionally uses power conservation methods to extend the life of the backup battery. In a battery-only implementation, the power source optionally includes one or more batteries. The batteries are optionally made of a variety of different compositions (e.g., alkaline or lithium iron disulfide), and are optionally in a variety of end-user configurations (e.g., permanent, user replaceable or non-user replaceable). In some implementations, Li-FeS 2 The six cells are arranged in three stacks of two side by side. Such an arrangement can provide a total available power of about 27000 mWh for the smart hazard detector 104.

[0380] In some implementations, the smart hazard detector 104 utilizes a radar system, such as one of the radar systems in FIGS. 14A-14C, to detect an approaching user and / or hand contact.

[0381] In some implementations, the smart hazard detector 104 includes a radar system, such as one of the radar systems of Figures 14A-14C, mounted on a cover plate 1728. For example, in some implementations, the radar system is located at least partially within region 1701 within the interior of the cover plate 1728. In some implementations, the radar system is located in an area above the cover plate 1728.

[0382] In some implementations, the smart hazard detector 104 includes a radar system mounted on the front casing 1726. For example, in some implementations, the radar system is located in an upper region of the front casing 1726.

[0383] In some implementations, smart hazard detector 104 includes a radar system located on top of circuit board 1700. For example, in some implementations, the radar system is located at least partially in area 1708 on top of circuit board 1700. In some implementations, smart hazard detector 104 includes a radar system located on the bottom of circuit board 1700. For example, in some implementations, the radar system is located at least partially in area 1709 on the bottom of circuit board 1700.

[0384] In some implementations, the smart hazard detector 104 includes a radar system located at least partially on a circuit board (e.g., a flex circuit board) connected to the circuit board 1700.

[0385] In some implementations, the radar system is positioned such that the control module (e.g., control module 1400) is located on a first component of smart hazard detector 104 and the antenna (e.g., transmitter 1402 and / or receiver 1404) is located on a second component of smart hazard detector 104. In some implementations, the radar system is positioned such that the control module (e.g., control module 1400) is located in a first region on circuit board 1700 (e.g., region 1709). The smart hazard detector 104 is positioned such that the antennas are positioned in a first region (e.g., in region 1708) and the antennas are positioned in a second region (e.g., in region 1708). In some implementations, one or more of the antennas are located on a flex circuit board or mounted to a cover of the smart hazard detector 104 (e.g., in region 1701).

[0386] In some implementations, a radar system (e.g., as described in FIGS. 14A-14E) is utilized in a smart pool or hot tub to determine if the pool or hot tub is occupied. In some implementations, the determination of occupancy is utilized by the smart pool or hot tub to adjust the operation of the smart pool or hot tub. In some implementations, the determination of occupancy is utilized by the smart pool or hot tub to adjust the water temperature. In some implementations, the determination of occupancy is utilized to generate an alert (e.g., if a child is detected in the pool). In some implementations, the determination of occupancy is utilized by the smart pool or hot tub to adjust lighting levels.

[0387] In some implementations, a radar system (e.g., as described in FIGS. 14A-14E) is used in a smart light to determine if a person is nearby. In some implementations, the presence determination is used by the smart light to adjust the operation of the smart light. In some implementations, the presence determination is used by the smart light to adjust the brightness of the light. In some implementations, the presence determination is used by the smart light to turn the light on or off (e.g., turn the light on when a person is detected).

[0388] In some implementations, a radar system (e.g., as described in Figures 14A-14E) is used in a smart irrigation system to determine if a person is nearby. In some implementations, the presence determination is used by the smart irrigation system to adjust the operation of the irrigation system (e.g., temporarily stop or temporarily redirect irrigation).

[0389] In some implementations, a radar system (e.g., as described in FIGS. 14A-14E) is utilized in a smart camera to detect and / or track objects. In some implementations, detection and tracking is aggregated with image data from an image sensor to improve object detection and tracking. In some implementations, detection and tracking is utilized to detect anomalies and to focus an image sensor on the detected anomalies. In some implementations, detection and tracking is utilized under conditions where image detection and / or tracking is limited, such as when lighting conditions of detected objects from an image sensor are reduced or obstructed (e.g., due to smoke or other objects).

[0390] In some implementations, a radar system (e.g., as described in FIGS. 14A-14E) is used in a smart thermostat to determine if a person is nearby and / or recognize a particular individual. In some implementations, the presence determination is used by the smart thermostat to adjust the operation of the HVAC system (e.g., increase or decrease the target temperature). In some implementations, the recognition of a particular individual is used to individualize HVAC settings.

[0391] In some implementations, a radar system (e.g., as described in FIGS. 14A-14E) is utilized in a smart hazard detector to determine if a person is nearby and / or to recognize a particular individual. In some implementations, the presence determination is utilized by the smart hazard detector to generate an alert. For example, the presence detection is used to notify emergency responders that someone (e.g., a recognized individual) is within a structure that is on fire.

[0392] In some implementations, a radar system (e.g., as described in FIGS. 14A-14E) is utilized in a smart entry device (e.g., a smart doorbell or smart door lock) to determine if a person is nearby and / or to recognize a particular individual. In some implementations, the presence determination is utilized by the smart entry device to generate an alert. For example, the presence detection is used to notify a homeowner that a person has been detected in a hallway. In some implementations, the recognition of a particular individual is utilized to lock and / or unlock a door, gate, or window. In some implementations, the recognition of a particular individual is utilized to set a smart home configuration (e.g., to a personalized configuration for an identified individual). In some implementations, the recognition of a particular individual is utilized to confirm the identity of the individual (e.g., along with data from other sensors and / or other information received from the individual, such as a passcode).

[0393] In some implementations, a radar system (e.g., as described in FIGS. 14A-14E) is utilized in a smart radar imaging device to classify detected objects. In some implementations, classification, location, and / or movement information from the radar system is utilized by the smart radar imaging device to identify what types of objects were detected and where, and display to a user (e.g., on a display screen coupled to the smart radar capture device). For example, a smart radar imaging device may be pointed outside a residence (e.g., in the front yard) and display an image to a user that includes the locations (and types) of detected objects, such as cars, pedestrians, pets, etc.

[0394] 18 is a block diagram illustrating a communication module 942 of a smart device utilizing a bypass circuit, according to some implementations. FIG. 18 illustrates a communication module 942 (e.g., FIG. 9A) of a smart device 204 (e.g., device 102, 104, 106, 108, 110, 112, 114, 116, 118, 120, and / or 122, such as camera 118, smart hazard detector 104, smart thermostat 102, etc.) according to any of the implementations disclosed in FIGs. 1-9.

[0395] In some implementations, the communication module 942 includes one or more radios 940, one or more radios 950, a front-end module 1800, and an antenna 1802. In some implementations, the one or more radios 940 and the radio 950 are communicatively coupled to transmit and receive respective signals via their respective antennas 1802. As an example, the communication module 942 of the camera 118 includes a first transceiver (e.g., Wi-Fi, radio 950) for transmitting video data, and additionally includes a second transceiver (e.g., Zigbee, radio 940) for transmitting and receiving packets from other smart devices in the smart home environment. Optionally, the combined transceiver is configured to transmit and receive data according to more than one communication protocol (e.g., radio 950 configured for Wi-Fi and Bluetooth LE). In some implementations, the radio 950 is utilized for radar communications. and radio 940 is utilized for wireless communications such as Wi-Fi or Bluetooth. The components of communications module 942 (e.g., radio 940, radio 950) are described in more detail in connection with FIG. 9A.

[0396] The front-end module 1800 of the communications module 942 includes various circuit elements for processing (e.g., amplifying) signals transmitted and received by the radio 940. In particular, the front-end module 1800 includes a switch 1804 for coupling the radio 940 to the antenna 1802 by selectively using an amplifier (e.g., amplifier 1806-1 and low-noise amplifier 1806-2) or a bypass line 1808. Whether the front-end module 1800 couples the radio 940 to the antenna 1802 by the low-noise amplifier 1806-2 or the bypass line 1808 depends on whether the radio 950 is active and transmitting signals using the second antenna 1802-2. Although not shown, other implementations of the front-end module 1800 may include additional circuit elements, including one or more additional switches, amplifiers, transistors, and / or any combination of active or passive elements. A combination of circuit elements may be used in the front-end module 1400, for example, to modify the sensitivity of the communications module 942 and signals transmitted and / or received via the radio 940.

[0397] As described above, poor signal isolation typically results from the close physical placement of antennas 1802. In particular, device components are at risk of damage when the signal power of a transmitting antenna in a nearby transmit chain (e.g., antenna 1802-2 for Wi-Fi transmission) exceeds the maximum input power of components in the receive chain (e.g., the 802.15.4 receive chain formed by radio 940 and antenna 1802-1).

[0398] Some methods for enabling and enhancing coexistence of transceiver devices (e.g., radio 940, radio 950) include using a bypass line 1808 in the front-end module 1800. In particular, by using a switch 1804 to automatically and selectively enable the bypass line 1808 when the radio 950 (e.g., Wi-Fi) is “on” and by transmitting a signal via antenna 1802-2 (e.g., radio 950 signals to the front-end module 1400 via connection 1810 that it is “on” and transmitting), transmissions received via antenna 1802-1 do not pass through the low-noise amplifier 1806-2 but instead pass through the bypass line 1808 to the radio 940. As a result, the effective maximum input power that can be tolerated by the front-end module 1800 may be higher, as determined by the characteristics of the bypass line 1808 rather than the low-noise amplifier 1806-2 (e.g., the front-end module 1800 has a higher input power threshold when the bypass line 1808 is used to couple the radio 940 and the antenna 1802-1). When the radio 950 is "off", the switch 1804 operates to couple the radio 940 and the antenna 1802-1 with the low-noise amplifier 1806-2 in the receiver path. This results in the front-end module 1800 having a higher receive sensitivity in terms of receiver chain sensitivity when the low-noise amplifier 1806-2 is used to couple the radio 940 and the antenna 1802-1. Assuming that the low noise amplifier 1806-2 is selectively activated based on the activity of the radio 950, the use of the low noise amplifier 1806-2 improves sensitivity while avoiding the risk of damaging the low noise amplifier 1806-2. Thus, the front end module 1800 transmits the low noise amplifier 1806-2 via the bypass line 1808 when the radio 950 is active and transmitting a signal using the antenna 1802-2 such that a signal received via the antenna 1802-1 is not amplified by the amplifier 1806-2 until it reaches the radio 940. The front-end module 1800 is configured to couple the antenna 1802-1 to the radio 940 via the amplifier 1806-2 when the radio 950 is not transmitting signals using the antenna 1802-2, such that signals received via the antenna 1802-1 are amplified by the amplifier 1806-2 before reaching the radio 940.

[0399] 19A and 19B show coexistence schemes for smart devices with multiple radios in a smart home environment 100 according to some implementations.

[0400] As shown in FIG. 19A, the smart device 204 (e.g., the camera 118) transmits and receives data to other nearby devices and / or electronic tags (e.g., the smart hazard detector 104, the camera 118, the smart thermostat 102...

Claims

1. 1. A method comprising:

1. A radar and radio enabled electronic device having one or more controllers and a memory, comprising: receiving a request from a remote electronic device via a standard wireless communication protocol to enable transmission of information of the radar and wireless enabled electronic device using radar technology, the request including device identification information, the method further comprising: determining whether the received device identification information matches device identification information for the radar and wireless enabled electronic device, the device identification information for the radar and wireless enabled electronic device being stored in the radar and wireless enabled electronic device, the method further comprising: pursuant to determining that the received device identification information matches the device identification information for the radar and wireless enabled electronic device, causing a signal indicative of a position and / or a velocity of the radar and wireless enabled electronic device to be transmitted pursuant to using one or more radar techniques.

2. receiving a radar signal from the remote electronic device; 2. The method of claim 1, wherein the step of transmitting a signal includes the step of modulating and reflecting the received radar signal such that a preset phantom velocity is present in the reflected radar signal.

3. operating in a low power mode prior to receiving the request to enable; and operating in a radar mode in accordance with a determination that the received device identification information matches the device identification information for the radar and wireless enabled electronic device, wherein operating in the radar mode consumes more power than operating in the low power mode.

4. An electronic device equipped with radar and radio, a first component configured to communicate with one or more remote electronic devices via a standard wireless communication protocol, said first component comprising: one or more antennas configured to communicate at a first frequency corresponding to the standard wireless communication protocol; a first controller coupled to the one or more antennas; The first controller includes: managing the one or more antennas to transmit signals in accordance with the standard wireless communication protocol; receiving a command from a remote device via the one or more antennas; determining whether the command was intended for the radar and radio onboard electronic devices; pursuant to a determination that the command was intended for the radar and radio onboard electronic devices, acting on the command; and configured to ignore the command in response to a determination that the command was not intended for the radar and radio on-board electronic device, the radar and radio on-board electronic device further comprising: a second component configured to communicate position and / or velocity information of the radar and wireless on-board electronic device to the one or more remote electronic devices using one or more radar techniques, the second component comprising: one or more second antennas configured to communicate over the radar band; and a radar controller coupled to the one or more second antennas and the first controller, the radar controller configured to manage the one or more second antennas to communicate signals according to the one or more radar techniques; the command includes a recipient identification, and the first controller determining whether the command was intended for the radar and radio onboard electronic device includes comparing the recipient identification to an identifier of the radar and radio onboard electronic device.

5. 5. The radar and radio onboard electronic device of claim 4, wherein the commands include a command to enable radar communications, and the first controller acting in accordance with the commands includes the first controller communicating an enable command to the radar controller.

6. 6. The radar and radio onboard electronic device of claim 4 or 5, wherein the commands are encrypted, the first controller is further configured to decrypt the commands, and the radar controller is further configured to encrypt information transmitted via the one or more second antennas.

7. 7. The radar and radio onboard electronic device of claim 4, wherein the first controller further comprises a decoder configured to decode signals received via the one or more antennas.

8. 8. The radar and radio onboard electronic device of claim 4, further comprising an energy storage component coupled to the first component and the second component, the energy storage component configured to selectively supply power to the first component and the second component.

9. The energy storage component comprises: one or more capacitors; one or more batteries; a circuit configured to harvest energy from a signal received via an antenna of the electronic tag; and a circuit configured to harvest thermal energy, seismic energy, electromagnetic energy and / or solar energy received by the electronic tag.

10. The radar controller is further configured to selectively enable radar communications via the one or more second antennas, the radar communications comprising: Reflecting the received radio waves, Amplifying the received radio waves; modulating the received radio waves; and 10. The radar and radio onboard electronic device of claim 4, further comprising at least one of: a.

11. The second component further comprises: a modulator configured to modulate radio waves received via the one or more second antennas; A radar and radio onboard electronic device according to any one of claims 4 to 10, comprising at least one of: a signal generator configured to generate radio waves at a particular frequency.

12. and one or more additional sensors coupled to the first controller, the one or more additional sensors comprising: a humidity sensor configured to sense humidity around the radar and the wireless on-board electronic device; a temperature sensor configured to sense one or more of a temperature of the radar and wireless on-board electronic device and an ambient temperature surrounding the radar and wireless on-board electronic device; an accelerometer configured to sense acceleration of the radar and wireless on-board electronic device; a gyroscope configured to sense the orientation of the radar and the wireless on-board electronic device; and an optical sensor configured to sense ambient light around the radar and wireless on-board electronic device.

13. The first component comprises: the one or more antennas; and the one or more second antennas; and 13. The radar and wireless onboard electronic device of claim 4, wherein the device information includes one or more of a device identifier, one or more device settings, information about the surroundings of the radar and wireless onboard electronic device, and information about operating conditions of the radar and wireless onboard electronic device.

Images