Event-based recording

Abstract

To provide a video clip related to a target event.SOLUTION: An electronic device includes an image sensor, one or more processors, and a memory that stores an instruction, the instruction includes receiving an event recording profile on the basis of configuration data of the electronic device, the configuration data including a position type or a power type, receiving a plurality of images of a scene acquired by the image sensor, detecting a trigger event on the basis of one or more of the plurality of images of the scene, identifying a target object from one or more of the plurality of images of the scene in response to detecting the trigger event, and creating an event clip from the stored images including the target object, and creating the event clip includes setting a clip length on the basis of the event recording profile and providing the event clip for display.SELECTED DRAWING: Figure 1A

Description

[Technical Field]

[0001] Related Applications This application claims priority to U.S. Provisional Patent Application No. 62 / 897,233, filed September 6, 2019, which is incorporated herein by reference in its entirety.

[0002] Technical Field TECHNICAL FIELD This application relates generally to electronic devices, including but not limited to cameras and electronic assistants, that provide video clips related to target events while providing power enhancements and bandwidth savings. [Background technology]

[0003] background Streaming devices are becoming increasingly popular. As the number of streaming devices increases, the demand for streaming increases, creating concerns about bandwidth limitations. For example, a home equipped with security cameras and streaming entertainment services can easily exceed the maximum monthly bandwidth allotment set by a residential Internet service provider, especially if these devices stream high-definition video data 24 hours a day. Summary of the Invention [Problem to be solved by the invention]

[0004] In addition to bandwidth issues, streaming consumes a lot of power. Power-hungry streaming devices increase electricity bills, and high power budgets hinder the ability of devices to shrink in size and become portable. Meanwhile, as electronic devices are designed to be more compact and mobile, it becomes difficult to continuously support power-hungry activities such as continuous video streaming.

[0005] Proposed solutions to the bandwidth and power problems caused by continuous streaming applications include targeted streaming approaches. However, limiting the scenarios in which a streaming device can acquire data and transmit it over a network creates various tradeoffs in device functionality. For example, to save bandwidth and power, a security camera can be designed to minimize recording and streaming, but this reduces camera usage and risks missing important security-related events. [Means for solving the problem]

[0006] overview Therefore, there is a need for a streaming system and / or device that reduces the impact on device functionality and conserves bandwidth and power in a more efficient, accurate, and intuitive manner. Such systems, devices, and methods complement or replace conventional systems, devices, and methods for event identification, classification, and / or presentation by optimizing device functionality and improving specific device operations, as needed.

[0007] The concepts described herein include the use of dynamic formulations, which change themselves based on the device's placement, the intended use of the device, and adaptation as the device learns its environment over time. Dynamic formulations can include adjustable parameters such as padding (e.g., recording time before and after detecting a target object), inactivity (e.g., time to wait before terminating an event instead of continuing the event to include subsequent activity), maximum length (e.g., For example, it is used for targeted operation of the device (e.g., targeted recording of events) by implementing parameters such as: (i) the location of the device (e.g., indoors, outdoors, room); (ii) the use of the device (e.g., what is in the field of view of the device and what the user wants to see); and / or (iii) the type of device (e.g., wired or battery-powered); (iv) the location of the device (e.g., indoors, outdoors, room); (v) the type of device (e.g., what is in the field of view of the device and what the user wants to see); and / or (vi) the type of device (e.g., wired or battery-powered).

[0008] In one aspect, a method is disclosed in which an electronic device having an image sensor, one or more processors, and a memory storing instructions for execution by the one or more processors includes acquiring an event recording profile for the electronic device, the event recording profile based on configuration data for the electronic device, the configuration data including a location type or a power type, acquiring and storing on the electronic device a plurality of images of a scene from the image sensor, detecting a trigger event based on one or more of the plurality of images of the scene, identifying a target object from one or more of the plurality of images of the scene in response to detecting the trigger event, and creating an event clip from the stored images including the target object, wherein creating the event clip includes setting a clip length based on the event recording profile, and providing the event clip for display.

[0009] In some embodiments, setting the clip length includes setting a padding value, an inactivity threshold, and / or a maximum event length.

[0010] In some embodiments, setting the clip length includes selecting a padding value, the padding value corresponding to a number of images acquired before one or more of the plurality of images including the detected target object, and creating the event includes adding the number of images to the plurality of images including the detected target object.

[0011] In some embodiments, setting the clip length includes selecting an inactivity threshold, the inactivity threshold corresponding to a number of captured images in which no target object is detected, and creating the event includes adding a number of images to the plurality of images that include the detected target object.

[0012] In some embodiments, setting the clip length includes selecting a maximum event length, the maximum event length corresponding to a maximum number of images for the event, and creating the event includes ending the event when the maximum number of images is reached.

[0013] In some embodiments, the configuration data includes a location type corresponding to a particular region of the environment, and setting the event length based on the event recording profile includes selecting a padding value, an inactivity threshold, and / or a maximum event length based on the particular region of the environment in which the electronic device is located.

[0014] In some embodiments, the configuration data includes a power type, and setting the event length based on the event recording profile includes selecting a padding value, an inactivity threshold, and / or a maximum event length based on whether the power type of the electronic device is wired or battery powered.

[0015] In some embodiments, the configuration data further includes object priority data, and setting the event length based on the event recording profile includes setting a padding value, an inactivity threshold, and / or a maximum event length based on a priority of the target object identified according to the object priority data. This includes selecting the event length.

[0016] In some embodiments, setting the clip length includes setting a cool-off value corresponding to the amount of time to wait between successive object detections after two or more object detections occur within a threshold time.

[0017] In some embodiments, setting the clip length includes setting a padding value, an inactivity threshold, and a maximum event length according to a combination of values ​​associated with an event recording profile.

[0018] In some embodiments, detecting the trigger event includes detecting motion in the scene based on an analysis of two or more of the images of the scene.

[0019] In some embodiments, detecting the trigger event includes detecting a target object within the scene based on an analysis of one or more of the images of the scene.

[0020] In some embodiments, the method further includes forming a composite event clip by compositing the event clip with a previously created event clip, and providing the event clip for display includes providing the composite event clip for display.

[0021] In another aspect, an electronic device comprises one or more processors and a memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 13.

[0022] In another aspect, a non-transitory computer-readable storage medium stores instructions that, when executed by an electronic device having one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 13.

[0023] The various embodiments described will be better understood by reference to the following description of the embodiments in conjunction with the following drawings, in which like reference numerals refer to like elements throughout the drawings, in which: [Brief explanation of the drawings]

[0024] [Figure 1A] FIG. 1 illustrates an exemplary environment, according to some embodiments. [Figure 1B] 1 is a block diagram illustrating a representative network architecture according to some embodiments. [Figure 2] FIG. 1 is a block diagram illustrating an exemplary operating environment, according to some embodiments. [Figure 3] FIG. 1 is a block diagram illustrating an exemplary electronic device according to some embodiments. [Figure 4] FIG. 1 is a block diagram illustrating a representative server system according to some embodiments. [Figure 5] 1 is a block diagram illustrating an example event formula according to some embodiments. [Figure 6] 1 is a block diagram illustrating a representative client device according to some embodiments. [Figure 7] 1 is a block diagram illustrating an exemplary event processing system according to some embodiments. [Figure 8] FIG. 1 illustrates an exemplary event according to some embodiments. [Figure 9] FIG. 1 illustrates an exemplary composite event, according to some embodiments. [Figure 10] FIG. 1 illustrates an exemplary user interface for viewing events according to some embodiments. [Figure 11] FIG. 10 illustrates an exemplary user interface for obtaining configuration data according to some embodiments. [Figure 12] 1 is a flow diagram illustrating an example event handling process according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0025] Description of the embodiment Like reference numbers refer to like elements in the several drawings.

[0026] The following detailed description will discuss the embodiments illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the various embodiments described. However, it will be apparent to those skilled in the art that the various embodiments described can be practiced without these specific details. Additionally, well-known methods, procedures, components, circuits, and networks are not described in detail to avoid obscuring the characteristics of the embodiments.

[0027] Cameras, e.g., security cameras, doorbell camera-equipped devices, and support devices integrated with cameras, can be installed or positioned in a scene to collect visual input from the scene (also referred to as a field of view). In some embodiments, these devices record clips of video data (also referred to herein as events) and provide the clips for viewing by occupants of the environment via a server system, hub, or other network-connected device. In some embodiments, the parameters used to determine which events to record, which events to provide for viewing, and how to composite event video clips are modified based on several characteristics of the device, including, but not limited to, the device's location, use, and power type.

[0028] 1A illustrates an exemplary environment 100, according to some embodiments. The term "environment" refers to any space containing electronic devices (devices that perform one or more support functions, e.g., security cameras, audio assistance devices) connected to one or more networks or connected to each other. Exemplary environments include homes (e.g., houses, apartments, row homes, multi-unit apartments), hotels, retail stores, office buildings, industrial buildings, yards, parks, and more generally any living or working space. An environment may be referred to herein as a residential environment, a home, or an environment.

[0029] Additionally, terms such as “user,” “customer,” “installer,” “homeowner,” “resident,” “guest,” “tenant,” “landlord,” and “repairman” are utilized to refer to persons acting in certain contexts described herein. However, these references do not limit the scope of the present teachings with respect to the persons or people performing such actions or present near or within the environment. Thus, for example, the terms “user,” “customer,” “purchaser,” “installer,” “subscriber,” and “homeowner” may refer to the same person who, in the case of a single-family home, makes the purchasing decision for the home, purchases the device (e.g., a network-connected electronic device), installs the device, configures the device, and / or uses the device. However, in other cases, such as in a landlord-tenant environment, the customer may be the landlord who purchases the device, the installer may be the apartment manager, the first user may be the tenant, and the second user may be the landlord with respect to the remote control functionality. Importantly, the identity of the person performing the action may be germane to the particular benefits provided by one or more of the embodiments. In the following description, such statuses should not necessarily be construed as limiting the scope of the present teachings to any particular individual having a particular status.

[0030] The environment 100 includes a structure 150 (e.g., a home, an office building, a garage, or a mobile home) with various integrated devices (also referred to herein as “connected devices,” “network-connected devices,” “interconnected devices,” or “smart” devices). The illustrated structure 150 includes multiple rooms 152 that are at least partially separated from one another by walls 154. The walls 154 may include interior or exterior walls. Each room may further include a floor 156 and a ceiling 158. The network-connected devices may be integrated into the environment 100 without including the entire structure 150, for example, an apartment, condominium, or office building. In some embodiments, the devices include one or more of a mobile device 104 (e.g., tablet, laptop, mobile phone, smartphone), a display device 106, a media casting or streaming device 108, a thermostat 122, a home protection device 124 (e.g., smoke, fire, and carbon dioxide detectors), a connected doorbell / camera 126, a connected lockset 128, a home security device (e.g., motion detectors, window and door sensors, and alarms) including a connected alarm system 130 and camera 132, a connected wall switch answering machine 136, a connected appliance 138, a Wi-Fi communication device 160 (e.g., hub, router, extender), a connected home cleaning device 168 (e.g., a vacuum cleaner or floor cleaner), a communication and control hub 180, and / or an electronic assistant device 190 (also referred to herein as an audio assistant device and a display assistant device).

[0031] By deploying one or more media devices in the environment 100, users can access media content that is stored locally or streamed from a remote content source (e.g., a content host 114). In some embodiments, the media devices include a media output device 106 for outputting / displaying / playing media content directly to a viewer and a casting device 108 for streaming media content received over one or more networks to the media output device 106. Examples of media output devices 106 include, but are not limited to, television (TV) display devices, music players, and computer monitors. Examples of casting devices 108 include, but are not limited to, media streaming boxes, casting devices (e.g., a Google® Chromecast device), set-top boxes (STBs), DVD players, and TV boxes.

[0032] In the exemplary environment 100, media output devices 106 are located in multiple locations, and each media output device 106 is connected to a corresponding casting device 108 or includes an embedded casting unit. Media output device 106-1 includes a TV display connected via a hardwire to a DVD player or set-top box 108-1. Media output device 106-3 includes a network-connected TV device that includes an embedded casting unit for streaming media content for display to a viewer. Media output device 106-2 includes a conventional TV display connected to a network-connected TV box 108-1 (e.g., Google® TV or Apple® TV). Such a TV box 108-2 provides access to the Internet for streaming media content received from a media content host server 114 and displaying Internet-based content on the media output device 106-2.

[0033] In addition to media devices 106 and 108, one or more electronic assistant devices 190 are disposed in environment 100. Electronic assistant device 190 collects audio input to activate various media playback functions of electronic assistant device 190 and / or media devices 106 and 108. In some embodiments, electronic assistant device 190 also receives audio input from locally stored media devices. The electronic assistant 190 is configured to provide media content, either locally or streamed from a remote content source. In some embodiments, the electronic assistant 190 is voice-activated and located in close proximity to a media device, such as in the same room as the casting device 108 and the media output device 106. Alternatively, in some embodiments, the voice-activated electronic assistant (e.g., 190-1 or 190-3) is located in a room with one or more devices but no media device. Alternatively, in some embodiments, the voice-activated electronic assistant 190 is located in a location without network-connected electronic devices. This enables the electronic assistant 190 to communicate with the media devices and share content displayed on one device with another device (e.g., from device 190-1 to device 190-2 and / or media device 108).

[0034] The voice-activated electronic assistant 190 includes at least one microphone, a speaker, a processor, and memory that stores at least one program executed by the processor. The speaker is configured to enable the electronic assistant 190 to deliver audio messages (e.g., messages related to a media content item being played or messages as part of a conversation between the user and the electronic assistant 190). In some embodiments, the electronic assistant 190 provides audible information to the user via the speaker in response to a user query. Instead of audio messages, visual signals can be used to provide feedback to the user of the electronic assistant 190 regarding the processing status of the audio input, e.g., a visual notification displayed on the device.

[0035] According to some embodiments, electronic device 190 is a voice-activated interface device configured to provide voice recognition functionality using server system 140. In some embodiments, server system 140 includes crowdcast service server 116 and / or audio / display assistance server 112. For example, in some embodiments, electronic device 190 includes a network-connected speaker that provides music (e.g., audio from video content being played on electronic assistance device 190 or display device 106) to the user and enables eyes-free and / or hands-free access to voice-assisted services (e.g., Google® Assistant). In some cases, electronic device 190 is a voice interface device, such as a speaker device, or a device that includes a display screen with or without touch-sensing capabilities.

[0036] In some embodiments, electronic assistant 190 includes an integrated display screen in addition to a microphone, speaker, processor, and memory (e.g., 190-2 and 190-4). This display screen is configured to provide additional visual information (e.g., media content, information about the media content) in addition to the audio information that may be broadcast through the speaker of electronic assistant 190. When the user is nearby and has an unobstructed line of sight, the user can view the additional visual information directly on the display screen of electronic assistant 190. Optionally, the additional visual information provides feedback to the user of electronic device 190 regarding the status of processing of the audio input. Optionally, the additional visual information may be provided in response to the user's previous audio input (e.g., a user query) and related to the audio information broadcast by the speaker. In some embodiments, the display screen of voice-activated electronic device 190 is a touch-sensitive display screen and is configured to detect touch input on its surface (e.g., commands provided via the touch-sensitive display screen). Alternatively, in some embodiments, the display screen of voice-activated electronic device 190 is not a touch-sensitive display screen.

[0037] When audio input from electronic device 190 is used to control electronic device 190 and / or to control media output device 106 via cast device 108, electronic assistant device 190 , can control cast-enabled media devices, regardless of whether they themselves have a display. In one example, electronic device 190 includes a speaker with far-field voice access capabilities and serves as a voice interface device for network-implemented assistance services (e.g., Google® Assistant).

[0038] Electronic device 190 may be located in any room within environment 100. In some embodiments, when multiple electronic assistant devices 190 are distributed across multiple rooms, electronic assistant devices 190 become synchronized audio receivers to receive audio input from each of the multiple rooms. For example, a first electronic device 190-1 can receive a user instruction (e.g., "OK Google, show this photo album on the kitchen device") directed to a second electronic device 190-2.

[0039] Specifically, in some embodiments, electronic device 190 includes a network-connected speaker (e.g., connected via a WiFi® network) that has a microphone connected to a voice-activated personal assistant service (e.g., Google® Assistant). A user can issue a media playback request through the microphone of electronic assistant device 190, asking the personal assistant service to play media content on electronic assistant device 190 and / or on another connected media output device 106. For example, a user can issue a media playback request by speaking near the speaker, "Okay Google, play cat videos on the living room TV." The personal assistant service fulfills the media playback request by playing the requested media content on the requested device using a default or specified media application.

[0040] The user can also make voice requests via the microphone of the electronic assistant 190 regarding media content that has already been played and / or is currently being played on the electronic assistant 190. For example, the user can instruct the electronic assistant to provide information related to the currently playing media content item, such as ownership information or subject matter of the media content. In some embodiments, in the absence of a remote control or if the user has access to a second display device, closed captioning of the currently displayed media content can be turned on or off on the display device by voice. Thus, the user can turn on closed captioning on the display device via the eyes-free and hands-free voice-activated electronic assistant 190 without using any other device with a physical user interface.

[0041] In some embodiments, electronic assistant device 190 includes a display screen and one or more built-in cameras configured to capture images and / or video that are transmitted (e.g., streamed) to server system 140 for display on a client device (e.g., an authorized client device 104).

[0042] In some embodiments, the voice-activated electronic assistant 190 may be mounted on, integrated with, and / or supported by the walls 154, floor 156, or ceiling 158 of the environment 100. Integrated devices include intelligent, multi-sensing, network-connected devices that are seamlessly integrated with each other in a network and / or with a central server or cloud computing system to provide a variety of useful functions. In some embodiments, one device is located in the same location in the environment 100 as the casting device 108 and / or the output device 106, and thus is already in close proximity to or relative to the casting device 108 and the output device 106. They are arranged to have a distance of knowledge.

[0043] In some embodiments, environment 100 includes one or more network-connected camera systems 132 (also referred to herein as cameras 132). In some embodiments, content captured by cameras 132 is displayed on electronic assistant 190 in response to a request from a user (e.g., a user instruction such as, "OK Google, show me the monitor in the baby's room") and / or according to a setting in environment 100 (e.g., a setting to display content captured by a particular camera 132 at night or in response to the detection of an intruder).

[0044] In some embodiments, the environment 100 includes one or more network-connected thermostats 122, hazard detectors 124, doorbells 126, door locks 128, alarm systems 130, camera systems 132, wall switches 136, appliances 138 (e.g., refrigerators, stoves, ovens, televisions, washers, and / or dryers), lights, stereos, intercom systems, garage door openers, floor fans, ceiling fans, wall air conditioners, pool heaters, irrigation systems, security systems, space heaters, window air conditioning (AC) units, powered duct vents, etc.

[0045] Environment 100 includes one or more other occupancy sensors (e.g., touchscreens, IR sensors, ambient light sensors, and motion detectors). In some embodiments, environment 100 includes radio frequency identification (RFID) readers (e.g., located in each room 152 or portion thereof) that determine occupancy based on RFID tags located on or implanted in the bodies of occupants. For example, RFID readers may be integrated into network-connected hazard detectors.

[0046] In some embodiments, in addition to including sensing capabilities, one or more devices included in environment 100 can communicate data, including share information, with other devices, a central server, a cloud computing system, and / or other networked devices (e.g., client device 104, cast device 108, and / or electronic assistant device 190). Similarly, in some embodiments, cast device 108 and electronic assistant device 190 can each communicate data, including share information, with other cast devices 108, other electronic assistant devices 190, a central server or cloud computing system 140, and / or other networked devices (e.g., client device 104). Data communication may occur using certain custom or standard wireless network protocols (e.g., IEEE 802.15.4, Wi-Fi®, ZigBee®, 6L0WPAN, Thread, Z-Wave®, Bluetooth® Smart, ISA100.11a, WirelessHART, MiWi) and / or certain custom or standard wired network protocols (e.g., Ethernet®, HomePlug), or any other suitable communication protocols, including communication protocols not yet developed as of the filing date of this application.

[0047] In some embodiments, the casting device 108, the electronic assistant device 190, and other devices included in the environment 100 function as wireless or wired repeaters. In some embodiments, a first one of the casting devices 108 communicates with a second one of the casting devices 108 or one or more other devices via a wireless router. The casting device 108, the electronic assistant device 190, and one or more other devices may also communicate with each other via a connection (e.g., network interface 160) to a network, such as the Internet 110. The casting device 108, the electronic assistant device 190, and / or one or more other devices may communicate with each other via the Internet 110 (also referred to herein as a central server system and / or a cloud computing system). The environment 100 may communicate with a server system 140 (such as a server system 140) that may be associated with a manufacturer, support entity, or service provider associated with one or more devices included in the environment 100 and / or media content being displayed or presented to a user.

[0048] In general, any of the connected electronic devices included in environment 100 may be configured with various capabilities for interacting with a user within environment 100. For example, an electronic device may be configured with one or more microphones, one or more speakers, and / or voice interaction capabilities, such that a user interacts with the electronic device via voice input received by the microphone and audible output played by the speaker to present information to the user. Similarly, an electronic device may be configured with buttons, switches, and / or other touch-responsive sensors (e.g., touchscreens, touch panels, or capacitive or resistive touch sensors) for receiving user input, and haptic or other tactile feedback capabilities for providing tactile output to the user. An electronic device may also be configured with visual output capabilities, such as a display panel and / or one or more indicator lights, for visually outputting information to the user, as described in U.S. Patent Application No. 15 / 592,120, entitled "LED Design Language for Visual Affordances of Voice User Interfaces," which is incorporated herein by reference. Additionally, the electronic devices included in environment 100 may be configured with motion sensors, such as radar transceivers or PIR detectors, capable of detecting the movement of objects and people near the electronic devices, as described in U.S. patent application Ser. No. 15 / 481,289, entitled "System, Method, and Apparatus for Utilizing a Radar-Based Touch Interface," which is incorporated herein by reference.

[0049] Input received by any of these sensors may be processed by the electronic device and / or by a server communicatively coupled to the electronic device (e.g., server system 140 of FIG. 1A ). In some embodiments, the electronic device and / or server processes and / or prepares a response to the user's input, and the electronic device outputs that response via one or more of the electronic device's output capabilities. In some embodiments, the electronic device outputs, via one or more of the electronic device's output capabilities, information that does not directly respond to the user input but has been sent to the electronic device by a second electronic device in environment 100 or by a server communicatively coupled to the electronic device. This sent information may be of any type that is displayable / playable via the electronic device's output capabilities.

[0050] Server system 140 provides data processing to facilitate monitoring and verification of events (e.g., motion, audio, and security) from data captured by devices included in environment 100, such as video camera 132, doorbell 126 (with a built-in camera), and electronic assistant device 190. In some embodiments, server system 140 may include an audio / display assistance server 112 for processing video and / or audio input (e.g., collected by electronic assistant device 190, doorbell / camera 126, or video camera 132), one or more content hosts 114 for providing media content for display on one or more devices included in environment 100, and a crowdcast service server 116 for creating virtual user domains based on distributed device terminals. In some embodiments, server system 140 includes a device registry 118 for maintaining a record of distributed device terminals within the virtual user environment. Examples of distributed device terminals include, but are not limited to, electronic assistant device 190, casting device 108, media output device 106, and / or any other device included in environment 100. In some embodiments, these distributed device terminals are linked to user accounts within a virtual user domain. In some embodiments, each of these functions and content hosts is a separate server within server system 140. In the present application, some of these functions are integrated into the server system 140 .

[0051] In some embodiments, network interface 160 comprises a conventional network device (e.g., a router). In some embodiments, environment 100 further comprises hub device 180, which is communicatively coupled to network 110 either directly or via network interface 160. Hub device 180 is also communicatively coupled to one or more devices included in environment 100. In some embodiments, one or more network-connected devices included in environment 100 may, as desired, communicate with one or more wireless communication networks (e.g., ZigBee, Z-Wave, Insteon, Bluetooth, WiFi, and / or other wireless communications). The client devices 104 communicate with the hub device 180 using a network (e.g., a mobile phone, a home controller, a laptop, a tablet computer, a game console, or a similar electronic device). In some embodiments, the client devices 104 may control or interact with the hub device 180 and devices coupled to the hub device 180 through applications running on the client devices 104 (e.g., a mobile phone, a home controller, a laptop, a tablet computer, a game console, or a similar electronic device). In some embodiments, a user of such applications may be able to view status information for the hub device or network-connected devices coupled to the hub device, configure the hub device to interoperate with devices newly introduced to the home network, activate the new devices, adjust or view settings for connected devices, etc.

[0052] FIG. 1B is a block diagram illustrating a representative network architecture 170 that includes network 102, according to some embodiments.

[0053] In some embodiments, the integrated devices of environment 100 include intelligent, multi-sensing, network-connected devices (e.g., devices 122, 124, 126, 128, 130, 132, 136, and / or 138), collectively referred to herein as devices 120. These devices are seamlessly integrated with each other in a network (e.g., 102 in FIG. 1B ) and / or with a central server or cloud computing system (e.g., server system 164) to provide a variety of useful functions.

[0054] In some embodiments, devices 120 and hub device 180 in environment 100 combine to create a mesh network in network 102. In some embodiments, one or more devices 120 in network 102 act as controllers. Additionally and / or alternatively, hub device 180 acts as the controller. In some embodiments, the controller has more computing power than the other devices. In some embodiments, the controller controls the operation of environment 100 by processing input (e.g., from devices 120, electronic device 190 (FIG. 1A), and / or server system 164) and sending commands (e.g., to devices 120 in network 102). In some embodiments, some of the devices 120 in network 102 (e.g., a mesh network) are “spokesman” nodes (e.g., 120-1), and other devices are “low power” nodes (e.g., 120-6). Some devices in environment 100 are battery powered, while others have a regular and reliable power source by connecting to wiring behind the environment's walls 154 (e.g., wiring having a 120 volt line voltage). Devices with a regular and reliable power source are referred to as "spokesman" nodes. These nodes are typically equipped to use wireless protocols to facilitate two-way communication with various other devices in environment 100 and with server system 164. In some embodiments, one or more "spokesman" nodes act as controllers. Devices that are battery powered are "low power" nodes. These low power nodes tend to be smaller than spokesman nodes. , typically Zigbee (registered trademark), Z-Wave (registered trademark), 6L0WPAN, Thread, Brute Force It communicates exclusively using wireless protocols that require very little power, such as Wi-Fi.

[0055] In some embodiments, some low-power nodes are not capable of two-way communication. These low-power nodes can send messages but cannot "listen." Therefore, other devices in environment 100, such as spokesman nodes, cannot send information to these low-power nodes. In some embodiments, some low-power nodes are only capable of limited two-way communication. For example, other devices can only communicate with the low-power nodes during certain periods of time.

[0056] As described above, in some embodiments, devices function as low-power and spokesman nodes to create a mesh network within environment 100. In some embodiments, some low-power nodes in the environment periodically transmit messages about what they are sensing, and other low-power nodes in the environment move messages between nodes (i.e., between devices) throughout network 102 by forwarding these messages in addition to transmitting their own messages. In some embodiments, spokesman nodes in network 102 that can communicate using a relatively high-power communication protocol, such as IEEE 802.11, can switch to a relatively low-power communication protocol, such as IEEE 802.15.4, receive these messages, convert them to another communication protocol, and transmit the converted messages to other spokesman nodes (e.g., using a relatively high-power communication protocol) and / or to server system 164. Thus, low-power nodes using a low-power communication protocol can send and receive messages throughout network 102 and can send and receive messages to server system 164 via Internet 110. In some embodiments, the mesh network allows the server system 164 to periodically receive data from most or all devices in the home, make inferences based on the data, facilitate state synchronization of devices inside and outside the network 102, and send commands to one or more devices to perform tasks within the environment.

[0057] As described above, spokesman nodes and some low power nodes can "listen." Thus, a user, other devices, and / or server system 164 can communicate control commands to the low power nodes. For example, a user can use an electronic device 104 (e.g., a phone or other mobile communications device) to send a command over the Internet to server system 164, which relays the command to one or more spokesman nodes in network 102. The spokesman nodes can use a low power protocol to communicate the command to low power nodes in network 102 and to other spokesman nodes that did not receive the command directly from server system 164.

[0058] In some embodiments, night-light 170 (FIG. 1A), an example of device 120, is a low-power node. In addition to housing a light source, night-light 170 houses an occupancy sensor, such as an ultrasonic or passive IR sensor, and an ambient light sensor, such as a photoresistor or single-pixel sensor, for measuring the light in the room. In some embodiments, night-light 170 is configured to activate the light source when the ambient light sensor detects that the room has become dark and when the occupancy sensor detects that someone is in the room. In other embodiments, night-light 170 is configured to activate the light source simply when the ambient light sensor detects that the room has become dark. In some embodiments, night-light 170 includes a momentary message that coincides with the occupancy sensor detecting the presence of a person in the room, and It includes a low-power wireless communications chip (e.g., a ZigBee chip) for periodically transmitting messages regarding occupancy and the amount of light in the room. As described above, these messages may be transmitted wirelessly between nodes (i.e., between devices) in the network 102 (e.g., using a mesh network) and over the Internet 110 to the server system 164.

[0059] Other examples of low-power nodes include battery-powered devices such as hazard detectors 124, cameras 132, and doorbells 126. These battery-powered devices are typically located in areas without access to constant and reliable power and may optionally include any number and type of sensors, such as image sensors, occupancy / motion sensors, ambient light sensors, ambient temperature sensors, humidity sensors, smoke / fire / heat sensors (e.g., thermal radiation sensors), carbon monoxide / carbon dioxide sensors, etc. Additionally, these battery-powered devices may transmit messages from their sensors to other devices and / or server system 164, for example, using the mesh network described above.

[0060] Examples of spokesman nodes include hardwire-powered doorbell 126, thermostat 122, wall switch 136, and wall plug 142. Because these devices are located near and connected to a reliable power source, they may contain more power-consuming elements, such as one or more communication chips capable of two-way communication over a variety of protocols.

[0061] In some embodiments, the environment 100 includes a service robot 168 (FIG. 1A) configured to autonomously perform specific household tasks.

[0062] 1A-1B, in some embodiments, the environment 100 of FIGS. 1A-1B includes a hub device 180 that is communicatively coupled to the network 110, either directly or via a network interface 160. The hub device 180 is also communicatively coupled to one or more devices that use a wireless communication network available to the environment 100. The communication protocols used by the wireless communication network may include ZigBee, Z-Wave, Insteon, EuOcean, Thread, OSIAN, Bluetooth, and the like. ® Low Energy, etc. In some embodiments, hub device 180 not only converts data received from each device to meet the data format requirements of network interface 160 or network 110, but also converts information received from network interface 160 or network 110 to meet the data format requirements of each communication protocol associated with the particular device. In some embodiments, in addition to data format conversion, hub device 180 also preliminarily processes data received from devices or information received from network interface 160 or network 110. For example, hub device 180 can aggregate inputs from multiple sensors / connected devices (including sensors / devices of the same and / or different types) and perform higher-level processing on these inputs (e.g., assessing the overall environment, coordinating operations among different sensors / devices), and / or provide instructions to different devices based on the collected inputs and programmed processing. Note that in some embodiments, network interface 160 and hub device 180 may be integrated into a single network device. The functionality described herein represents specific implementations of devices, control applications running on representative electronic devices (e.g., telephones or other mobile communications devices), hub device 180, and servers coupled to the hub device via the Internet or other wide area network. All or part of the functionality and associated operations may be performed by any element of the described system. For example, all or part of the functionality described herein as being performed by implementations of the hub device may be performed by different system embodiments. In some embodiments, the system may run in whole or in part on a server, one or more connected devices and / or a controlling application, or different combinations thereof.

[0063] FIG. 2 illustrates a representative operating environment 200. In this operating environment 200, a server system 164 (sometimes referred to as a “hub device server system,” “video server system,” or “hub server system”) provides data processing to facilitate monitoring and verification of motion events in video streams captured by video cameras 132. As shown in FIG. 2, the server system 164 receives video data from video sources 222 (including cameras 132, doorbells 126, and / or electronic devices 190) located in various physical locations (e.g., homes, restaurants, stores, streets, parking lots, and / or within the environment 100 of FIG. 1). Each video source 222 may be tied to one or more reviewer accounts. The server system 164 provides video surveillance data of the video sources 222 to client devices 204 associated with the reviewer accounts. For example, the portable electronic device 104 is an example of a client device 204.

[0064] In some embodiments, provider server system 164 or components thereof correspond to the server systems described with reference to Figures 1A and 1B. In some embodiments, server system 164 is a dedicated video processing server or includes dedicated video processing elements for providing video processing services to video sources and client devices 204, as described with reference to Figures 1A and 1B, independent of other services provided by the server system.

[0065] In some embodiments, each video source 222 includes one or more video cameras 132 for capturing video (as described in more detail below with reference to events and video clips) and transmitting the captured video to server system 164 in substantially real time or clip by clip. In some embodiments, one or more video sources 222 optionally include a controller device (not shown) that acts as an intermediary between one or more cameras 132 and server system 164. The controller device receives video data from one or more cameras 132, optionally performs some pre-processing on the video data, and transmits the video data to server system 164 in substantially real time on behalf of one or more cameras 132. In some embodiments, each camera has on-board processing capability for performing some pre-processing on the captured video data and transmits the processed video data (along with metadata captured by the pre-processing) to the controller device and / or server system 164. All embodiments of the present disclosure are described with reference to video cameras 132 as video sources 222. However, the embodiments may also be applied to any other device in environment 100 that includes a camera, such as doorbell 126 or assistive device 190 that includes a camera.

[0066] 2 , according to some embodiments, each client device 204 includes a client-side module 202. The client-side module 202 communicates with a server-side module 206 executing on a server system 164 via one or more networks 110. The client-side module 202 provides client-side functionality for event monitoring and review and communication with the server-side module 206. The server-side module 206 provides server-side functionality for event monitoring and review for any number of client-side modules 202 each resident on a corresponding client device 204. The server-side module 206 also provides server-side functionality for video processing and camera control for any number of control devices and any number of video sources 222, including cameras 132.

[0067] In some embodiments, the server-side module 206 may include one or more processors 2 12, a video storage database 214, a device and account database 216, an I / O interface 218 to one or more client devices, and an I / O interface 222 to one or more video sources. The I / O interface 218 to one or more client devices facilitates client-side input and output processing to the server-side module 206. The database 216 stores profiles of multiple reviewer accounts registered with the video processing server. Each user profile includes account credentials for the corresponding reviewer account and one or more video sources linked to the corresponding reviewer account. The I / O interface 222 to one or more video sources facilitates communication with one or more video sources 222 (e.g., a group consisting of one or more cameras 132 and associated controller devices). The video storage database 214 stores raw video data received from the video sources 222 and various types of metadata, such as motion events, event categories, event category models, event filters, and event masks, used for data processing for event monitoring and verification of each reviewer account.

[0068] Examples of representative client devices 204 include, but are not limited to, handheld computers, wearable computing devices, personal digital assistants (PDAs), tablet computers, laptop computers, desktop computers, cellular phones, mobile phones, media players, navigation devices, game consoles, televisions, remote controls, point-of-sale (POS) terminals, in-vehicle computers, e-book readers, or any combination of two or more of these or other data processing devices.

[0069] Examples of the one or more networks 110 include a local area network (LAN) and a wide area network (WAN) such as the Internet. The one or more networks 110 may be implemented using any known network protocol, including various wired or wireless protocols such as Ethernet, Universal Serial Bus (USB), Firewire, Long Term Evolution (LTE), GSM, Enhanced Data GSM Environment (EDGE), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wi-Fi, Voice over Internet Protocol (VoIP), Wi-MAX, or any other suitable communication protocol, as appropriate.

[0070] In some embodiments, server system 164 is implemented on one or more standalone data processing devices or a distributed computer network. In some embodiments, server system 164 utilizes various virtual devices and / or the services of third-party service providers (e.g., third-party cloud service providers) to provide the underlying computing and / or infrastructure resources for server system 164. In some embodiments, server system 164 includes, but is not limited to, a handheld computer, a tablet computer, a laptop computer, a desktop computer, or a combination of any two or more of these or other data processing devices.

[0071] 2 includes both a client-side portion (e.g., client-side module 202) and a server-side portion (e.g., server-side module 206). The division of functionality between the client and server portions of operating environment 200 may vary in different embodiments. Similarly, the division of functionality between video source 222 and server system 164 may vary in different embodiments. For example, in some embodiments, client-side module 202 provides only user-side input and output processing functionality, and all other data processing functionality is provided to a back-end server (e.g., server system 164). 64). Similarly, in some embodiments, each video source 222 is a simple video capture device that captures video data (e.g., events in the form of video clips) and performs no or limited local pre-processing on the video data before streaming it to server system 164. While many aspects of the technology have been described in terms of server system 164, it will be apparent to one skilled in the art without inventive effort that client device 204 and / or video source 222 can perform corresponding operations. Similarly, some aspects of the technology may be described in terms of a client device or video source, and it will be apparent to one skilled in the art without inventive effort that a video server can perform corresponding operations. Furthermore, some aspects of the technology may be performed cooperatively by server system 164, client device 204, and video source 222.

[0072] It should be noted that operating environment 200, including server system 164, video source 222, and video camera 132, is merely an example. In general, many aspects of operating environment 200 are applicable to other operating environments, where the server system provides data processing to facilitate monitoring and review of data acquired by other types of electronic devices (e.g., thermostat 122, hazard detector 124, doorbell 126, wall plug 142, appliance 138).

[0073] The electronic devices, client devices, and server system communicate with each other via one or more communication networks 110. In an exemplary environment, two or more devices (e.g., network interface device 160, hub device 180, and client devices 204-n) are located in close proximity to each other such that they can be communicatively coupled within the same subnetwork 110A via a wired connection, a WLAN, or a Bluetooth® personal area network (PAN). The Bluetooth® PAN is established based on classic Bluetooth® technology or Bluetooth® Low Energy (BLE) technology, as appropriate. The environment further includes one or more other wireless communication networks 110B, and at least some of the electronic devices of video sources 222-n exchange data with hub device 180 via wireless communication network 110B. Alternatively, in some situations, some of the electronic devices of video source 222-n communicate directly with network interface device 160 via the same subnetwork 110A that couples devices 160, 180, and 204-m. In some embodiments (e.g., in the case of network 110C), both client device 204-m and the electronic devices of video source 222-n communicate directly through network 110, without going through network interface device 160 or hub device 180.

[0074] In some embodiments, during normal operation, network interface device 160 and hub device 180 communicate with each other to form a network gateway for exchanging data with the electronic devices of video sources 222-n. As described above, network interface device 160 and hub device 180 communicate with each other, as needed, via sub-network 110A.

[0075] 3 is a block diagram illustrating an exemplary electronic device 222 in environment 100, according to some embodiments. For example, electronic device 222 may be security camera 132, doorbell camera 126, or camera-equipped support device 190. Electronic device 222 typically includes one or more processors (CPUs) 302, one or more network interfaces 304, memory 306, and one or more communication buses 308 for interconnecting these elements (also referred to as a chipset). Electronic device 222 includes one or more cameras 362 configured to capture images and / or video. Electronic device 222 may include: The electronic device 222 includes one or more output devices 312, such as one or more speakers, a display, and / or one or more indicator lights (e.g., LEDs), configured to display a visual indication of the status of the camera 362. In some embodiments, the electronic device 222 includes sensors 363 (e.g., motion sensors, radar sensors, and / or presence sensors) for detecting an event or change. In some embodiments, the detection of an event or change is triggered by the detection of motion within the field of view of the camera 362.

[0076] In some implementations of electronic device 222 (e.g., assist device 190), electronic device 222 includes one or more input devices 310 to facilitate user input, such as one or more microphones, a volume control, and a privacy control. The volume control is configured to accept user actions (e.g., pressing a volume up button or a volume down button, or pressing and holding both the volume up button and the volume down button) to adjust the speaker volume level or reset display assist device 300. The privacy control is configured to accept user actions to control privacy settings of the display assist device (e.g., whether to disable the microphone and / or camera 362). In some embodiments, the privacy control is a physical button located on electronic device 222. In some embodiments, input device 310 of electronic device 222 includes a touch detection module integrated on a display panel and configured to detect touch input on the display panel surface. In some embodiments, input device 310 of electronic device 222 includes a camera module configured to capture images and / or video streams within a field of view.

[0077] In some embodiments, electronic device 222 includes a presence sensor 363 configured to detect the presence of a user within a predetermined area around display assist device 190. In some situations, display assist device 190 operates in a sleep or hibernation mode in which it stops detecting and processing audio input, and does not wake up from the sleep or hibernation mode or listen to its surroundings (i.e., does not process audio signals collected from the environment) until the presence sensor detects the presence of a user within the predetermined area. An example of a presence sensor is an ultrasonic sensor configured to detect the presence of a user.

[0078] Memory 306 includes high-speed random access memory such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices, and optionally includes non-volatile memory such as one or more magnetic disk storage devices, one or more optical disk storage devices, one or more flash memory devices, or one or more other non-volatile solid-state storage devices. Memory 306 optionally includes one or more storage devices located remotely from one or more processors (or CPUs) 302. Memory 306, or alternatively, non-volatile memory within memory 306, includes a non-transitory computer-readable storage medium. In some embodiments, memory 306 or the non-transitory computer-readable storage medium of memory 306 stores the following programs, modules, and data structures, or a subset or superset thereof: an operating system 316 that contains procedures for handling various basic system services and performing hardware-dependent tasks; a network communications module 318 for connecting the electronic device 222 to other devices (e.g., server system 164, client device 104, client device 204, device 120, hub device 180, and / or other electronic devices 222) via one or more (wired or wireless) network interfaces 304 and one or more networks 110, e.g., the Internet, other wide area networks, local area networks, metropolitan area networks; an input / output control module 320 for receiving input via one or more input devices 310 that enable the presentation of information on the display; This input / output control module 320 an audio processing module 322 for processing audio input or audio messages collected from the environment around the electronic device 222 or preparing the collected audio input or audio messages for processing by the server system 164 (audio / display assistance server 112); a display assistance module 324 for displaying additional visual information, including but not limited to media content items (e.g., YouTube® video clips), news posts, social media messages, weather information, personal photos, the status of voice input processing, and device readings; a touch detection module 326 for detecting touch events on the top surface of the electronic device 222; The above-mentioned programs, modules, and data structures are an event processing module 350 for detecting an event and processing a video clip associated with the event; This event processing module 350 a trigger detection module 350 for detecting an event trigger (e.g., motion in the scene or the presence of a foreground object); an object recognition module 354 for performing object recognition analysis on objects detected from the scene (e.g., as part of a determination as to whether an object should trigger the creation of an event); an event synthesis module 356 for synthesizing a video clip including the frame containing the event and / or additional frames before and after the event, where synthesizing includes taking into account event parameters such as an inactivity threshold and a maximum event length; The above-mentioned programs, modules, and data structures are a video processing module 358 for acquiring image frames from the image sensor of the camera 362 and processing the video stream (e.g., a continuous video stream, a video clip, and / or one or more still images), which in some embodiments includes compressing the processed video data for transmission over a network; The above-mentioned programs, modules, and data structures are a power detection module 359 for detecting the power type of the electronic device 222 (e.g., whether the device is battery-powered or wired); data 330; Data 330 is device settings 332 for storing information related to the electronic device 222, such as common settings for the device (e.g., service tier, device model, storage capacity, processing capabilities, communication capabilities) and information for user accounts in the virtual user domain linked to the electronic device 222; parameters used by the event processing module 350 to generate events, such as event profile settings 380 including padding 382, ​​inactivity threshold 384, maximum event length 386, cool-off threshold 388, and / or object filter and / or priority 390; The padding 382 includes a pre-roll value (e.g., the time included in the event clip before detecting an object or target occurrence, or the number of images captured before the image frame containing the target object or occurrence, as shown in padding windows 814 and 914 in FIGS. 8 and 9 ) and a post-roll value (e.g., the time included in the event clip after the detected target object or occurrence is not present in the camera's scene or field of view, or the number of images captured until the target object or occurrence is no longer detected, as shown in padding windows 816 and 916 in FIGS. 8 and 9 ). The inactivity threshold 384 may be a wait time before terminating an event, or a time after a target object or occurrence is no longer detected, instead of continuing the event to include subsequent activity, as shown, for example, in the inactivity window between time D / E in FIG. 8 and time D / F and I / J in FIG. 9. is the number of images taken until the event is over, and this number of images corresponds to the waiting time before terminating the event. The maximum event length 386 is the event duration before the event ends, regardless of whether a target object or occurrence is present in the camera's scene or field of view, as shown, for example, in event segments 812 and 912 in FIGS. 8 and 9, or the maximum number of images associated with the time specified as the maximum event length; in some embodiments, the maximum event length includes a padding window; in some embodiments, the maximum event length does not include a padding window; The cool-off threshold 388 is, for example, the speed of object detection above which event recording stops; Object filters and / or priorities 390 determine the objects for which an event is recorded, for example, as shown in the event priority list in the example formula 442 of FIG. 5 ; The above-mentioned programs, modules, and data structures are an image buffer (also referred to as an input buffer) 392 for storing image frames captured by the image sensor of the camera 362; voice control data 336 for storing voice signals, voice messages, response messages, and other data related to the voice interface functions of the electronic device 222; authorized user data 338 for storing information about users authorized to use the display assist device, such as images, voice information, and fingerprint information of authorized users; and local data storage 340 for selectively storing raw or processed data related to electronic device 222, for example, event data and / or video data captured by camera 362.

[0079] Each of the above-described elements may be stored in one or more memory devices as described above and corresponds to an instruction set for performing the functions described above. The above-described modules or programs (i.e., instruction sets) need not be implemented as separate software programs, procedures, modules, or data structures. Thus, in various embodiments, these modules may be combined or rearranged in various subsets. In some embodiments, memory 306 stores a subset of the above-described modules and data structures, as desired. Memory 306 also stores additional modules and data structures not described above, as desired.

[0080] In some embodiments, one or more of the elements described above may be stored on or implemented in a server system (e.g., server system 164). For example, event processing module 350 may be stored on server system 164. In such an embodiment, electronic device 222 transmits a video stream including image data acquired from camera 362 to server system 164, and event processing module 350 performs trigger detection, object recognition, and / or event composition on server system 164. As a result of one or more of the processes described above, server system 164 transmits an event clip (e.g., event clip 740, described in more detail below with reference to FIG. 7) to electronic device 222 for display (e.g., on output device 312 of electronic device 222).

[0081] 4 is a block diagram illustrating a server system 164, according to some embodiments. The server system 164 includes one or more processors (e.g., CPUs) 402, one or more network interfaces 404, memory 406, and one or more communication buses 408 for interconnecting these elements (also referred to as a chipset). The memory 406 may include high-speed random access memory, such as DRAM, SRAM, DDR SRAM, or other random access solid-state memory devices, and may optionally include one or more The memory 406 may include non-volatile memory such as a magnetic disk storage device, one or more optical disk storage devices, one or more flash memory devices, or one or more other non-volatile solid-state storage devices. The memory 406 optionally includes one or more storage devices located remotely from the one or more processors 402. The memory 406, or alternatively the non-volatile memory within the memory 406, includes a non-transitory computer-readable storage medium. In some embodiments, the memory 406 or the non-transitory computer-readable storage medium of the memory 406 stores the following programs, modules, and data structures, or a subset or superset thereof: an operating system 410 that contains procedures for handling various basic system services and performing hardware-dependent tasks; a network communications module 412 for connecting the server system 164 to other systems and devices (e.g., client devices, electronic devices, and systems connected to one or more networks 110) via one or more (wired or wireless) network interfaces 404; a server-side module 414 that provides server-side functionality such as device control, data processing, and data validation; The server-side module 414 a data receiving module 416 configured to receive data from the electronic device (e.g., event data from the electronic device 222) and prepare the received data for further processing and storage in the server database 428; a device control module 418 configured to generate and send server-initiated control commands to modify an operational mode of an electronic device (e.g., electronic device 222) and / or receive and forward user-initiated control commands (e.g., from client device 204 and client device 104) to modify an operational mode of the electronic device (e.g., receive device configuration data 438 for electronic device 222 and forward one or more event handling formulas 442 corresponding to the configuration data 438); a data processing module 420 configured to process data provided by the electronic device and / or prepare and transmit the processed data to a reviewing device (e.g., the client device 204 to be reviewed by a user), including, but not limited to, a video processing module 422 for processing (e.g., classifying and / or recognizing) entities and / or candidate events detected from received video clips (e.g., video clips received from the electronic device 222 and corresponding to detected events); a user interface module 424 for communicating with the user (e.g., sending alerts, timeline events, etc., and receiving user edits and zone definitions, etc.); and an entity recognition module 426 for analyzing and / or identifying people detected from the environment; The programs, modules, and data structures described above include a server database 428; The server database 428 a device and account database 216 for storing device and account data; This device and account database 216 includes: device information 436 relating to one or more devices (e.g., electronic devices 222); device configuration data 438 including a device identifier 448, installation location data 449a, device use information 449b, and / or device power type data 449c; User account information, such as user profile, information and settings of linked hub devices and electronic devices (e.g., hub device ID), unique secret of the hub device, associated user and hardware characteristics (e.g., service tier, subscription, device model, storage capacity, processing power), user interface settings, data verification preferences, and account data 432 associated with the user account, including information related to the electronic device. the information includes, but is not limited to, one or more device identifiers (e.g., MAC address and UUID), a unique secret of the device, a displayed title, and profiles of reviewer accounts registered with the video processing server, each user profile including an account certificate for the corresponding reviewer account and one or more video sources linked to the corresponding reviewer account; The server database 428 a video storage database 214 (see FIG. 2 ) for storing video data received from video sources (e.g., video clips received from one or more electronic devices 222) and various types of metadata, e.g., motion events, event categories, event category models, event filters, and event masks, used in data processing for event monitoring and verification of each reviewer account; data storage 430 for storing data associated with each electronic device (e.g., each electronic device 222) for each user account, data processing models, data processing results, and other metadata associated with the data (e.g., name of the data result, location of the electronic device, time of creation, duration, electronic device settings), where all or a portion of the data and / or processing results associated with the hub device 180 or device are securely stored (as needed); an authorized person database 242 for storing information, such as images, voiceprints, fingerprints, and trust levels, of authorized users of an electronic device (e.g., electronic device 222); Event information 440, such as an event record and context information (e.g., context data describing the surroundings of an approaching visitor); a predetermined or programmed formula (also referred to herein as a recipe) of event parameters corresponding to particular configuration data 438, e.g., an event formula 442 including a particular combination of padding values ​​443a, inactivity values ​​443b, length values ​​443c, cool-off values, and / or priority values ​​443d; past images 444, such as previous background and / or object images captured by the camera under various lighting conditions; and entity information 446, such as information that identifies and / or characterizes an entity (e.g., within environment 100).

[0082] Each of the above-described elements may be stored in one or more memory devices as described above and corresponds to an instruction set for performing the functions described above. The above-described modules or programs (i.e., instruction sets) need not be implemented as separate software programs, procedures, or modules. Thus, in various embodiments, these modules may be combined or rearranged in various subsets. In some embodiments, memory 406 stores a subset of the above-described modules and data structures as needed. Memory 406 also stores additional modules and data structures not described above as needed (e.g., an account management module for linking client devices, devices, and environments).

[0083] In some embodiments, memory 406 includes an audio / display assistance application (not shown) that, when executed, prepares audio processing of audio messages received from voice-activated electronic device 190, processes the audio messages directly to extract user voice commands and instructions for cast device 108 or another voice-activated electronic device 190, and / or causes voice-activated electronic device 190 to play media content (audio or video).

[0084] In some embodiments, the server system 164 includes a crowdcast service (e.g., the crowdcast server 116 of FIG. 1A). In some embodiments, the memory 406 further includes a cast device application that, when executed, performs device provisioning associated with the cast device 108, It provides server-side functionality for performing device control and user account management. Further details of the Crowdcast functionality are described in PCT application PCT / US2015 / 64449, filed December 7, 2019, and entitled "Display Assistance Device," the entirety of which is incorporated herein by reference.

[0085] FIG. 5 includes two example event formulas (eg, formula 442 of FIG. 4) according to some embodiments.

[0086] The outdoor formula 502 is for an electronic device 222 located in an outdoor environment (e.g., an outdoor security camera or doorbell camera). In the outdoor formula 502, events are padded with two seconds of video before the initial event trigger (e.g., before motion is first detected or before a target object entering the scene is recognized) and two seconds of video after each event is completed (e.g., after motion is no longer detected). These padding values ​​are sometimes referred to herein as pre-roll and post-roll values. The inactivity threshold is 30 seconds, and the maximum event length is five hours. The outdoor formula also includes a list of target objects / events and their priorities. In some embodiments, if two objects / events are detected simultaneously during a particular portion of an event, that portion of the event is labeled with the object / event with the higher priority. In some embodiments, only objects / events with a priority higher than the threshold are used as the criteria for creating an event and / or sending a notification to a client device.

[0087] The indoor formula 504 is for electronic devices 222 located in an indoor environment (e.g., an indoor security camera or camera-equipped support device). In this example, events occurring indoors are given additional post-roll padding time (5 seconds compared to just 2 seconds for the outdoor formula). The inactivity threshold is also 30 seconds, but the maximum event length is only 1 hour. Also, because objects / events such as pets, door knocks, glass breaking, and crying babies are more likely to occur in an indoor environment and are therefore more relevant, these objects / events are given a higher priority in the event priority list than the objects / events in the outdoor formula 502.

[0088] Formulas 502 and 504 are exemplary. Other combinations of values, other device locations, and configurations may be implemented in the event formula without departing from the scope of the concepts described herein. In some embodiments, as described below, formula 442 may include baseline parameter values ​​(e.g., those shown in the example of FIG. 5 ) that are configured to change based on current configuration data, user preferences, and / or device learning algorithms.

[0089] FIG. 6 is a block diagram illustrating a representative client device 204 (client device 204 of FIG. 2 and client device 104 of FIG. 1) associated with a user account, according to some embodiments. The client device 204 typically includes one or more processors (e.g., CPUs) 602, one or more network interfaces 604, memory 606, and one or more communication buses 608 for interconnecting these elements (sometimes referred to as a chipset). Optionally, the client device also includes a user interface 610 and one or more sensors 690 (e.g., accelerometers and gyroscopes). The user interface 610 includes one or more output devices 612, such as one or more speakers and / or one or more displays, that enable presentation of media content. The user interface 610 includes one or more input devices 614, which may be user interface elements that facilitate user input, such as a keyboard, a mouse, a voice command input unit or microphone, a touchscreen display, a touch-sensitive input pad, a gesture-capture camera, or other input devices. The client device may include a keyboard, a power button, or a knob. Additionally, some client devices use a microphone and voice recognition or a camera and gesture recognition to complement or replace a keyboard. In some embodiments, the client device includes one or more cameras, scanners, or photo sensor units (not shown) for acquiring images. Optionally, the client device includes a location detection element 616, such as a GPS (Global Positioning Satellite) sensor or other geolocation receiver, for determining the location of the client device (e.g., indoors, outdoors, or a particular room or area within an environment).

[0090] The memory 606 includes high-speed random access memory such as DRAM, SRAM, DDR SRAM, or other random access solid-state memory devices, and optionally includes non-volatile memory such as one or more magnetic disk storage devices, one or more optical disk storage devices, one or more flash memory devices, or one or more other non-volatile solid-state storage devices. The memory 606 optionally includes one or more storage devices located remotely from the one or more processors (or CPUs) 602. The memory 606, or alternatively the non-volatile memory within the memory 606, includes a non-transitory computer-readable storage medium. In some embodiments, the memory 606 or the non-transitory computer-readable storage medium of the memory 606 stores the following programs, modules, and data structures, or a subset or superset thereof: an operating system 618 that includes procedures for handling various basic system services and performing hardware-dependent tasks; a network communications module 620 for connecting the client device 204 to other systems and devices (e.g., client devices, electronic devices, and systems connected to one or more networks 110) via one or more (wired or wireless) network interfaces 604; an input processing module 622 for detecting one or more user inputs or interactions from one of the one or more input devices 614 and interpreting the detected inputs or interactions; and one or more applications 623 (e.g., games, social networking applications, applications 624, and / or other web-based or non-web-based applications) executed by the client devices to control the devices (e.g., send commands to the hub device and / or other client or electronic devices, configure settings, and input configuration data for the electronic devices 222) and to review data captured by the devices (e.g., device status and settings, captured data, event video clips, or other information about the hub device or other connected devices). In some embodiments, a user can use the applications 624 to configure settings for the display assist device 190, such as on / off mode monitoring (e.g., live view, event history, notifications), home / away assist, and activity zone settings. In some embodiments, a user can use the applications 624 to schedule times for activating the cameras 362 to monitor the home. In some embodiments, a user can configure the quality of image and / or video delivery, the bandwidth used, and microphone settings via the applications 624. In some embodiments, the application 624 provides user education (e.g., training videos, manuals, pop-up message notifications) that moving the electronic device 222 affects what is or is not recorded within the activity zone. In some embodiments, the application 624 disables or adjusts the zone when the electronic device 222 is moved. In some embodiments, the electronic device 222 is configured to send a notification to the cloud (e.g., server system 164) when it is moved, The above-mentioned programs, modules, and data structures are Configuring and / or viewing settings, acquired data, and / or other data for one or more devices (e.g., devices 120 and voice-activated display assistance devices 190 within environment 100) a user interface module 626 for providing and displaying a user interface that allows a client-side module 628 that provides client-side functionality for device control, data processing, and data validation; The client-side module 628 a device control module 630 for generating control commands to change the operating mode of a device (e.g., electronic device 222, and optionally other electronic devices) according to user input; a video analysis module 632 for viewing and / or analyzing received video data (e.g., event video clips) to detect and / or recognize people, objects, animals, and events; a data verification module 634 for providing a user interface for verifying data from the server system 164 or video source 222, including, but not limited to, an event verification module 636 for verifying events (e.g., motion and / or audio events) and allowing user editing and / or updates to the events, as needed; and a people verification module 638 for verifying data and / or images regarding detected people and other entities and allowing user editing and / or updates to people data, as needed; The client-side module 628 a presentation module 640 for presenting a user interface and response options for interacting with the electronic device 222 and / or the server system 164; a remote interaction module 642 for interacting with a remote person (e.g., a visitor to the environment 100), for example, via the electronic device 222 and / or the server system 164; The above-mentioned programs, modules, and data structures are client data 644 that stores data related to user accounts and electronic devices; Client data 644 is account data 646 for storing information about both the user account loaded on the client device and the electronic device (e.g., of the video source 501) associated with the user account, including cached login credentials, hub device identifiers (e.g., MAC addresses and UUIDs), electronic device identifiers (e.g., MAC addresses and UUIDs), user interface settings, display preferences, authentication tokens and tags, password keys; a local data storage 648 for selectively storing raw or processed data related to the electronic device (e.g., of the video source 222), and the aforementioned substantive data as needed; and previous images 650, such as, but not limited to, previous background images and / or object images captured by a camera under various lighting conditions.

[0091] Each of the above-described elements may be stored in one or more memory devices as described above and corresponds to an instruction set for performing the functions described above. The above-described modules or programs (i.e., instruction sets) need not be implemented as separate software programs, procedures, modules, or data structures. Thus, in various embodiments, these modules may be combined or rearranged in various subsets. In some embodiments, memory 606 stores a subset of the above-described modules and data structures, as desired. Memory 606 also stores additional modules and data structures not described above, as desired.

[0092] 7 is a block diagram illustrating an event processing system 700 according to some embodiments. Similar features to those in FIG. 3 are indicated with similar numbers, and some of them will not be further described for brevity. In some embodiments, system 700 is implemented on an electronic device (e.g., electronic device 222) equipped with a camera. In some embodiments, various modules of system 700, such as object recognition module 354, are implemented on a server system (e.g., server system 164). System 700 processes events corresponding to detected target objects. In this disclosure, the term "event" refers to a portion of video data (e.g., a video clip) that contains something of interest to an occupant (e.g., a person or object) or that contains an occurrence of interest (e.g., motion). The term "event" may also refer to the occurrence itself (e.g., a motion event) that is the basis of a video clip. Unless otherwise specified, the terms "event," "clip," "event clip," and "video clip" are used interchangeably in this disclosure. Events, event elements, and element composition are further described below with reference to FIGS. 8 and 9.

[0093] Referring again to FIG. 7 , the image sensor of camera 362 acquires image data and stores the acquired image data as image frames in buffer 392. In some embodiments, this buffer is a circular buffer. A circular buffer ensures the availability of a constantly up-to-date log of previously acquired frames, as the oldest frame is constantly replaced by the newest frame. Trigger detection module 352 detects event triggers. In some embodiments, detecting a trigger involves detecting motion within the camera's field of view (e.g., by comparing subsequent frames to detect changes in pixel values ​​indicative of an object moving within the field of view, or by detecting motion using motion sensor 363). In some embodiments, detecting a trigger involves detecting the presence of an object in the foreground of the camera's field of view (e.g., by subtracting the current image from a background reference image to detect a foreground object, or by detecting presence using presence sensor 363). Upon detecting a trigger, object recognition module 354 determines whether the trigger represents a target object or occurrence for generating an event. In some embodiments, the object recognition module 354 performs an object or pattern recognition process (e.g., using computer vision techniques) to detect the identity of an object, the identity of a person, the type of object (e.g., a person, an animal, a vehicle, or a package), or attributes of an object that are unknown to the processing module 350 at the time of trigger detection. The event composition module 356 composes an event clip 740 (described in more detail below with reference to FIGS. 8 and 9 ) according to an event profile setting 380. In some embodiments, the event profile setting 380 is based on a formula 442 received from the server 164. In some embodiments, the server selects the formula 442 based on device configuration data of the device 222, the device configuration data being based at least in part on the device's power type. To this end, the power detection module 359 determines whether the device 222 is powered by an external power source 712 or a battery 714.In some embodiments, the power detection module 359 is connected to an external power bus and a battery power bus, and the power detection module 359 determines the power type based on the power bus being used. In some embodiments, the formula stored in the event profile settings 380 includes optimization for two types of power. Thus, the event composition module 356 composes event clips according to the power type currently detected by the power detection module 359.

[0094] In some embodiments, the power type information is set by the user during a configuration process. In some embodiments, the power type is detected by the device itself (e.g., power detection module 359), and the device 222 (e.g., event processing module 350) adjusts the event parameters 380 based on the detected power type. In some embodiments, the detected power type is sent to the server 164 and included in the formula configuration process performed by the device control module 418. In some embodiments, , dynamically updating event recording parameters 380 (e.g., based on changes in configuration data such as power type) without the need to communicate with server 164. In these embodiments, various event profiles are configured to automatically adjust, for example, upon detection of a change in power type. For example, in some embodiments, when device 222 is unplugged, the device switches to battery-powered mode, which causes the event processing module to change various event recording parameters to conserve power (e.g., shorter inactivity thresholds and event length settings, fewer target objects included in priority settings 390).

[0095] In some embodiments, the event recording formula is further updated to optimize battery life for battery-powered devices 222. For example, as the battery level and / or estimated battery life value decreases, event recording parameters such as the inactivity threshold and maximum event length may be decreased, cool-off parameters (e.g., the amount of time to wait before processing a new event) may be increased, and the number of object and target occurrences included in an event may be reduced to further conserve battery power.

[0096] In some embodiments, one or more of the above-described elements may be stored on or implemented in a server system (e.g., server system 164). For example, event processing module 350 (or one or more of modules 352, 354, 356, and 380 associated with event processing module 350) may be stored on server system 164. In such an embodiment, electronic device 222 transmits a video stream including image data acquired from camera 362 and / or image buffer 392 to server system 164, and event processing module 350 performs trigger detection, object recognition, and / or event composition on server system 164. An event clip (e.g., event clip 740) resulting from one or more of the above-described processes is transmitted from server system 164 to electronic device 222 and displayed (e.g., on output device 312 of electronic device 222).

[0097] FIG. 8 illustrates an exemplary event 810, according to some embodiments. The event is processed on electronic device 222 (e.g., by event processing system 700 of FIG. 7). For the purposes of this example, device 222 is located in a living room. However, the exact location of the device in this example is not meant to limit the concepts described herein. System 700 uses formula 802 according to the device's living room location. Living room formula 802 specifies padding parameters including a 2-second pre-roll and a 2-second post-roll, an inactivity threshold of 30 seconds, and a maximum event length of 5 hours. Timing marks (A-E) in the illustration appear sequentially over time. At time A, motion is detected (e.g., by trigger detection module 352). Object recognition module 354 determines identifying attributes of the motion. At time B, the motion is identified as being caused by a person known to system 700 (Bob). Thus, system 700 labels the event with the identity of the detected object and other information about the event (e.g., "Bob detected in the living room"). The event continues until the event time (e.g., the time elapsed since the initial trigger was detected at time A) reaches the maximum event length. At time C, Bob leaves the living room, and at time D, there is no motion, which preliminarily ends the event. The preliminarily ending of the event at time D begins an inactivity count. In this example, the inactivity threshold is 30 seconds, so the inactivity count begins at time D and ends 30 seconds later, at time E. If no trigger is detected within the 30-second inactivity window (between time D and time E), event composition module 356 terminates the event and composites a video clip for the event according to the padding parameters. The video clip begins at time A', two seconds before time A, when the trigger was detected, and ends at time D', two seconds after time D, when the subject of the event left the room. Two 2-second windows 814 and 816 (times A' and E) are used to composite the video clip. The windows 814 and 816 (the window between A and A, and the window between times D and D′) represent pre-roll and post-roll padding values ​​and are useful for indicating user-added context of the event (e.g., the state of the room just before Bob entered and the state of the room just after Bob left). The video clip of event 810 includes image data from image frames captured during padding windows 814 and 816 and image data from image frames captured during motion window 812.

[0098] FIG. 9 illustrates an exemplary composite event 910, according to some embodiments. The event is processed by electronic device 222 located in the living room and follows living room formula 802. As in FIG. 8, motion is detected at time A, an object in the scene is recognized as Bob at time B, and Bob leaves the room at time C, thereby ending the occurrence associated with the detected trigger at time D. Also as in FIG. 8, a 30-second inactivity threshold counter begins at time D. However, before the 30-second threshold is reached at time F, another motion trigger is detected at time E. This motion is determined to be associated with another known person, Charlie, at time G. Charlie leaves the room at time H, thereby ending the occurrence associated with the subsequent motion at time I. Another inactivity window begins at time I and ends 30 seconds later at time J because no additional triggers are detected during the window. The end of the inactivity window (due to the inactivity threshold being reached) triggers the creation of video clips for both events (because the subsequent event began during the inactivity window after the first event). A video clip of the composite event 910 is created according to padding values ​​914 and 916. Thus, the clip begins at time A′ (two seconds before the motion begins at time A) and ends at time I′ (two seconds after the motion ends at time I). Importantly, the video clip of the composite event 910 includes only a single pre-roll window 914 and a single post-roll window 916, and the motion window 912 includes occurrences of both detected events (e.g., detection of both Bob and Charlie). Thus, the system 700 marks the composite event with a single label that describes both occurrences (e.g., “Bob and Charlie detected in the living room”). This single label conveys information from multiple occurrences while providing a more streamlined user experience through a simpler user interface display.In other words, rather than conveying multiple closely-occurring events to the user on the display as separate events / elements, composite events that summarize all or some of the occurrences provide a cleaner approach to displaying large amounts of information that might otherwise be ignored due to the number of occurrences.

[0099] 10 shows an exemplary user interface for displaying events (e.g., events 810 and 910). In some embodiments, the user interface is implemented by the user interface module 626 of the client device 204.

[0100] User interface 1002 includes an event list 1004. Some events, such as event 1006 (corresponding to event 910), are video-based, while other events are not. For example, an event may be generated when the resident interacts with assistive device 190 by asking a question (e.g., "What time is it?") or issuing a command (e.g., play jazz music), when interacting with TV 108 (e.g., by playing a movie), when interacting with thermostat 122 (e.g., by turning on the heater), or when interacting with any device in any way. If an event includes image or video data 1006, list of events 1004 may optionally include a thumbnail 1008 containing a screenshot associated with the event (e.g., an image including both Bob and Charlie).

[0101] When a user selects event 1006 (e.g., via input 614), user interface 1022 is displayed. In some embodiments, part or all of user interface 1022 is included in user interface 1002. In some embodiments, user interface 1022 is displayed separately (as shown). Elements in both user interfaces may be mixed and matched in other combinations without departing from the scope of the concepts described herein. User interface 1022 displays video data 1024 related to the event. In some embodiments, video data 1024 can be played by selecting a video control (e.g., play, pause). The interface includes a description 1026 of the event, such as summary data (e.g., "Bob and Charlie detected"), time and location data (e.g., 3:32 PM - Living Room), and / or other information describing the event. The interface also displays a visual representation 1030 of the event's length, indicating the event timing. In some embodiments, visual representation 1030 is substantially rectangular (sometimes referred to as a pill), and its length is based on the event's length. In some embodiments, the visual representation 1030 moves (e.g., scrolls) along its long axis as the video clip 1024 plays. This indicates the position of the currently playing clip 1024 relative to the overall event. In the illustration, this is shown as a timeline with the clip 1030 already two seconds forward. Other visual representations of the event may be implemented without departing from the scope of the concepts described herein. In some embodiments, the interface includes attributes 1032 related to the detected event (e.g., results of an object recognition process). In the illustration, these attributes include the identity of the known person detected in the scene (Bob, Charlie), the type of object detected in the scene (Person), and the type of occurrence detected in the scene (Conversation).

[0102] 11 shows an exemplary user interface for obtaining device configuration data 438 (e.g., location, use, and power data 449 of FIG. 4 ) for electronic device 222. In some embodiments, the user interface is implemented by user interface module 626 of client device 204. In some embodiments, a resident uses an application (e.g., application 624 of FIG. 6 ) as part of the installation process when configuring devices in the environment.

[0103] The user interface 1110 prompts the resident to add a particular device (e.g., electronic device 222) to configure for the application. In some embodiments, the resident scans a code (e.g., a QR code) or manually enters information used by the application to identify the particular device.

[0104] The user interface 1120 prompts the resident to select a device usage (e.g., a device profile indicating its role as home or business monitoring, baby monitor, etc.). In some embodiments, the selected profile is stored on the server system 164 as device usage information 449b (FIG. 4).

[0105] The user interface 1130 prompts the resident to select a location for the device (e.g., the installation location, or, if the device is a portable device, e.g., a battery-powered security camera, the location where the device will be installed during operation). In some embodiments, the location includes a type of location (e.g., indoors, outdoors), a specific room (e.g., living room, nursery), and / or an area or zone (e.g., entrance, hallway). In some embodiments, the selected location data is stored in the server system 164 as device location information 449a (FIG. 4).

[0106] The user interface 1140 prompts the resident to select device notifications (e.g., detected objects and / or occurrences for which the resident wants to receive electronic notifications on the client device 204). In some embodiments, the notification corresponds to an identified person (e.g., known person, unknown person), a type of object (e.g., animal, vehicle, package, person), an audio occurrence (e.g., dog barking, glass breaking, baby crying, noise), or any other type of object or occurrence (e.g., those included in the example formula 442 of FIG. 5). In some embodiments, the selected notification data is stored in the server system 164 as device usage information 449b (FIG. 4).

[0107] 12 is a flow diagram illustrating an event handling process 1200 according to some embodiments. This process may be performed on an electronic device (e.g., electronic device 222) including one or more processors (e.g., CPU 302) and memory (e.g., memory 306) storing one or more programs executed by the one or more processors, a server system (e.g., server system 164) including one or more processors (e.g., CPU 402) and memory (e.g., memory 406) storing one or more programs executed by the one or more processors, and / or a client device (e.g., client device 204) including one or more processors (e.g., CPU) 602 and memory (e.g., memory 606) storing one or more programs executed by the one or more processors. In some embodiments, the electronic device, the server system, and the client device include one or more programs and memory storing the one or more programs executed by one or more processors, respectively, where the one or more programs include instructions for performing process 1200. In some embodiments, each non-transitory computer-readable storage medium stores a corresponding one or more programs, the one or more programs including instructions that, when executed by an electronic device including one or more processors, a server system including one or more processors, and a client device including one or more processors, cause the electronic device, the server system, and the client device to perform process 1200.

[0108] Process 1200 begins 1202 when client device 204 receives configuration data (e.g., one or more of location data 449a, usage data 449b, and / or power data 449c) for a particular electronic device 222. In some embodiments, the configuration data is received using one or more interfaces described above with reference to FIG. 11 . Recognizing that a user may be interested in viewing different types of event-related data based on the location of the event, the location data specifies the device's installed location (e.g., relative to user interface 1130) or a location to be monitored with the device. Further, recognizing that a user may be interested in viewing different types of event-related data based on the type of event, the usage data specifies a predetermined use of the device, for example, based on a device profile (e.g., shown in user interface 1120) or notification preferences (e.g., shown in user interface 1140). For example, a user may want to receive events from an exterior security camera if the events captured by the camera include occurrences related to a person or package within the camera's field of view. On the other hand, a user may not want to receive events from an exterior security camera if the events captured by the camera include noise- or vehicle-related occurrences. Similarly, for an electronic device 222 installed in a nursery and used as a baby monitor, the user may want to receive events related to a crying baby, but not events related to vehicles, luggage, etc.

[0109] The client device 204 sends the configuration data to the server 164. The server 164 (e.g., using the device control module 418) determines and sends 1204 one or more event formulas 442 based on the configuration data 438 to the particular electronic device 222. The control module 418 determines the event formula based on the configuration data. In other words, the server determines the device's event parameters, such as padding, inactivity threshold, and maximum event length, based on the device's location and intended use. In some embodiments, the formula is dynamic, meaning that the parameters change dynamically based on the type of event, the type of object detected, the length of the event, and / or any other attributes that define or describe the event. In some embodiments, the dynamic formula sets the parameters 443 to initial values ​​that are configured to change dynamically based on the event-related attributes described above. In some embodiments, the server sends one or more event formulas 442 to the electronic device 222 as an event recording profile. In some embodiments, the server sends the formulas 442 to the electronic device 222 individually.

[0110] The electronic device 222 retrieves, receives, or obtains (1206) an event formula from the server 164. In some embodiments, the device retrieves from the server an event recording profile that includes profile recording parameters 380. In some embodiments, the parameters 380 are set according to the event formula and / or the event recording profile that includes the formula. Event recording parameters 380 are used for specific event recording operations (e.g., recording a specific event) of device 222 and include padding parameters 382 (e.g., recording times before and after detecting a target object, as shown by 914 and 916 in FIG. 9 ), an inactivity threshold 384 (e.g., waiting time before terminating an event instead of continuing the event with subsequent activity, as shown by times D and I in FIG. 9 ), a maximum event length parameter 386 (e.g., the duration of an event before the device stops recording), a cool-off parameter 388 (e.g., the rate of object detection above which event recording stops), and / or object filter and priority settings 390 (e.g., determining the objects that serve as criteria for recording an event, as shown by the exemplary formula in FIG. 5 ). In some embodiments, the server sets these adjustable parameters based on configuration data 438 of the electronic device 222, such as (i) the location of the device (e.g., indoors, outdoors, room), (ii) the intended use of the device (e.g., what is within the field of view of the device and what the user wants to see), and / or (iii) the power type of the device (e.g., wired or battery powered).

[0111] The electronic device 222 captures, receives, or acquires (1208) a video stream (e.g., multiple images of a scene captured by the camera 362). In some embodiments, the electronic device 222 stores at least a portion of the video stream locally on the device 222 (e.g., in the buffer 392). The device 222 detects (1210) a trigger event based on the captured video stream (e.g., based on one or more multiple images of the scene obtained by detecting motion or another trigger, as described above with reference to the trigger detection module 352). In response to detecting the trigger event, the device 222 identifies (1212) a target object or occurrence from the one or more multiple images of the scene (e.g., by performing one or more object recognition processes, as described above with reference to the object recognition module 354). The device 222 creates (1214) an event clip from the images including the target object according to the event recording and processing configuration 380 (e.g., as described above with reference to the event composition module 356). The device 222 provides the event clip for display. In some embodiments, providing the event clip for display includes transmitting 1216 the event clip to the server 164 or hub 180 for storage and later viewing 1218 on the client device 204. In some embodiments, particularly if the device 222 includes a display screen, providing the event clip for display includes storing the event clip locally and displaying the event clip on the device 222 (e.g., in response to a user opening or selecting the event clip for display).

[0112] In some embodiments, to further optimize the quality of event detection and processing from the user's perspective, the event recording formula is provided to a machine learning algorithm implemented on either device 222 or server 164. For example, in some embodiments, a resident uses client device 204 to input feedback regarding one or more events (e.g., event 1006 of FIG. 10 ). Exemplary feedback includes rejection feedback (e.g., for an event and / or its underlying object or events classified by the resident as irrelevant or uninteresting) and / or customization feedback to adjust one or more event recording parameters within a particular formula (e.g., adjusting a padding value for detecting a particular type of object, adjusting a maximum event length value for detecting a particular type of occurrence). In some embodiments, the machine learning module adjusts the event profile settings 380 for a particular type of event and device configuration based on the resident's feedback.

[0113] The terms "a," "an," and "the" used in the description of the various embodiments described herein are for the purpose of describing particular embodiments only and are not intended to be limiting. The singular includes the plural unless the context clearly dictates otherwise. As used herein, the term "and / or" should be understood to refer to or encompass any combination and possible combination of one or more of the associated listed items. The terms "include," "including," "comprise," and / or "comprising," when used herein, should be understood to specify the presence of stated features, integer variables, steps, operations, elements, and / or components, but not to preclude the presence or addition of one or more other features, integer variables, steps, operations, elements, components, and / or combinations thereof. Furthermore, although terms such as "first," "second," etc. are used herein to describe various elements, these elements are not limited by these terms. These terms are used only to distinguish one element from another.

[0114] The term "if" as used herein shall be interpreted to mean "when" or "when" or "response to determining" or "response to detecting" or "following a determination that," depending on the context. Similarly, the phrase "determined" or "if [a predetermined condition or event] is detected" shall be interpreted to mean "upon determining" or "in response to determining" or "upon detecting [the stated condition or event]" or "in response to detecting [the stated condition or event]" or "following a determination that [the predetermined condition or event] is detected," depending on the context.

[0115] For purposes of explanation, the foregoing description has been set forth with reference to specific embodiments. However, the illustrative description above is not intended to be exhaustive or to limit the scope of the claims to the precise forms described. Many modifications and variations are possible in light of the above teachings. These embodiments were chosen and described in order to best explain the principles of operation and practical application so as to enable others skilled in the art to practice these embodiments.

[0116] Although the various figures depict multiple logic stages in a particular order, stages that are not order-dependent may be rearranged, and other stages may be combined or separated. While some rearrangements or groupings are specifically mentioned, other rearrangements or groupings will be apparent to those skilled in the art, and as such, the ordering and groupings described herein are not an exhaustive list of alternatives. Furthermore, it should be recognized that stages may be implemented in hardware, firmware, software, or any combination thereof. be.

[0117] For purposes of explanation, the foregoing description has been set forth with reference to specific implementations. However, the illustrative description above is not intended to be exhaustive or to limit the scope of the claims to the precise form described. Many modifications and variations are possible in light of the above teachings. The implementations are selected to best explain the principles underlying the claims and their practical application. This will enable others skilled in the art to best utilize the implementations with various modifications as appropriate for the particular use intended.

Claims

1. 1. A method comprising: receiving an event recording profile of the electronic device, the event recording profile being based on configuration data of the electronic device, the configuration data including a location type or a power type; receiving a plurality of images of a scene captured by one or more image sensors of the electronic device; detecting a trigger event based on one or more of the plurality of images of the scene; identifying a target object from one or more of the plurality of images of the scene in response to detecting the trigger event; creating an event clip from the stored image including the target object, wherein creating the event clip includes setting a clip length based on the event recording profile; providing the event clip for display.

2. The method of claim 1 , wherein setting the clip length comprises setting a padding value, an inactivity threshold, and / or a maximum event length.

3. setting the clip length includes selecting a padding value; the padding value corresponds to a number of images acquired prior to one or more of the plurality of images including the detected target object; The method of claim 1 , wherein creating the event comprises adding the number of images to the plurality of images containing the detected target object.

4. setting the clip length includes selecting an inactivity threshold; the inactivity threshold corresponds to a number of images acquired in which no target object is detected; The method of claim 1 , wherein creating the event comprises adding the number of images to the plurality of images that include the detected target object.

5. setting the clip length includes selecting a maximum event length; the maximum event length corresponds to a maximum number of images for the event; The method of claim 1 , wherein creating the event comprises ending the event when the maximum number of images is reached.

6. the configuration data includes location types corresponding to particular regions of the environmental structure; 6. The method of claim 1, wherein setting the event length based on the event recording profile comprises selecting the padding value, the inactivity threshold, and / or the maximum event length based on the particular region of the structure or environment in which the electronic device is located.

7. The setting data includes a type of power, 7. The method of claim 1, wherein setting the event length based on the event recording profile comprises selecting the padding value, the inactivity threshold, and / or the maximum event length based on whether the power type of the electronic device is wired or battery-powered.

8. The setting data further includes object priority data; Setting the event length based on the event recording profile includes: The method of claim 1 , comprising selecting the padding value, the inactivity threshold, and / or the maximum event length based on a priority of the target object identified according to priority data.

9. 9. The method of claim 1, wherein setting the clip length comprises setting a cool-off value corresponding to a time to wait between successive object detections after two or more object detections occur within a threshold time.

10. 10. The method of claim 1, wherein setting the clip length comprises setting a padding value, an inactivity threshold, and a maximum event length according to a combination of values ​​associated with the event recording profile.

11. The method of claim 1 , wherein detecting the trigger event comprises detecting motion within the scene based on analysis of two or more of the plurality of images of the scene.

12. 12. The method of claim 1, wherein detecting the trigger event comprises detecting the target object in the scene based on analysis of one or more of the plurality of images of the scene.

13. further comprising forming a composite event clip by combining the event clip with a previously created event clip; The method of claim 1 , wherein providing the event clip for display comprises providing the composite event clip for display.

14. one or more processors; and a memory storing instructions that, when executed by said one or more processors, cause said one or more processors to perform the method of any one of claims 1 to 13.

15. one or more processors; and a memory storing instructions that, when executed by said one or more processors, cause said one or more processors to perform the method of any one of claims 1 to 13.

16. A computer-readable storage medium storing instructions that, when executed by an electronic device having one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 13.

Images