Camera event processing

Abstract

In some embodiments, a method may include receiving, by an application executing on a computing system operated by a first user, data representing at least a first image acquired by a camera at a property, and causing the computing system to output a first indication of an event at the property, the first indication including at least the first image. Causing, by the application, a notification about the event to be sent to an endpoint device associated with a second user based at least in part on an input indicating that the event is a first user of a first type and preference data indicating that a notification of the first type of event is to be sent to the second user, the notification causes the endpoint device to output a second indication of the first event.

Description

Background Art

[0001] Some security systems enable remote monitoring of a location using cameras and other devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Additional examples of the present disclosure, as well as their features and advantages, will become more apparent by reference to the drawings described herein and incorporated into and forming a part of the present disclosure. The drawings are not necessarily to scale.

[0003] Figure 1A An example security system configured in accordance with some embodiments of the present disclosure is shown.

[0004] Figure 1B An example process for performing related image processing in accordance with some embodiments of the present disclosure is shown.

[0005] Figure 1C is a conceptual diagram highlighting certain benefits of the security system disclosed herein.

[0006] Figure 2 An example event table that can be used by the security system shown in Figure 1A to store events related to data is shown.

[0007] Figure 3 A first example screen that can be presented on a monitoring device of the security system shown in Figure 1A is shown.

[0008] Figure 4 An example detailed view of a delay bar in accordance with some embodiments of the present disclosure is shown. Figure 3 is shown. Figure 5 An example drop-down menu that can be presented when a monitoring agent closes Figure 3 one of the event windows shown is shown.

[0009] Figure 6 A second example screen that can be presented on a monitoring device of the security system shown in Figure 1A is shown.

[0010] Figure 7 An example customer profile table that can be used by the security system shown in Figure 1A to store customer-specific data is shown.

[0011] Figure 8A A first example screen in accordance with some embodiments of the present disclosure is shown, Figure 1AThe client application shown can present the first example screen to the client to enable selection of notification preferences.

[0012] Figure 8B A second example screen according to some embodiments of the present disclosure is shown, Figure 1A The client application shown can present the second example screen to the client to enable selection of notification preferences.

[0013] Figure 8C A third example screen according to some embodiments of the present disclosure is shown, Figure 1A The client application shown can present the third example screen to the client to enable selection of notification preferences.

[0014] Figure 9A An example screen according to some embodiments of the present disclosure is shown, Figure 1A The client application shown can present the example screen to the client so that the client can apply one or more filters to view information about a subset of events that have been detected at one or more monitoring locations.

[0015] Figure 1B An example screen according to some embodiments of the present disclosure is shown, Figure 1C The client application shown can present the example screen to the client so that the client can view information about events that have been marked for the client to review.

[0016] Figure 1A ... \br>Figure 1A Shows Figure 9B The client application shown can be based on a selection of Figure 1A The example screen presented to the client based on the selection of the "Review" user interface element shown.

[0017] Figure 10A Shows Figure 9B The client application shown can be based on a selection of Figure 10B The example screen presented to the client based on the selection of the "Agent Response" label on the screen shown.

[0018] Figure 1A Illustrates how a client according to some embodiments of the present disclosure can provide specific gestures, such as tap and hold gestures, on a thumbnail image on the screen shown to associate the face represented in the thumbnail image with a profile of an individual and / or provide feedback on the image processing performed to identify features represented in the thumbnail image. Figure 10A The client application shown can, in response to an input selecting a thumbnail image, such as

[0019] Figure 10C An example screen according to some embodiments of the present disclosure is shown, Figure 10A The client application shown can, in response to an input selecting a thumbnail image, such as Figure 10Dpresent the example screen on the client device by way of, e.g., the tap and hold gesture illustrated Figure 1A illustrates an example implementation of a security system including Figure 10C the various components shown.

[0020] Figure 11 illustrates an example implementation of Figure 1A a base station of the security system shown, according to some embodiments of the present disclosure.

[0021] Figure 12 illustrates an example implementation of Figure 11 a keypad of the security system shown, according to some embodiments of the present disclosure.

[0022] Figure 13 illustrates an example implementation of Figure 1 1 a security sensor of the security system shown, according to some embodiments of the present disclosure.

[0023] Figure 14 illustrates an example implementation of Figure 11 a monitoring center environment and a surveillance center environment of the security system shown, according to some embodiments of the present disclosure.

[0024] Figure 15 is a sequence diagram of a monitoring process that can be performed by components of the security system according to some embodiments of the present disclosure Figure 11 shown.

[0025] Figure 16 illustrates an example process for establishing a peer-to-peer connection between components of the security system according to some embodiments of the present disclosure for enabling streaming of video and / or audio data Figure 11 shown.

[0026] Figure 17 is a sequence diagram showing an example signaling process according to some embodiments of the present disclosure that can be used to establish a peer-to-peer connection between components of the security system Figure 11 shown to enable streaming of video and / or audio data.

[0027] Figure 18 is a schematic diagram of a computing device according to some embodiments of the present disclosure that can be used to implement Figure 11 and Figure 19 a client device, a monitoring device, and / or one or more services of the security system shown.

[0028] Figure 1A illustrates an example token that can be employed by various components of the system disclosed herein, according to some embodiments of the present disclosure.

[0029] Figure 11 illustrates an example process that can be performed by the edge image processing component and / or the remote image processing component shown in Figure 20 accordance with some embodiments of the present disclosure.

[0030] Figure 21 illustrates an example process that can be performed by one or more components of a security system to generate Figure 1A the delay bar shown in accordance with some embodiments of the present disclosure.

[0031] Figure 22 illustrates an example process that can be performed by the Figure 4 monitoring application shown in accordance with some embodiments of the present disclosure to send a notification about an event to a customer based on the customer's notification preferences.

[0032] Figure 23 illustrates an example process that can be performed by one or more components of a security system to allow a customer to specify one or more filters to be applied to an event presented on the screen of a customer device, in accordance with some embodiments of the present disclosure. Figure 1A

[0033] Figure 24 illustrates an example process that can be performed by the Figure 1A monitoring application shown in accordance with some embodiments of the present disclosure to flag events viewed by a customer based on the customer's event viewing preferences.

[0034] Figure 25 illustrates an example process that can be performed by the Figure 1A customer application shown in accordance with some embodiments of the present disclosure to enable a customer to provide feedback for changing one or more functions of an image processing component. DETAILED DESCRIPTION

[0035] ​Some security systems provide a mechanism for notifying customers of various events detected at a property. However, a customer's smartphone is already loaded with applications that inundate the customer with various types of notifications. Since security cameras tend to detect a large number of harmless events, adding a camera-based application to this mix can significantly increase the volume of such notifications. Customers using such camera-based applications are forced to sift through a large number of security event notifications they receive to separate important notifications (e.g., notifications regarding actual security issues) from "noise" (e.g., notifications regarding harmless events such as swaying trees, authorized visitors, yard service personnel, etc.). Additionally, when receiving event notifications from such applications, customers typically need to view and / or fast forward video recordings to determine what happened and / or who appeared. Although some advanced applications have added rich notifications that include thumbnails of first key frames extracted from video events processed using computer vision (CV) and have a preroll feature that starts the video approximately five seconds before the key frame is captured, due to the inaccuracy of the CV processing employed, users of such advanced applications still tend to be inundated with a large number of irrelevant notifications. Thus, reviewing event notifications in existing systems can be a tedious and time-consuming process that results in a poor customer experience. Some existing systems use CV solutions to reduce or prioritize the event notifications sent to customers. Although helpful to some extent, the CV solutions used by such systems still allow an unacceptable amount of "noise" notifications to reach customers. The inventors have recognized and realized several reasons for this performance shortfall. First, existing systems only run the CV process on cameras monitoring the property and / or associated base stations. This processing can be referred to as "edge processing" or "processing at the edge". Such edge processing often has self-imposed limitations to reduce the cost of hardware products, thereby requiring the use of CV models with limited capabilities. Thus, using such CV models with limited performance often results in incorrect or inadequate feature detection.

[0036] Additionally, existing systems typically do not include any mechanism capable of performing another level of verification on an event (e.g., by a monitoring agent) before the notification of the event is sent to the customer to determine whether the CV model accurately classifies the event. Instead, in such systems, all events detected and / or labeled as potential security issues by the CV model are immediately notified to the customer without any further verification of the events. Furthermore, in existing systems that employ CV processing to process video recordings, customers are unaware at any time of the specific CV model being used, the frames processed by such CV model, or the predictions made by them. This lack of transparency can drive customers to fear the unknown (e.g., whether their privacy is being violated and in what way), and thus turn off any and all features involving CV processing.

[0037] A security monitoring system is provided that is configured to minimize the number of irrelevant event notifications sent to customers and also allows customers to specify the types of events for which notifications are to be sent. In some embodiments, the system may additionally allow customers to understand and / or provide feedback on how to view and analyze images from the customer's property. In some embodiments, the system may use a combination of image processing (e.g., using one or more trained machine learning (ML) models or otherwise performing CV processing) and subsequent review / verification to reliably screen and classify events detected by security cameras. For example, one or more computing devices may be configured to perform supplementary review / verification of events and / or enable a human monitoring agent to review / verify events after the image data has been processed by one or more image processing components, to allow the (one or more) computing devices / monitoring agent to evaluate the accuracy of feature predictions (e.g., "motion", "person", "face", "recognized face") made by these (one or more) image processing components and classify the events (e.g., classify as "urgent", "immediate", "trusted person", "service / delivery person", "unidentified person", "general event", etc.).

[0038] After the (one or more) computing devices / monitoring agent have performed supplementary review / verification of the events, notifications about the events can be sent to the customer according to the customer preference settings (e.g., as a push notification, short message service (SMS) message, email, phone call, etc.). Information about the events can be additionally or alternatively added to an event list again based on the customer preference settings for the customer to view, for example, via a mobile application. The information provided for viewing in this way can include details about the feature predictions made for the events (e.g., as a timeline showing what the image processing components predicted occurred at a specific time) and / or the sequence of actions taken by the monitoring agent to review the events. One or more user interface elements can also be provided to the customer that enable the customer to provide feedback on the feature predictions made. The system can also make similar information about other events processed by the system available for viewing in a similar manner and can enable the customer to apply one or more filters to specify one or more criteria, such as the type of event, the time the event occurred, the location where the event occurred, the camera used to capture the event, etc., to control the events for which such information is provided.

[0039] In some embodiments, videos / frames can be processed both on the camera and in the cloud, e.g., processed by a server operating in a remote computing environment. Performing image processing in the cloud (e.g., using one or more trained ML models or otherwise performing CV processing) can provide substantially unlimited capabilities for processing videos / frames of events from the camera. For example, frames of an event can first be processed on the camera, allowing one or more image processing models at the edge to identify key frames, temporarily store the identified frames, and send these frames to the cloud for further processing. One or more image processing models in the cloud can then use more processing power to process at least the same key frames to further eliminate event “noise”. When the event video recording is shared with the cloud service, additional image data of the event (e.g., additional frames of the recorded video) can also be processed by the cloud-based image processing model(s), allowing additional models that cannot run at the edge to identify additional key frames. In some embodiments, the sequence and frequency at which certain image processing models process videos / frames can sometimes depend on the results of predictions of other imaging processing models. This multi-layer or “piggybacked” approach can further eliminate “noisy” events, reduce latency in the overall system / process, and save hosting costs.

[0040] In some embodiments, a customer can, for example, use a mobile app on a smart phone to view a carousel of thumbnail images associated with an event to determine what / who was present during the event without playing and / or navigating through the event video recording. The customer may not be able to play the video due to reasons such as cellular or WIFI restrictions or personal circumstances (e.g., in a work meeting). If the customer chooses to view the event video, they can easily jump to that point within the event video recording using timeline markers indicating predictions and types of predictions. The customer can also provide feedback, for example, by accepting or rejecting thumbnails, which can improve the performance of the image processing model(s) by, for example, retraining the model(s) using the feedback, thereby further reducing noise. The customer can alternatively or additionally choose to mark images and assign them to a profile or cluster, which can be used as a filter to eliminate events that contain known people that the customer deems to be “noise”.

[0041] Some embodiments of the present disclosure provide transparency to customers by presenting all or nearly all processing frames and predictions for each event to the customer. The customer can filter their timeline and set notification preferences for events containing one or many specific types of predictions. For example, the customer's timeline can be filtered to show only events containing a selected prediction type, and the displayed thumbnails can also be filtered to show only those same prediction types. For example, for a customer who only wants to view events containing people, a timeline showing only events involving people can be presented, and the event will only show thumbnails containing people predictions.

[0042] To facilitate an understanding of the principles of the present disclosure, reference will now be made to the examples illustrated in the accompanying drawings, and specific language will be used to describe these examples. However, it should be understood that it is not intended to limit the scope of the examples described herein.

[0043] Figure 26 An example security system 100 configured in accordance with some embodiments of the present disclosure is shown. As shown, the security system 100 can include: one or more cameras 102 disposed at monitoring locations 104 (e.g., a residence, a company, a parking lot, etc.); a monitoring service 106 (e.g., including one or more servers 108), which is located remotely from the (one or more) cameras 102 (e.g., within a cloud-based service such as the monitoring center environment 1126 described below in conjunction with Figure 1A and Figure 1A ); one or more monitoring devices 110, which are operated by respective monitoring agents 112; and one or more customer devices 114, which are operated by respective customers 116. Although not illustrated in Figure 11 , it should be understood that the various illustrated components can communicate with each other via one or more networks (e.g., the Internet).

[0044] The camera 102 can include a motion sensor 130, an image sensor 118, and an edge image processing component 120, among other components. The monitoring service 106 can include a remote image processing component 122 and one or more data storages 124, among other components. The (one or more) data storages 124 can correspond to, for example, the location data storage 1502 and / or the image data storage 1504 described below in conjunction with Figure 15 . The monitoring device 110 can include a monitoring application 126 (e.g., executed on one or more processors of the monitoring device 110), among other components. Finally, the customer device 114 can include a customer application 128 (e.g., executed on one or more processors of the customer device 114), among other components. As shown, the remote image processing component 122, the monitoring application 126, and the customer application 128 can all communicate with the (one or more) data storages 124, e.g., via one or more networks, such as those described below in conjunction withFigure 1A and Figure 15 the network 1120 described. In some embodiments, the monitoring service 106 or another component within the monitoring center environment 1126 (see Figure 11 and Figure 15 ) may provide one or more application programming interfaces (APIs) that can be used by the remote image processing component 122, the monitoring application 126, and the client application 128 to write data to and / or retrieve data from the (one or more) data stores 124 as needed.

[0045] As Figure 11 an example, the image sensor 118 may obtain image data (e.g., digital data of a frame representing one or more acquired pixel values) from the monitoring location 104 and pass the image data to the edge image processing component 120 and / or the remote image processing component 122 for processing. In some embodiments, for example, the motion sensor 130 may detect motion at the monitoring location 104 and provide an indication of the detected motion to the image sensor 118. The motion sensor 130 may be, for example, a passive infrared (PIR) sensor. In response to the indication of the detected motion, the image sensor 118 may begin acquiring frames of image data from within the field of view of the camera. In some embodiments, the image sensor 118 may continue to collect frames of image data until the motion sensor 130 does not detect motion for a threshold period of time (e.g., twenty seconds). Thus, the image data acquired by the image sensor 118 may represent a video clip of the scene within the field of view of the camera that begins when motion is first detected and ends after the motion has stopped for the threshold period of time. In some embodiments, rather than relying on the motion sensor 130 (e.g., a PIR sensor) to trigger the collection of frames of image data, the camera 102 may continuously collect frames of image data and rely on one or more image processing modules (e.g., machine learning (ML) models and / or other CV processing components) of the edge image processing component 120 to process the collected frames to detect motion within the field of view of the camera 102. Thus, in such embodiments, rather than relying on the motion indication provided by the motion sensor 130 to determine the start and end of the video clip for further processing, the camera 120 may rely on the motion indication provided by such (one or more) image processing modules for that purpose.

[0046] The edge image processing component 120 may include one or more first image processing modules (e.g., machine learning (ML) models and / or other CV processing components) configured to identify corresponding features within the image data, and the remote image processing component 122 may include one or more second different image processing modules (e.g., ML models and / or other CV processing components) configured to identify corresponding features within the image data. The first image processing module and / or the second image processing module may be configured, for example, to perform processing on the image data to detect motion, identify people, identify faces, perform face recognition, etc. In some embodiments, the processing capabilities of the (one or more) servers 108 employed by the monitoring service 106 may be significantly higher than the processing capabilities of the (one or more) processors included in the edge image processing component 120, thereby allowing the monitoring service 106 to employ more complex image processing modules and / or execute a greater number of such image processing modules in parallel.

[0047] In some embodiments, the processing performed by one or more first image processing modules of the edge image processing component 120 may be used to notify and / or enhance the processing performed by one or more second image processing modules of the remote image processing component 122. As an example, one or more first image processing modules of the edge image processing component 120 may perform basic processing to identify key frames within the image data that potentially represent motion, people, faces, etc., and one or more second image processing modules of the remote image processing component 122 may perform more complex processing only on the key frames identified by one or more first image processing modules of the edge image processing component 120. As another example, one or more first image processing modules of the edge image processing component 120 may perform processing on the image data to identify specific frames including motion, and one or more second image processing modules of the remote image processing component 122 may perform processing to detect people only on the specific frames identified by one or more first image processing modules of the edge image processing component 120. As yet another example, one or more first image processing modules of the edge image processing component 120 may perform processing on the image data to identify specific frames including images of people, and one or more second image processing modules of the remote image processing component 122 may perform processing to detect and / or identify faces only on the specific frames identified by one or more first image processing modules of the edge image processing component 120. As yet another example, one or more first image processing modules of the edge image processing component 120 may perform processing on the image data to identify specific frames including face images, and one or more second image processing modules of the remote image processing component 122 may perform processing to perform enhanced face recognition and / or identify faces only on the specific frames identified by one or more first image processing modules of the edge image processing component 120.

[0048] Further, in some embodiments, the remote image processing component 122 may itself perform processing using multiple different image processing models, where certain image processing modules depend on results obtained by one or more other image processing modules.

[0049] Figure 15 is a flowchart showing an example process 131 that may be used by the monitoring service 106 to perform related image processing according to some embodiments of the present disclosure. As Figure 1A shown, the process 131 may start at step 132, where the monitoring service 106 may receive the next frame of the recorded video data of the event from the camera 102 (e.g., from the image sensor 118 or the edge image processing component 120).

[0050] At step 134 of the process 131, the monitoring service 106 may cause one or more first image processing modules of the remote image processing component 122 to process the frame (and possibly one or more adjacent frames) to determine whether the frame includes a moving object. In some embodiments, for example, motion may be detected by detecting significant differences between frames of image data using one or more functions of the OpenCV library (accessible at the Uniform Resource Locator (URL) "opencv.org").

[0051] According to the determination 136, if the monitoring service 106 determines that the frame includes a moving object, the process 131 may proceed to step 138, where the monitoring service 106 may cause one or more second image processing modules of the remote image processing component 122 to process the frame to determine whether the frame includes a person. However, if the monitoring service 106 determines (at the determination 136) that the frame does not include a moving object, the process 131 may instead terminate. An example of an ML model that can be used for person detection is YOLO (accessible via the URL "github.com").

[0052] According to the determination 140, if the monitoring service 106 determines that the frame includes a person, the process 131 may proceed to step 142, where the monitoring service 106 may cause one or more third image processing modules of the remote image processing component 122 to process the frame to determine whether the frame includes a face. However, if the monitoring service 106 determines (at the determination 140) that the frame does not include a person, the process 131 may instead terminate. An example of an ML model that can be used for face detection is RetinaFace (accessible via the URL "github.com").

[0053] According to determination 144, if the monitoring service 106 determines that the frame includes a face, process 131 may proceed to step 146, where the monitoring service 106 may cause one or more fourth image processing modules of the remote image processing component 122 to perform an enhanced face recognition process to more accurately identify and locate the face in the frame. However, if the monitoring service 106 determines (at determination 144) that the frame does not include a face, process 131 may instead terminate. An example of an ML model that can be used for enhanced face detection is MTCNN_face_detection_alignment (accessible via the URL "github.com").

[0054] Finally, after one or more fourth image processing modules of the remote image processing component 122 more accurately identify and locate the face in the frame according to step 146, process 131 may proceed to step 148, where the monitoring service 106 may cause one or more fifth image processing modules of the remote image processing component 122 to perform face recognition on the frame (e.g., by generating biometric embeddings of the detected face and comparing these embeddings with a library of known faces) to attempt to determine the identity of a person based on the recognized face. An example of an ML model that can be used for face recognition is AdaFace (accessible via the URL "github.com").

[0055] The following will be combined with Figure 1B Describe another example process 2100 that can be executed by the edge image processing component 120 and / or the remote image processing component 122.

[0056] After the remote image processing component 122 has detected and / or confirmed the presence of one or more relevant features (e.g., motion, person, face, recognized face, etc.) in the image data, the remote image processing component 122 may upload the image data and data reflecting the recognized features (collectively referred to as "event data") to one or more data stores 124 (e.g., to the rows of the event table 202 described below in combination with Figure 21 Describe) for further review (e.g., by operating the monitoring agent 112 of the monitoring application 126 and / or the customer 116 of the customer application 128). In other embodiments, the image data (e.g., video corresponding to the motion detected by the motion sensor 130) may be uploaded to the (one or more) data stores 124 (e.g., to the corresponding rows of the event table 202) when it is acquired, and the remote image processing component 122 may simply supplement the uploaded image data with data about the (one or more) features recognized by the remote image processing component 122 (e.g., by filling one or more columns of the event table 202). Figure 2is a conceptual diagram highlighting certain benefits of the security system 100 disclosed herein. As Figure 1C shown, in some embodiments, sending a notification to a client device 114 (e.g., via a push notification service 150, an email service 152, a short message service (SMS) 154, or an audio or telephone service 156) can be delayed to allow the remote image processing component 122 and / or the monitoring agent 112 of the operation monitoring device 110 to process / view the image data acquired by the camera 120. This can be contrasted with conventional methods where, upon detection of a motion event, the camera immediately sends a push notification directly to the client device. The additional processing performed by the remote image processing component 122 and / or the monitoring agent 112 can allow the remote image processing component 122 and / or the monitoring agent 112 to add additional context or descriptive information to one or more notifications 158a, 158b sent to the client device 114, and can allow (e.g., via the monitoring device 110) the selection of an appropriate delivery method for such a notification 158 (one or more) via, for example, the push notification service 150, the email service 152, the SMS service 154, or the audio service 156. r>Figure 1C An example event table 202 that can be used to store event data for various events detected by the system 100 is shown. As shown, for each event, the event table 202 can be populated with data representing an event identifier (ID) 204, a timestamp 206, a location ID 208, a camera ID 210, image data 212, one or more feature predictions 214, one or more agent actions 216, a cancellation reason 218, an event disposition 220, a client review status 222, and a client feedback 224, etc.

[0057] The event ID 204 can represent different events that the security system 100 has detected, and the data in the same row as a given event ID 204 can correspond to that same event.

[0058] The timestamp 206 can indicate the date and time when the corresponding event was detected.

[0059] The location ID 208 can identify the monitoring location 104 where the event was detected.

[0060] The camera ID 210 can identify the camera 102 associated with the corresponding detected event.

[0061] The image data 212 can represent one or more images (e.g., snapshots or videos) acquired by the camera 102 identified by the corresponding camera ID 210 at the time of event detection.

[0062] One or more feature predictions 214 may include information about one or more features identified by the edge image processing component 120 and / or the remote image processing component 122. Such information may include, for example, an indication of motion detected during an event, an indication of a person detected corresponding to the event, an indication of a face detected corresponding to the event, a threat level assigned to the event, and the like. In some embodiments, the feature prediction 214 may also include information identifying a temporal location within the video clip at which the corresponding feature is identified by the edge image processing component 120 and / or the remote image processing component 122. As explained below in connection with Figure 2 , Figure 3 , Figure 4 , Figure 6 , Figure 9A , Figure 9B and Figure 10A , such timing information may be used, for example, to present one or more feature indicators 408, 1034 at appropriate locations on the delay bars 310, 1020 to allow playback of the video clip to begin at or shortly before the time the feature is identified (e.g., in response to selection of the feature indicators 408, 1034, thumbnail image 1022, feature descriptor 1036, time indicator 1038, or other UI element corresponding to the identified feature), and / or to associate the feature identifier with the thumbnail image for presentation to the monitoring agent 112 (e.g., via the monitoring application 126) and / or the customer 116 (e.g., via the customer application 128).

[0063] One or more agent actions 216 may represent one or more actions taken by the monitoring agent 112 during its review of an event. One or more agent actions may include, for example, an indication that the monitoring agent 112 is validating an indication of an event (e.g., when the event is in the monitoring agent's queue of events to be reviewed), an indication that the event is being actively reviewed by the monitoring agent 112 (e.g., when the monitoring agent is viewing a live camera feed from the monitoring location 104), an indication that the monitoring agent 112 is following up on an event (e.g., to provide a notification to a user and / or complete the disposition of the event), and an indication that the monitoring agent 112 is resolving an event, and the like. As explained below in connection with Figure 10B and Figure 9A , in some embodiments, this information may be used to provide real-time or near real-time status information to the customer 116 (e.g., via the customer application 128) regarding what the monitoring agent 112 is currently doing to resolve an event detected by the security system 100. Further, as explained below in connection with Figure 9BAs explained, this data can alternatively or additionally be used in conjunction with other data (e.g., (one or more) feature predictions 214, data indicating when an event is placed in the review queue of the monitoring agent, etc.) to generate information 1016 that can be presented (e.g., via the client application 128) on the client device 114 in the form of an "agent response" description.

[0064] The cancellation reason 218 can represent the reason provided by the monitoring agent 112 for determining that the event does not present a security issue. As explained below in connection with Figure 10B , Figure 3 and Figure 5 such a reason can be provided, for example, when the monitoring agent 112 selects the closing element 312 of the event window 306 of the review queue of the monitoring agent, or when it is determined that there is no security issue after reviewing live video from a monitoring location within one or more video feed windows 604 or the main viewer window 606 and / or after reviewing one or more event data windows 608. The event disposition 220 can represent the disposition of the event after being reviewed by the monitoring agent 112. The event disposition 220 can indicate, for example, that the event is an emergency event (e.g., when a life - threatening or violent situation occurs) or an urgent event (e.g., package theft, property damage, or vandalism), that the event is handled by the monitoring agent 112, that the police or fire department is dispatched, that the event is cancelled after a person accurately provides a security password, that the event is cancelled by the customer 116 (e.g., via the client application 128), etc. As discussed in more detail below, in some embodiments, the mentioned event disposition 220 can be used, for example, to determine whether to send a notification (e.g., a push notification, an SMS message, an email, etc.) to the customer 116, whether to flag the event for review by the customer 116 (e.g., on the "Events for Review" screen 904 as shown in Figure 6 ), such that events can be filtered based on one or more customer - specified criteria (e.g., via the "Event Type" user interface element 912c on the screen 902 as shown in Figure 9B ), and / or to populate various fields that can be included on the screens 902, 904, 1002, 1004 as shown in Figure 9A , Figure 9A , Figure 9B and Figure 10A .

[0065] The customer review status 222 can indicate, for example, that the event is waiting for review by the customer 116 (e.g., by appearing in the "Events for Review" queue - see Figure 10B ), that the event has been reviewed and "dismissed" by the customer 116 (e.g., as described below in connection with br>Figure 9B ), or that the event has not been flagged for customer review. As explained below in connection with Figure 9B and Figure 7More specifically, it can be determined whether a particular event is queued for a given customer 116 to view based on notification preferences 704 and / or event viewing preferences 706 that can be set by or for the corresponding customer 116.

[0066] Customer feedback 224 can represent information provided by customer 116 based on the customer's viewing of an event (e.g., via customer application 128). For example, as described below in connection with Figure 9B and Figure 10A In some embodiments, the customer can identify instances where they believe the edge image processing component 120 and / or the remote image processing component 122 have incorrectly identified features, or can indicate whether or how they believe the event may have been incorrectly processed by the monitoring agent 112. In some embodiments, customer feedback 224 can be used to retrain or otherwise modify the image processing module (e.g., the (one or more) ML models and / or the (one or more) other CV processing components) used to identify features, so that the monitoring agent 112, etc. can be trained or retrained.

[0067] Although Figure 10B not illustrated in, it should be understood that the event table 202 can additionally include other data, which can be used for various purposes, such as facilitating the assignment of events to specific agents, the description of events (e.g., "Motion detected by backyard camera"), the alarm status of the monitoring location 104, one or more recorded audio tracks associated with the event, the status change of one or more sensors (e.g., door lock sensor) at the monitoring location 104, etc. As described in more detail below in connection with Figure 2 In some embodiments, some such data can be used to populate one or more event data windows 608 that can be displayed by the monitoring device 110, and one or more live video feeds within the (one or more) video feed windows 604 and / or the main viewer window 606.

[0068] Notifications regarding "actionable" events (e.g., events where the remote image processing component 122 identifies one or more features of interest) represented in the event table 202 can be dispatched to the corresponding monitoring application 126 for viewing by the monitoring agent 112. In some embodiments, the monitoring service 106 can use the content of the event table 202 to assign individual events to the respective monitoring agents 112 that are currently online with the monitoring application 126. The monitoring application 126 operated by a given monitoring agent 112 can then add the events assigned to that monitoring agent 112 to an event queue for viewing by that monitoring agent 112. When an event is added to the viewing queue of a monitoring agent, the monitoring application 126 can cause an indication that the event has been verified by the monitoring agent 112 to be added to the event table 202, as the agent action 216. As described below in connection with Figure 6 andFigure 9A and Figure 9B As discussed in more detail, in some embodiments, the mentioned proxy action 216 can be used to cause the client application 128 to provide real-time or near-real-time information 914 about the actions taken by the monitoring agent 112 and / or to provide historical information 1016 about the actions taken by the monitoring agent 112 when reviewing events.

[0069] Figure 10B An example screen 302 that can be presented on the monitoring device 110 of the security system 100 is shown. As shown, the monitoring device 110 can be operated by the monitoring agent 112, and the screen 302 can include a set of event windows 306 corresponding to the respective events currently in the queue of the monitoring agent for review. In some embodiments, for example, each event window 306 can be configured to playback a recorded video corresponding to the respective event detected at the respective monitoring location 104. As used herein, a "monitoring location" can correspond to a specific location monitored for security or other purposes (e.g., a house and associated lot, office space, parking lot, etc.). A given monitoring location 104 can be associated with a particular customer, for example, via a customer ID. In some embodiments, the recorded video can be configured and / or played back at an increased rate (e.g., twice the standard speed) at least initially to increase the rate at which the monitoring agent 112 can review the video for potential threats or objects of interest.

[0070] As Figure 3 shown, in some configurations, the screen 302 can include a control interface 308 that includes one or more user interface (UI) elements to allow the monitoring agent 112 to control various aspects of the agent's queue, such as the maximum number of notifications that can be added to the agent's queue for presentation in the respective event windows 306. In some embodiments, event notifications can be distributed to each of the monitoring agents 112 included in a pool of available monitoring agents 112 such that all available monitoring agents 112 have approximately the same number of events in their review queues at a given time.

[0071] Also as Figure 3 shown, in some embodiments, each event window 306 can include a delay bar 310 that includes various features and controls to facilitate review of the video presented in the event window 306. Figure 3 A detailed view of an example delay bar 310 is shown in Figure 4As shown, the delay bar 310 may include a playback progress indicator 402 and an associated time indicator 404 showing the time position of the currently displayed image within the recorded video clip. In the illustrated example, the playback progress indicator 402 and the time indicator 404 indicate that the current image is from the seventh second of a twenty-two second long video clip. The delay bar 310 may also include a play / pause button 406 that may be selected to switch between a "play" mode and a "pause" mode, in which the video clip is played back in the "play" mode and the playback of the video clip is paused at a particular frame in the "pause" mode. The monitoring agent 112 may additionally navigate to a particular time position within the recorded video clip by selecting (e.g., clicking) a particular position on the playback progress indicator 402.

[0072] Advantageously, the delay bar 310 may additionally include feature indicators 408a, 408b, 408c corresponding to features of the image data identified by one or more image processing modules of the edge image processing component 120 and / or the remote image processing component 122. In some embodiments, the color and / or vertical position of the respective feature indicator 408 may indicate the type of feature prediction performed. For example, in the illustrated example, the feature indicator 408a may correspond to the detection of a person by the edge image processing component 120 and / or the remote image processing component 122, the feature indicator 408b may correspond to the detection of motion by the edge image processing component 120 and / or the remote image processing component 122, and the feature indicator 408c may correspond to the detection of a face by the edge image processing component 120 and / or the remote image processing component 122. The inclusion of such feature indicators 408 allows the monitoring agent 112 to quickly navigate to and review portions of the video clip identified by the edge image processing component 120 and / or the remote image processing component 122 as including particular features of potential interest. As described above, the data enabling the presentation of the feature indicators 408 may be stored as the feature prediction 214 of the event table 202.

[0073] In some embodiments, when the edge image processing component 120 and / or the remote image processing component 122 identify a feature in a frame of the image data acquired by the image sensor 118, metadata identifying the relative position of the frame within the sequence of frames acquired for the event (e.g., a frame identifier or a timestamp) may be stored as a component of the feature prediction 214, thereby enabling the corresponding feature indicator 408 to be placed at the correct relative position on the delay bar 310. Accordingly, the monitoring application 126 may be configured such that selecting one of the displayed feature indicators 408 causes the playback of the recorded video to begin at or shortly before the time at which the corresponding feature was identified. Although not illustrated in Figure 4 , in some embodiments, the frame 302 may additionally or alternatively present one or more other UI elements corresponding to the identified features (e.g., as described below in connection withFigure 3 The described thumbnail image 1022, feature descriptor, and / or time indication 1036 (similar thumbnail images, feature descriptors, and / or time indications), and these UI elements can be similarly selected such that the playback of the recorded video starts at or shortly before the time when the corresponding feature is recognized.

[0074] In some embodiments, the customer 116 operating the customer device 114 can also be provided with a delay bar 310 including the same or similar features when presented with a video clip in response to selecting an event under the "Events to Review" label 906 or the "All Events" label 908, including feature indicators 408a, 408b, 408c, as described below in connection with Figure 10A and Figure 9A further described. Below in connection with Figure 9B Describe an example process 2200 that can be performed by one or more components of the security system 100 to generate the delay bar 310.

[0075] When reviewing an event window 306, for example, by viewing a recorded video clip corresponding to the detected motion, the monitoring agent 112 can determine that there is no potential security threat and provide an input indicating to the monitoring application 126 to remove the event notification from the agent's review queue, thereby releasing the corresponding event window 306 to display another event notification. Such an input can, for example, involve selecting (e.g., clicking) the close element 312 of the event window 306.

[0076] In some embodiments, the monitoring agent 112 can identify the reason for removing each notification from the agent's queue, for example, by selecting an option from a drop-down menu presented when selecting the close element 312. Figure 22<\br >Figure 5 An example drop-down menu 502 that can be presented on the screen 302 in response to selecting the close element 312 is shown. As indicated, example reasons for canceling an event notification can include "duplicate event", "delivery", "no person", "passerby", "outdoor service", "household activity", "technical problem", "weather event", "pet or other animal", "adjacent activity", and "other". In response to the selection of such a reason by the monitoring agent 112, the monitoring application 126 can input an indication of the reason as the cancellation reason 218 of the notification into the event table 202. As discussed in more detail below, in some embodiments, for example, the mentioned cancellation reason 218 can be used to determine whether to send a notification (e.g., a push notification, an SMS message, an email, etc.) to the customer 116, whether to mark an event for the customer 116 to review (e.g., on the Figure 9B shown "Events to Review" screen 904), and / or enable filtering of events based on one or more customer-specified criteria (e.g., viaFigure 9A The "event type" user interface element 912c) on the depicted screen 902.

[0077] The monitoring application 126 can additionally input an indication that an event has been resolved by the monitoring agent 112 as a proxy action 216 into the event table 202. As discussed in more detail below in conjunction with Figure 9A 、 Figure 9B and Figure 10B In some embodiments, the mentioned proxy action 216 can be used to cause the client application 128 to provide real-time or near-real-time information 914 about the actions taken by the monitoring agent 112 and / or to provide historical information 1016 about the actions taken by the monitoring agent 112 when reviewing events.

[0078] Alternatively, after reviewing an event window 306 (e.g., by viewing a recorded video clip corresponding to the detected motion), the monitoring agent 112 may determine that there is a potential threat or other security issue (referred to herein as an "incident") and determine that further review is needed. In such a case, the monitoring agent 112 can click or otherwise select the event window 306 that displays the recorded video in question. In response to such a selection, the monitoring device 110 can begin receiving live video and / or audio streamed from one or more cameras at the monitoring location 104, and / or the monitoring agent 112 can otherwise be provided with additional data (e.g., other recorded video and / or audio, still images, sensor data, artificial intelligence (AI) assessment results, etc. from the edge image processing component 120 and / or the remote image processing component 122), enabling the monitoring agent 112 to evaluate whether the incident presents an actual security risk. In some embodiments, for example, one or more peer-to-peer connections can be established between one or more cameras 102 at the monitoring location 104 and the monitoring device 110 using the Web Real-Time Communication (WebRTC) function of a browser on the monitoring device 110 to enable the streaming of video data and / or audio data between the (one or more) cameras 102 and the monitoring device 110. Example procedures for securely establishing peer-to-peer connections between the monitoring device 110 and the cameras 102 to enable such live streaming are described below in conjunction with Figure 17 and Figure 18 When the monitoring agent 112 begins to review an event (e.g., by selecting the event window 306), the monitoring application 126 can cause an indication that the event is being actively reviewed by the monitoring agent 112 to be added to the event table 202 as a proxy action 216. As discussed in more detail below in conjunction with Figure 9A 、 Figure 9B and Figure 10BAs discussed in more detail, in some embodiments, the mentioned agent action 216 can be used to cause the client application 128 to provide real-time or near real-time information 914 about the actions taken by the monitoring agent 112 and / or provide historical information 1016 about the actions taken by the monitoring agent 112 when reviewing events.

[0079] Figure 6 illustrates an example screen 602 that can be presented by the monitoring device 110 in response to the selection of Figure 3 one of the illustrated event windows 306. In the illustrated example, the screen 602 includes three video feed windows 604 that are configured to display streaming video from three different cameras 102 at the monitoring location 104 corresponding to the selected event window 306. Although Figure 6 not illustrated in, controls can be provided additionally or alternatively to allow the monitoring agent 112 to listen to the streaming audio from the corresponding camera(s) 102 and speak into a microphone so that the speaker(s) of one or more of such cameras 102 output audio representing the voice of the monitoring agent. In the illustrated example, the screen 602 also includes a larger, main viewer window 606 in which the streaming video of one of the video feed windows 604 can be optionally played, thereby making it easier for the monitoring agent 112 to view the content of the video. In some embodiments, the monitoring agent 112 can cause the streaming video from a particular camera 102 to be played in the main viewer window 606 by selecting the video feed window 604 of that camera (e.g., by clicking on it).

[0080] As Figure 6 illustrated, in some embodiments, the monitoring application 126 can cause the screen 602 of the monitoring device 110 to present one or more event data windows 608 that contain additional information about the event. Such additional information can include, for example, image frames identified by the edge image processing component 120 and / or the remote image processing component 122 as including features of interest (e.g., motion, people, faces, etc.), indications of the outputs from other sensors at the monitoring location 104 (e.g., door sensors, glass break sensors, motion sensors, smoke detectors, etc.), images of individuals authorized to be at the monitoring location 104, etc. Further, also as illustrated, in some embodiments, the monitoring application 126 can cause the screen 602 of the monitoring device 110 to present a scroll bar 610 that the monitoring agent 112 can manipulate, for example, to access and review additional event data windows 608 that cannot fit within the display area of the monitoring device 110.

[0081] The monitoring agent 112 can take appropriate actions based on a review of live video and / or audio from the camera(s) 102. If the monitoring agent 112 determines that there is no security issue, the monitoring agent 112 can cancel the event notification (e.g., by clicking or otherwise selecting a "Cancel" button - not illustrated), thereby removing it from the agent's queue. In response to selecting such a cancel button, the monitoring application 126 can cause the screen 602 to present a dropdown menu that is the same as or similar to the above-described dropdown menu 502, thereby allowing the monitoring agent 112 to select a reason for canceling the notification. Again, in response to the monitoring agent 112's selection of such a reason, the monitoring application 126 can enter an indication of the reason as the cancellation reason 218 of the notification into the event table 202, and can additionally enter an indication that the event has been resolved by the monitoring agent 112 as the agent action 216 into the event table 202. As discussed in more detail below, in some embodiments, for example, the mentioned cancellation reason 218 can be used to determine whether to send a notification (e.g., a push notification, an SMS message, an email, etc.) to the customer 116, whether to flag the event for review by the customer 116 (e.g., on the Figure 9B "Events to Review" screen 904 shown), and / or enable filtering of events based on one or more customer-specified criteria (e.g., via the Figure 9A "Event Type" user interface element 912c on the screen 902 shown). Further, as discussed in more detail in connection with Figure 9A 、 Figure 9B and Figure 10B , in some embodiments, the mentioned agent action 216 can be used to cause the customer application 128 to provide real-time or near-real-time information 914 about the actions taken by the monitoring agent 112 and / or provide historical information 1016 about the actions taken by the monitoring agent 112 when reviewing events.

[0082] On the other hand, if the monitoring agent 112 continues to believe that there may be a threat or other security issue based on a review of live video and / or audio from the camera(s) 102, the monitoring agent 112 can instead determine to continue evaluating the event, such as by communicating verbally with one or more individuals at the monitoring location 104 (e.g., via the speaker on the camera 102). In some embodiments, the monitoring application 126 can present a user interface element (e.g., a "Continue" button - not illustrated), and the monitoring agent 112 can click or otherwise select the user interface element to indicate that the monitoring agent 112 continues to review the event. When the user interface element is selected, the monitoring application 126 can enter an indication that the monitoring agent 112 is "taking action" as the agent action 216 into the event table 202. As discussed in more detail in connection with Figure 9A 、 Figure 9B r>Figure 9B and Figure 10BAs discussed in more detail, in some embodiments, the mentioned agent action 216 can be used to cause the client application 128 to provide real-time or near real-time information 914 about the actions taken by the monitoring agent 112 and / or to provide historical information 1016 about the actions taken by the monitoring agent 112 when reviewing an event. In some embodiments, when the "Continue" user interface element is selected, the monitoring application 126 can present authentication information to the monitoring agent 112, which can be used to help determine whether the individual at the monitoring location 104 is authorized to be there. Such authentication information can include, for example, the customer's contact information, a security password set by the customer, etc.

[0083] After further review by the monitoring agent 112, interaction with one or more individuals at the monitoring location 104, etc., the monitoring agent 112 can determine the disposition of the event and may take one or more remedial actions, such as dispatching the police or fire department to the monitoring location 104. In some embodiments, the monitoring application 126 can present a user interface element (e.g., a "Processed" button - not illustrated) to allow the monitoring agent 112 to indicate when the monitoring agent 112 has determined the disposition of the event. In response to selecting this user interface element, the monitoring application 126 can input an indication that the event has been processed by the monitoring agent 112 into the event table 202 as the agent action 216. As discussed in more detail below in connection with Figure 9A , Figure 9B and Figure 10B As discussed in more detail, in some embodiments, the mentioned agent action 216 can be used to cause the client application 128 to provide real-time or near real-time information 914 about the actions taken by the monitoring agent 112 and / or to provide historical information 1016 about the actions taken by the monitoring agent 112 when reviewing an event.

[0084] Further, in some embodiments, the monitoring application 126 can prompt the monitoring agent 112 to send one or more follow-up communications (e.g., email, push notification, text message, etc.) to the customer 116 describing the event and its disposition. In some embodiments, the monitoring application 126 can additionally prompt the monitoring agent 112 to select one or more key frames including features identified by the edge image processing component 120 and / or the remote image processing component 122 (e.g., by using a toggle switch to select such items in the event data window 608), and can attach an indication of the selected frame(s) and feature(s) to the notification sent to the customer 116.

[0085] In some embodiments, the monitoring application 126 may also present user interface elements (e.g., a toggle switch) that allow the monitoring agent 112 to mark frames identified by the edge image processing component 120 and / or the remote image processing component 122 as having incorrect or inaccurate feature recognition, and the data so collected may subsequently be used to retrain the edge image processing component 120 and / or the remote image processing component 122.

[0086] Further, in some embodiments, the monitoring application 126 may prompt the monitoring agent 112 to identify the final disposition of an event (e.g., by selecting a disposition from a drop-down menu or list of possible dispositions). The monitoring application 126 may input an indication of the final disposition of the event identified by the monitoring agent 112 into the event table 202 as the event disposition 220. As discussed in more detail below, in some embodiments, for example, the mentioned event disposition 220 may be used to determine whether to send a notification to the customer 116 (e.g., a push notification, an SMS message, an email, etc.), whether to flag the event for the customer 116 to review (e.g., on the Figure 9B "Events for Review" screen 904 shown), such that events can be filtered based on one or more customer-specified criteria (e.g., via the Figure 9A "Event Type" user interface element 912c on the screen 902 shown), and / or to populate various fields that may be included on the Figure 9A 、 Figure 9B 、 Figure 10A and Figure 10B screens 902, 904, 1002, 1004 shown.

[0087] In some embodiments, the monitoring application 126 may initially prompt the monitoring agent 112 to indicate the final disposition of an event and may only prompt the monitoring agent 112 to provide such (one or more) communications if the preferences set by the customer 116 indicate that one or more follow-up communications are to be sent in that case. In other embodiments, in response to certain data (e.g., the feature prediction 214, the agent action 216, the cancellation reason 218, the event disposition 220, etc.) being added to the event table 202, the monitoring application 126 may alternatively or additionally automatically send an event notification to the customer 116 according to the preferences set by the customer 116. Figure 7 An example customer profile table 700 is shown in which this type of customer preference data may be stored.

[0088] As Figure 7 shown, the customer profile table 700 may include profile data for each customer 116 indexed, for example, by a customer ID 702. As illustrated, the customer profile table 700 may include notification preferences 704 for each customer. The notification preferences 704 may be set by the respective customers.

[0089] Figures 8A to 8C Illustrates example screens 800, 810, and 820 that the customer application 128 can present to the customer 116 to enable selection of notification preferences. For example, by selecting the "Settings" user interface element 802 from the main menu 804 of the customer application 128, screens 800, 810, and 820 and other customer preference settings described herein can be accessed. As Figure 8A shown, the customer application 128 can present user interface elements that allow the customer 116 to select one or more specific types of events for which to receive notifications (see Figure 8B and Figure 8C ), and one or more specific ways in which to send such notifications (see Figures 8A to 8C ). In the illustrated example, the customer application 128 allows the customer 116 to select to receive (1) notifications about specific types of people, including (A) people who match a profile associated with the customer 116 (e.g., as further described below), (B) service people such as delivery people or yard service people, and / or (C) unidentified people (e.g., people visible on the property who cannot be identified as service or delivery people), and / or (2) notifications about general events (such as animals or pets on the property, natural events such as trees moving in the wind, etc.). Although not shown in br>Figure 8C , it should be understood that in the case where the customer 116 has multiple properties, different notification preferences can be set for each property. In some embodiments, notifications for events that have been classified as "urgent" or "immediate" can always be sent to the customer. However, in other embodiments, the customer can be allowed to opt out of receiving even these types of notifications.

[0090] Based on the notification preferences 704 of the customer 116, the monitoring application 126 can cause notifications about events to be sent to the customer in a specific manner in a specific environment. For example, if the notification preferences 704 indicate that the customer wants to receive "push" notifications about "general events" (e.g., as Figure 23 illustrated), then the monitoring application 126 can automatically send a push notification to the customer in response to the input of an event cancellation reason 218 corresponding to an event that corresponds to a general event (e.g., "weather event" or "animal or other pet"). As another example, if the notification preferences 704 indicate that the customer wants to receive "SMS" notifications about "unidentified people", then after the monitoring agent 112 has processed the event and / or in response to the input of a feature prediction 214 and / or event disposition 220 indicating an unidentified person on the property, the monitoring application 126 can send an SMS notification to the customer 116 as specified by the monitoring agent 112.

[0091] The following is in conjunction with Figure 9ADescribe an example process 2300 that can be performed by the monitoring application 126 to send a notification about an event to the customer 116 based on the notification preferences 704 of the customer 116.

[0092] In some embodiments, the data in the event table 202 can be additionally or alternatively used to enable the customer application 128 to present information about all detected events (or a selected subset of detected events) associated with the customer 116 and / or specific events that have been marked for the customer 116 to review. Figure 9B and Figure 9A Respectively illustrate example screens 902 and 904 that can be presented by the customer application 128 to allow the presentation of such information. In the illustrated example, the screens 902, 904 allow the customer 116 to select a "To Be Reviewed Events" label 906 or an "All Events" label 908 to switch between the screens 902 and 904. For example, the screens 902 and 904 can be accessed by selecting a "Home" user interface element 922 from the main menu 804. As shown, in some embodiments, the screens 902 and 904 can present a user interface element 924 that can be selected to expose quick access options for performing operations such as "Dispatch Police", "Talk to Agent", or "Pause All Cameras". Further, also as Figure 9B and Figure 17 shown, the screens 902 and 904 can present a "Cameras" user interface element 926 that the customer 116 can select to obtain access to a live video stream from one or more cameras 102, for example, as described below in connection with Figure 9A description.

[0093] First, refer to Figure 9A, Screen 902 shows example information that the client application 128 can display based on data retrieved from the event table 202 when the "All Events" label 908 is selected. As described above, in some embodiments, the client application 128 can obtain such data from the event table 202, for example, via one or more APIs provided by the monitoring service 106 or another component within the monitoring center environment 1126. In the illustrated example, information for only two most recent events (labeled "Event A" and "Event B") is presented. For example, information about less recent events can be viewed by scrolling the screen 902 to view other events. As shown, the screen 902 can include user interface elements 910a, 910b, 910c, and 910d that allow the client 116 to filter the displayed events based on one or more properties for which event data has been obtained. In the example shown, three properties are identified (corresponding to user interface elements 910a, 910b, 910c), and the user has selected the "Family Residence" user interface element 910a, which indicates that event information will be presented only for a specific one of the client's properties (which the client has designated as the "Family Residence"). The user interface element 910d can display the label "All Properties" when selected, and in the illustrated example can be selected to indicate that event data for all three of the client's properties (corresponding to user interface elements 910a, 910b, and 910c) is to be presented. Similarly, as Figure 7 shown, the screen 902 can additionally or alternatively include other user interface elements 912a, 912b, and 912c to allow the client 116 to specify additional filters to be applied to events associated with the client 116. Specifically, the user interface element 912a can allow the client 116 to filter events by date or date range (e.g., based on the timestamp 206 in the event table 202), the user interface element 912b can allow the client 116 to filter events by the camera 102 that acquired the image data of the event (e.g., based on the camera ID 210 in the event table 202), and the user interface element 912c can allow the client 116 to filter events by event type (e.g., based on the cancellation reason 218 and / or event disposition 220 in the event table 202). Although not illustrated, it should be understood that selecting one of the respective user interface elements 912 can cause a drop-down menu, calendar, etc. to be presented to allow the client 116 to select appropriate options.

[0094] The identity of the monitoring location 104 to be reviewed by the monitoring agent 112, the cameras 102 to be made available for such review, and the time period for performing such review, among other information, can be stored as monitoring preferences 710 in the client profile table 700 ( Figure 24 as shown).

[0095] Below, in combination withFigure 9A Example process 2400 that can be performed by one or more components of security system 100 to allow customer 116 to specify one or more filters to be applied to events presented on screen 902.

[0096] In Figure 9A the illustrated example, "Event A" represents an event currently being reviewed by monitoring agent 112. In addition to several thumbnail images and corresponding features (e.g., detected motion, detected person, detected face, etc.), information 914 representing actions recently taken by monitoring agent 112 can be presented. Information 914 can be derived, for example, based on recent agent action 216 mentioned in event table 202. As shown, the screen can additionally present information 916 describing the event, such as the name of camera 102 and / or the time the event was detected. Information 916 can be derived, for example, based on camera ID 210 and / or timestamp 206 of the event in event table 202. Information 918 can represent the disposition of the event, if any. Information 918 can be derived, for example, based on event disposition 220 of the event in event table 202. Since "Event A" is still under review, the disposition indicated by information 918 can be "under review" or some similar indication. For Figure 9B "Event B" in, similar information is depicted, except for the lack of an indication of the current agent action since the event is no longer being reviewed by monitoring agent 112. For example, the disposition indication 920 for "Event B" can be derived based on cancellation reason 218 in event table 202 (if the event was cancelled) or event disposition 220 in event table 202 (if the event was processed by monitoring agent 112).

[0097] Next, referring to Figure 7 , screen 904 shows example information that customer application 128 can display based on data retrieved from event table 202 when "Events to Review" label 906 is selected. In the illustrated example, user interface element 910d has been selected to indicate that marked events from all three properties of the customer will be presented for review. Further, in the example shown, the text associated with label 906 indicates that a total of six such events have been marked for customer 116 to review. It can be seen that the information presented for the marked events can be the same or similar to the information presented for the events under "All Events" label 908.

[0098] Similar to the manner in which event notifications can be sent to a customer based on customer-specific preferences as described above, events can be marked for a customer 116 to review based on preferences set by the customer or preferences set for the customer. Events can be marked for review, for example, by writing data to a customer review status 222 of an event table 202 that indicates that such an event is to be reviewed by the customer 116. As Figure 9B shown, for example, a customer profile table 700 can include an event review preference 706 that can be used for this purpose. In some embodiments, all events that are sent to the customer 116 as notifications can also be marked for the customer 116 to review. In such a case, the notification preference 704 can also serve as the event review preference 706. In other embodiments, the notification preference 704 and the event review preference 706 can indicate that the cases in which notifications are sent to the customer 116 are different from the cases in which events are marked for the customer 116 to review. In such an embodiment, similar to the notification preference 704, the customer application 128 can present a user interface through which the customer 116 can specify the specific cases in which events are to be marked for the customer 116 to review such that the marked events appear below a "To Be Reviewed Events" tab 906, as Figure 25 shown.

[0099] The following describes in conjunction with Figure 9B an example process 2500 that can be performed by a monitoring application 126 to mark events for a customer 116 to review based on the event review preference 706 of the customer 116.

[0100] In some embodiments, after the monitoring agent 112 has determined the disposition of an event, the event can be marked for the customer 116 to review (as described above). In such a case, the screen 904 will not display information 914 regarding the most recent action performed by the monitoring agent 112, as Figure 9B depicted. In other embodiments, all detected events of the customer 116 can initially be marked for review such that they appear under the "To Be Reviewed Events" tab 906, but can be unmarked if and when they are subsequently cancelled and / or assigned a disposition by the monitoring agent 112 such that they no longer meet the user-specified criteria for events to be marked for review, thereby removing them from the screen 904. Such an embodiment will allow the customer to view actions in near real-time under the "To Be Reviewed Events" tab 906 through the monitoring agent 112 (e.g., via the information 914) when the monitoring agent 112 reviews these events. As used herein, the term "near real-time" is used to refer to actions / events that occur within a close proximity in time to each other (e.g., less than a few seconds). In such a case, the disposition indicated by the information 918 can be "Under Review" or some similar indication until the monitoring agent 112 clears the event or assigns a disposition to the event.

[0101] AsFigure 10A As shown, the client application 128 can cause the event to be viewed to be accompanied by user interface elements 926 and 928 that allow the customer 116 to view the event or dismiss the event. In response to selecting the "Dismiss Event" user interface element 928, the client application 128 can update the customer viewing status 222 to indicate that the event is no longer marked for viewing, thereby causing information about the event to be removed from the screen 904. On the other hand, in response to selecting the "View" user interface element 926, the client application 128 can obtain additional information about the event for display on the client device 114.

[0102] Figure 9B Illustrated is an example screen 1002 that the client application 128 can present on the client device 114 based on a selection of the Figure 10B shown "View" user interface element 926. Figure 10A Illustrated is an example screen 1004 that the client application 128 can present on the client device 114 based on a selection of the Figure 10B shown "Agent Response" label 1006 on the screen 1002. The client application 128 can again cause the client device 114 to present the screen 1002 in response to a selection of the Figure 9A shown "Event Timeline" label 1008 on the screen 1004. If the customer 116 selects an event represented on the Figure 10A shown "All Events" screen 902 (e.g., by touching or otherwise selecting the area of the screen 902 that depicts information about the event), the same or a similar screen can be presented as well.

[0103] As Figure 10B and Figure 3 illustrated, the screens 1002 and 1004 can include an event window 1012 that is the same as or similar to the event window 306 described above in connection with Figure 4 and associated controls 1014 that allow the customer 116 to view a recorded event video and selectively play, rewind, and fast forward the video. Further, the screens 1002 and 1004 can also include a time delay bar 1020, which can be the same as or similar to the time delay bar 310 described above in connection with Figure 10A to allow the customer 116 to quickly navigate to and view the video portion corresponding to the feature identified by the edge image processing component 120 and / or the remote image processing component 122. For example, the client application 128 can be configured such that a selection of one or more feature indicators 1034 can cause the playback of the recorded video to start in the event window 1012 at or shortly before the time when the corresponding feature was identified.

[0104] As Figure 10AAs shown, when the "Event Timeline" label 1008 is selected, the client application 128 can present a thumbnail image 1022 of a frame of image data in which the edge image processing component 120 and / or the remote image processing component 122 has identified one or more features (e.g., motion, person, face, etc.), as well as a feature descriptor 1036 of such identified features and / or a time indication 1038 identifying the time at which these frames were captured. Thereby, rich details about the results of the image processing performed and the frames marked by the analysis can be provided to the client 116. In some embodiments, the client application 128 can alternatively or additionally be configured such that selection of the thumbnail image 1022, the feature descriptor 1036, and / or the time indication 1038 can cause playback of the recorded video in the event window 1012 to start at or shortly before the time at which the corresponding feature was identified.

[0105] As Figure 7 shown, in some embodiments, the client application 128 can output a prompt 1010 indicating that the client 116 can provide a specific input, such as a long press gesture (sometimes referred to as a "tap and hold" gesture), on the thumbnail image to create a new profile of the person represented in the thumbnail image and / or provide feedback regarding the image analysis performed on the image. In some embodiments, the security system 100 can store profiles for individual persons, where such profiles can include, for example, one or more images of the individual and data indicating whether the individual is authorized to be present at one or more specific monitoring locations 104. The client 116 can create one or more such profiles, for example, by selecting the "Profile" user interface element 806 of the main menu 804. Data representing these profiles can be stored, for example, as a guest profile 708 in the client profile table 700 ( Figure 10A shown). In embodiments that employ such profiles, the edge image processing component 120 and / or the remote image processing component 122 can use the stored profiles to determine whether a person detected in the captured image data is an authorized visitor or potentially an intruder.

[0106] After viewing the view screen 1002 ( Figure 10C shown), the client 116 can determine that an individual marked as an unknown person is actually an authorized visitor, for example. Based on such a determination, the client 116 can perform a "tap and hold" gesture on the corresponding thumbnail image to create a profile of the individual, thereby preventing the security system 100 from making the same misidentification in the future. Figure 10D An example of such a gesture is illustrated (i.e., where the user provided a tap and hold gesture on the thumbnail image 1022). Figure 9AFIG. 1024 shows an example screen that the client application 128 can present on the client device 114 in response to detecting such a gesture on a thumbnail image (e.g., thumbnail image 1022). It should be understood that in some embodiments, the client application 128 can similarly present the same or a similar screen in response to detecting such a gesture on other thumbnail images presented therein (e.g., thumbnail images presented on the screen 902 and / or Figure 9B the screen 904 shown). Figure 10D As shown in

[0107] FIG. Figure 10C 10, in some embodiments, the client application 128 can cause the client device 114 to display a face 1026 present in the selected thumbnail image (e.g., Figure 10D the thumbnail image 1022 shown), and can also provide one or more user interface elements 1028 that the client 116 can select to identify an existing profile to which the face 1026 is to be added (e.g., by selecting the “Add to Profile” user interface element 1030), and / or provide a user interface element 1032 that the client 116 can select to create a new profile (e.g., a profile of a new individual) to which the face 1026 is to be added.

[0108] Although not illustrated in Figure 10D FIG. 10, it should be understood that in some embodiments, the client application 128 can additionally or alternatively cause the client device 114 to present one or more user interface elements that the client 116 can select to enable the client 116 to provide feedback regarding the accuracy of performing image processing. For example, in the case where the edge image processing component 120 and / or the remote image processing component 122 identifies a person or a face in a thumbnail image (e.g., thumbnail image 1022), the client application 128 can cause the client device 114 to present a “No Person” user interface element or a “No Face” user interface element on the screen 1024 or a similar screen, which allows the client 116 to indicate that the selected thumbnail image (e.g., thumbnail image 1022) actually does not include a person or actually does not include a face. This feedback for identifying incorrect / inaccurate feature detection can be recorded in the event table 202 as client feedback 224. In some embodiments, this feedback can subsequently be used to retrain the edge image processing component 120 and / or the remote image processing component 122.

[0109] An example process 2600 that can be performed by the client application 128 to enable the client 116 to provide feedback for changing one or more functions of the edge image processing component 120 and / or the remote image processing component 122 is described below in conjunction with r>Figure 26 FIG.

[0110] FIG.Figure 10B As shown, when the "Agent Response" label 1006 is selected, the client application 128 can present information 1016 listing the actions taken by the monitoring agent 112 when evaluating events, and may also present other events related to the review of the monitoring agent, such as the time when one or more image processing modules detect a person, the time when an event notification is loaded into the review queue of the monitoring agent, etc. The presented information 1016 related to the actions taken by the monitoring agent can be derived, for example, from the data representing the agent action(s) 216 in the event table 202.

[0111] Similarly, as Figure 10B illustrated, in some embodiments, the client application 128 can output a prompt 1018 indicating that the client 116 can provide feedback on the manner in which the monitoring agent 112 processes events. In some embodiments, such feedback can be stored as client feedback 224 in the event table 202 and can be used, for example, for quality control purposes and / or for agent training / retraining.

[0112] Figure 11 is a schematic diagram of an example security system 1100 that can be employed with various aspects of the present disclosure. As shown, in some embodiments, the security system 1100 can include multiple monitoring locations 104 (only one of which is illustrated in Figure 11 ), a monitoring center environment 1122, a surveillance center environment 1126, one or more client devices 114, and one or more communication networks 1120. The monitoring locations 104, the monitoring center environment 1122, the surveillance center environment 1126, the one or more client devices 114, and the communication network(s) 1120 can each include one or more computing devices (e.g., as described below with reference to Figure 19 ). The client device(s) 114 can include one or more client applications 128, such as applications hosted on or otherwise accessible by the client device(s) 114. In some embodiments, the client application 128 can be embodied as a web application that can be accessed via a browser of the client device(s) 114. The monitoring center environment 1122 can include one or more monitoring applications 126, such as applications hosted on or otherwise accessible by computing devices within the monitoring center environment 1122. In some embodiments, the monitoring application 126 can be embodied as a web application that can be accessed via a browser of a computing device operated by the monitoring agent 112 within the monitoring center environment 1122. The surveillance center environment 1126 can include a surveillance service 1130 and one or more transmission services 1128.

[0113] As Figure 11As shown, the monitoring location 104 may include one or more image capture devices (e.g., cameras 102A and 102B), one or more contact sensor assemblies (e.g., contact sensor assembly 1106), one or more keypads (e.g., keypad 1108), one or more motion sensor assemblies (e.g., motion sensor assembly 1110), a base station 1112, and a router 1114. As illustrated, the base station 1112 may host a surveillance client 1116.

[0114] In some embodiments, the router 1114 may be a wireless router configured to communicate with devices (e.g., devices 102A, 102B, 1106, 1108, 1110, and 1112) disposed at the monitoring location 104 via communication consistent with a communication standard such as any of the various Institute of Electrical and Electronics Engineers (IEEE) 108.11 standards. As Figure 11 illustrated, the router 1114 may also be configured to communicate with the network(s) 1120. In some embodiments, the router 1114 may implement a local area network (LAN) within or near the monitoring location 104. In other embodiments, other types of networking technologies may be additionally or alternatively used within the monitoring location 104. For example, in some embodiments, the base station 1112 may receive and forward communication packets transmitted by one or both of the cameras 102A, 102B via a point-to-point personal area network (PAN) protocol such as Bluetooth. Other suitable wired, wireless, and mesh network technologies and topologies will be apparent in light of the present disclosure and are intended to fall within the scope of the examples disclosed herein.

[0115] One or more networks 1120 may include one or more public and / or private networks that support, for example, Internet Protocol (IP) communications. One or more networks 1120 may include, for example, one or more local area networks (LANs), one or more personal area networks (PANs), and / or one or more wide area networks (WANs). LANs that may be employed include wired or wireless networks that support various LAN standards, such as IEEE 802.11 versions, etc. PANs that may be employed include wired or wireless networks that support various PAN standards, such as Bluetooth, ZigBee, etc. WANs that may be employed include wired or wireless networks that support various WAN standards, such as Code Division Multiple Access (CDMA), Global System for Mobile Communications (GSM), etc. Regardless of the specific networking technology employed, one or more networks 1120 may connect components within monitoring location 104, monitoring center environment 1122, surveillance center environment 1126, and one or more client devices 114 and enable data communication therebetween. In at least some embodiments, both monitoring center environment 1122 and surveillance center environment 1126 may include networking components (e.g., similar to router 1114) configured to communicate with one or more networks 1120 and various computing devices within these environments.

[0116] Surveillance center environment 1126 may include physical space, communication, cooling, and power infrastructure to support the networking operations of a large number of computing devices. For example, the infrastructure of surveillance center environment 1126 may include rack space into which computing devices may be installed, uninterruptible power supplies, cooling plenums and equipment, and networking equipment. Surveillance center environment 1126 may be dedicated to security system 1100, may be a non-dedicated, commercially available cloud computing service (e.g., Microsoft Azure, Amazon Web Services, Google Cloud, etc.), or may include a hybrid configuration consisting of dedicated and non-dedicated resources. Regardless of its physical or logical configuration, as Figure 11 shown, surveillance center environment 1126 may be configured to host surveillance services 1130 and one or more transport services 1128.

[0117] Monitoring center environment 1122 may include a plurality of computing devices (e.g., desktop computers) and network equipment (e.g., one or more routers) that enable communication between the computing devices and one or more networks 1120. One or more client devices 114 may each include a personal computing device (e.g., desktop computer, laptop computer, tablet computer, smart phone, etc.) and network equipment (e.g., router, cellular modem, cellular radio transceiver, etc.). As Figure 11As an example, the monitoring center environment 1122 can be configured to host one or more monitoring applications 126, and one or more customer devices 114 can be configured to host one or more customer applications 128.

[0118] Devices 102A, 102B, 1106, and 1110 can be configured to acquire analog signals via sensors incorporated into the devices, generate digital sensor data based on the acquired signals, and transmit the sensor data to one or more components within the base station 122 and / or the monitoring center environment 1126 (e.g., the aforementioned remote image processing component 122) (e.g., via a wireless link with the router 1114). The type of sensor data generated and transmitted by these devices can vary depending on the characteristics of the sensors they include. For example, the image capture devices or cameras 102A and 102B can acquire ambient light, generate one or more frames of image data based on the acquired light, and transmit the frame(s) to one or more components within the base station 1112 and / or the monitoring center environment 1126, although the pixel resolution and frame rate can vary depending on the capabilities of the devices. In some embodiments, the cameras 102A and 102B can also receive and store filter zone configuration data and use one or more filter zones (e.g., regions within the FOV of the camera from which image data is redacted for various reasons such as to exclude trees that may generate false positive motion detection results on windy days) to filter the frame(s) before transmitting the frame(s) to one or more components within the base station 1112 and / or the monitoring center environment 1126. In Figure 11 In the example shown, the camera 102A has a field of view (FOV) that starts near the front door of the monitoring location 104 and can acquire images of the sidewalk 1136, the road 1138, and the space between the monitoring location 104 and the road 64A0. On the other hand, the camera 102B has an FOV that starts near the bathroom of the monitoring location 104 and can acquire images of the living room and dining area of the monitoring location 104. The camera 102B can also acquire images of the outdoor area outside the monitoring location 104, for example, through the windows 1118A and 1118B on the right hand side of the monitoring location 104.

[0119] Each sensor component (e.g., Figure 11 the contact sensor component 1106 shown) deployed at the monitoring location 104 can include, for example, a sensor that can detect the presence of a magnetic field generated by a magnet when the magnet approaches the sensor. When a magnetic field is present, the contact sensor component 1106 can generate boolean sensor data specifying the closed state of a window, door, etc. When no magnetic field is present, the contact sensor component 1106 can instead generate boolean sensor data specifying the open state of a window, door, etc. In either case, Figure 11The contact sensor assembly 1106 shown can transmit sensor data indicating whether the front door at the monitored location 104 is open or closed to the base station 1112.

[0120] Each motion sensor assembly deployed at the monitored location 104 (e.g., Figure 11 the motion sensor assembly 1110 shown) can include, for example, components that can emit high-frequency pressure waves (e.g., ultrasonic waves) and sensors that can acquire reflections of the emitted waves. When the sensor detects a change in the reflected pressure wave, for example, because one or more objects are moving within the space monitored by the sensor, the motion sensor assembly 1110 can generate Boolean sensor data specifying an alarm state. When the sensor does not detect a change in the reflected pressure wave, for example, because no objects are moving within the monitored space, the motion sensor assembly 1110 can instead generate Boolean sensor data specifying a stationary state. In either case, the motion sensor assembly 1110 can transmit the sensor data to the base station 1112. It should be noted that the above specific sensing modalities are not limited to the present disclosure. For example, merely as an example of an alternative implementation, the motion sensor assembly 1110 can instead (or additionally) operate based on the detection of changes in reflected electromagnetic waves.

[0121] Although the above describes a particular type of sensor, it should be understood that other types of sensors can be additionally or alternatively employed within the monitored location 104 to detect the presence and / or movement of a person, or other conditions of interest, such as smoke, elevated carbon dioxide levels, water accumulation, etc., and transmit data indicating these conditions to the base station 1112. For example, although not illustrated in Figure 11 some embodiments, one or more sensors can be employed to detect sudden changes in measured temperature, sudden changes in incident infrared radiation, sudden changes in incident pressure waves (e.g., sound waves), etc. Still further, in some embodiments, some such sensors and / or the base station 1112 can be additionally or alternatively configured to identify specific signal curves indicating specific conditions, such as sound curves indicating glass breakage, footsteps, coughing, etc.

[0122] Figure 11The keypad 1108 shown can be configured to interact with a user and interoperate with other devices disposed in the monitoring location 104 in response to such interaction. For example, in some examples, the keypad 1108 can be configured to receive input from a user specifying one or more commands and transmit the specified commands to one or more addressed devices and / or processes, such as one or more devices disposed in the monitoring location 104, the monitoring application(s) 126, and / or the monitoring service 1130. The transmitted commands can include, for example, codes for authenticating the user as a resident of the monitoring location 104 and / or codes for requesting activation or deactivation of one or more devices disposed in the monitoring location 104. In some embodiments, the keypad 1108 can include a user interface (e.g., a tactile interface, such as a set of physical buttons or a set of "soft" buttons on a touch screen) configured to interact with the user (e.g., receive input from the user and / or present output to the user). Further, in some embodiments, the keypad 1108 can receive a response to the transmitted commands and present such response as a visual or audio output via the user interface.

[0123] Figure 19 The base station 1112 shown can be configured to interoperate with other security system devices disposed at the monitoring location 104 to provide local command and control and / or store-and-forward functionality via the execution of the monitoring client 1116. To implement the local command and control functionality, the base station 1112 can perform various programmed operations via the execution of the monitoring client 1116 in response to various events. Examples of such events include receiving a command from the keypad 1108, receiving a command from one of the monitoring application(s) 126 or the client application 128 via the network(s) 1120, and detecting the occurrence of a scheduled event. The programmed operations performed by the base station 1112 in response to an event via the execution of the monitoring client 1116 can include, for example, activation or deactivation of one or more of the devices 102A, 102B, 1106, 1108, and 1110; sounding an alarm; reporting the event to the monitoring service 1130; and / or transmitting "location data" to one or more of the transmission services 1128. Such location data can include, for example, data specifying sensor readings (sensor data), image data acquired by one or more cameras 102, configuration data of one or more devices disposed at the monitoring location 104, commands input and received from a user (e.g., via the keypad 1108 or the client application 128), or data derived from one or more of the foregoing data types (e.g., filtered sensor data, filtered image data, a summary of sensor data, event data specifying an event detected at the monitoring location 104 via sensor data, etc.).

[0124] In some embodiments, to implement the store-and-forward function, base station 1112 can receive sensor data by monitoring the execution of client 1116, encapsulate the data for transmission, and store the encapsulated sensor data in local memory for subsequent transfer. Such communication of the encapsulated sensor data can include, for example, transmitting the encapsulated sensor data as the payload of a message to one or more of the (one or more) transport services 1128 when the communication link to the (one or more) transport services 1128 via the (one or more) networks 1120 is operable. In some embodiments, such encapsulation of the sensor data can include filtering the sensor data using one or more filter zones and / or generating one or more summaries (maximum value, average value, change in value since the previous transmitted value, etc.) of the multiple sensor readings.

[0125] The (one or more) transport services 1128 that monitor the central environment 1126 can be configured to receive messages from monitoring locations (e.g., monitoring location 104), parse the messages to extract the payloads included therein, and store the payloads and / or data derived from the payloads in one or more data repositories hosted in the central monitoring environment 1126. Examples of such data repositories are described below in conjunction with Figure 11 In some embodiments, the (one or more) transport services 1128 can expose and implement one or more application programming interfaces (APIs) that are configured to receive, process, and respond to calls from base stations (e.g., base station 1112) via the (one or more) networks 1120. The various instances of the (one or more) transport services 1128 can be associated with certain manufacturers and / or models of location-based monitoring devices (e.g., SIMPLISAFE devices, RING devices, etc.) and be specific to these manufacturers and / or models.

[0126] One or more Application Programming Interfaces (APIs) of the transport service(s) 1128 may be implemented using various architectural styles and interoperability standards. For example, in some embodiments, one or more such APIs may include a web service interface implemented using the Representational State Transfer (REST) architectural style. In such an embodiment, the Hypertext Transfer Protocol (HTTP) may be used in conjunction with JavaScript Object Notation (JSON) and / or Extensible Markup Language to encode API calls. Such API calls may be addressed to one or more Uniform Resource Locators (URLs) corresponding to API endpoints monitored by the transport service(s) 1128. In some embodiments, portions of the HTTP communication may be encrypted to increase security. Alternatively (or additionally), in some embodiments, one or more APIs of the transport service(s) 1128 may be implemented as.NET web APIs in response to an HTTP data post to a specific URL. Alternatively (or additionally), in some embodiments, one or more APIs of the transport service(s) 1128 may be implemented using Simple File Transfer Protocol commands. Thus, the API(s) of the transport service(s) 1128 are not limited to any particular implementation.

[0127] The monitoring service 1130 in the monitoring center environment 1126 may be configured to control the overall logical settings and operations of the security system 1100. As such, the monitoring service 1130 may communicate and interoperate via the network(s) 1120 with the transport service(s) 1128, the monitoring application(s) 126, the client application(s) 128, and various devices disposed at the monitoring location 104. In some embodiments, the monitoring service 1130 may be configured to monitor data from various sources for events (e.g., a break-in event), and when an event is detected, notify one or more of the monitoring application 126 and / or the client application(s) 128 of the event.

[0128] In some embodiments, the monitoring service 1130 may additionally be configured to maintain status information regarding the monitored location 104. Such status information may indicate, for example, whether the monitored location 104 is secure or threatened. In some embodiments, the monitoring service 1130 may be configured to change the status information to indicate that the monitored location 104 is secure only upon receiving a communication indicating a clearance event (e.g., rather than making such a change merely due to the absence of detected additional events). This feature can prevent "smash and grab" thefts (e.g., where an intruder quickly vandalizes or deactivates the monitoring equipment) from being successfully carried out. Additionally, in some embodiments, the monitoring service 1130 may be configured to monitor one or more specific areas within the monitored location 104, such as one or more specific rooms or other distinct areas within and / or around the monitored location 104 and / or one or more defined areas within the FOV of the respective image capture devices (e.g., Figure 3 the cameras 102A and 102B shown) deployed at the monitored location.

[0129] The (one or more) individual monitoring applications 126 of the monitoring center environment 1122 may be configured to enable a monitoring person to interact with the respective computing device to provide monitoring services for the corresponding location (e.g., the monitored location 104) and perform various programmed operations in response to such interaction. For example, in some embodiments, the monitoring application 126 may control its host computing device to provide information about events detected at a monitored location such as the monitored location 104 to the person operating the computing device. Such events may include, for example, detected movement within a specific area of the monitored location 104. As described above in connection with Figure 6 and Figure 6 , in some embodiments, the monitoring application 126 may cause the monitoring device 110 to present a video of an event within the respective event windows 306 of the screen 302, and may also establish a streaming connection with one or more cameras 102 at the monitored location and cause the monitoring device 110 to provide streaming video from such (one or more) cameras 102 within the video feed window 604 and / or the main viewer window 606 of the screen 602, as well as permit audio communication between the monitoring device 110 and the (one or more) cameras 102. (One or more) client applications 128 of (one or more) client devices 114 may be configured to enable a client to interact with their computing device (e.g., their smart phone or personal computer) to access various services provided by security system 1100 for their individual home or other location (e.g., monitored location 104), and perform various programming operations in response to such interaction. For example, in some embodiments, client application 128 may control client device 114 (e.g., smart phone or personal computer) to provide information about events detected at a monitored location such as monitored location 104 to the client operating the client device 114. Such events may include, for example, detected movement within a specific area of monitored location 104. In some embodiments, client application 128 may additionally or alternatively be configured to process input received from the client to enable or disable one or more devices arranged within monitored location 104. Further, as described above in connection with Figure 12 it, client application 128 may additionally or alternatively be configured to establish a streaming connection with one or more cameras 102 at the monitored location, and cause client device 114 to display streaming video from such (one or more) cameras 102, and allow audio communication between client device 114 and (one or more) cameras 102.

[0130] Turning now to Figure 12 , an example base station 1112 is schematically illustrated. As Figure 11 shown, base station 1112 may include at least one processor 1202, volatile memory 1204, non-volatile memory 1208, at least one network interface 1206, user interface 1214, battery assembly 1216, and interconnect mechanism 1218. Non-volatile memory 1208 may store executable code 1210, and as illustrated, may also include data storage 1212. In some embodiments, the features of base station 1112 listed above may be incorporated within housing 1220, or otherwise supported by it. In some embodiments, user interface 1214 of base station 1112 may include only: one or more speakers that provide an audio output to the user regarding a change in the operating state of security system 1100, detected threats, etc.; and / or one or more visual indicators (e.g., light emitting diode (LED) indicators) that indicate when base station 1112 is operable in response to user input (e.g., via keypad 1108), etc. In other embodiments, the user interface may additionally or alternatively include more complex output components (e.g., display screen) and / or may include one or more user input components, such as one or more microphones (e.g., to receive voice commands) and / or keypad (e.g., to receive tactile input).

[0131] In some embodiments, the non-volatile (non-transitory) memory 1208 may include one or more read-only memory (ROM) chips; one or more hard disk drives or other magnetic or optical storage media; one or more solid-state drives (SSDs), such as flash drives or other solid-state storage media; and / or one or more hybrid magnetic and SSDs. In some embodiments, the code 1210 stored in the non-volatile memory may include an operating system and one or more applications or programs configured to execute under the control of the operating system. In some embodiments, the code 1210 may additionally or alternatively include proprietary firmware and embedded software that is executable without relying on a commercially available operating system. In any case, regardless of how the code 1210 is specifically implemented, the execution of the code 1210 may implement Figure 11 the monitoring client 1116 shown, and enable the storage and manipulation of data of the monitoring client 1116 within the data store 1212.

[0132] The processor 1202 of the base station 1112 may include one or more processors configured to execute instructions encoded in a computer-readable medium (such as a computer program embodied by the code 1210) to control the operation of the base station 1112. As used herein, the term "processor" describes circuitry that performs functions, operations, or sequences of operations. The functions, operations, or sequences of operations may be hard-coded into the circuitry or soft-coded by instructions held in a memory device (e.g., volatile memory 1204) and executed by the circuitry. In some embodiments, the processor 1202 may be embodied by one or more application-specific integrated circuits (ASICs), microprocessors, digital signal processors (DSPs), graphics processing units (GPUs), neural processing units (NPUs), microcontrollers, field-programmable gate arrays (FPGAs), programmable logic arrays (PLAs), and / or multi-core processors.

[0133] Before executing the code 1210, the processor 1202 may copy at least a portion of the code 1210 from the non-volatile memory 1208 to the volatile memory 1204. In some embodiments, the volatile memory 1204 may include one or more static or dynamic random access memory (RAM) chips and / or cache memory (e.g., memory disposed on the silicon die of the processor 1202). The volatile memory 1204 may provide a faster response time than main memory (such as the non-volatile memory 1208).

[0134] By executing code 1210, the processor 1202 can control the operation of the network interface 1206. For example, in some embodiments, the network interface 1206 may include one or more physical interfaces (e.g., radio transceivers, Ethernet ports, universal serial bus (USB) ports, etc.) and a software stack that includes drivers and / or other code 1210 configured to communicate with the one or more physical interfaces to support one or more LAN, PAN, and / or WAN standard communication protocols. Such communication protocols may include, for example, the Transmission Control Protocol (TCP) and the User Datagram Protocol (UDP). Thus, the network interface 1206 can enable the base station 1112 to access and communicate with other computing devices (e.g., other devices arranged at the Figure 11 LAN established by the router 1114 of Figure 11 the (one or more) networks 1120 and / or point-to-point connections). For example, in some embodiments, the network interface 1206 may utilize sub-GHz wireless networking to send wake-up messages to other computing devices to request sensor data streams. Figure 13 monitoring location 104 of

[0135] By executing code 1210, the processor 1202 can additionally control the operation of the hardware and software stack that includes drivers and / or other code 1210 configured to communicate with other system devices. Thus, the base station 1112 can interact with other system components in response to received inputs. Such inputs may specify, for example, values to be stored in the data storage 1212. The base station 1112 can also provide outputs representing the values stored in the data storage 1212. In some embodiments, the base station 1112 may additionally include one or more light-emitting diodes (LEDs) or other visual indicators that visually convey information such as system status or alert events. Further, in some embodiments, the base station 1112 may additionally or alternatively include a siren (e.g., a 95 decibel (dB) siren) or other audio output device that can be controlled by the processor 1202 to output an audible indication of a detected intrusion event.

[0136] The various components of the base station 1112 described above may communicate with each other via an interconnection mechanism 1218. In some embodiments, the interconnection mechanism 1218 may include a communication bus. Further, in some embodiments, the battery assembly 1216 may be configured to supply operating power to the various features of the base station 1112. In some embodiments, the battery assembly 1216 may include at least one rechargeable battery (e.g., one or more nickel metal hydride (NiMH) or lithium batteries). In some embodiments, such a rechargeable battery (or rechargeable batteries) may have a runtime capacity sufficient to operate the base station 1112 for twenty-four hours or more when the base station 1112 is disconnected from line power or otherwise not receiving line power. In some embodiments, the battery assembly 1216 may additionally or alternatively include a power circuit that receives, regulates, and distributes line power to operate the base station 1112 and / or recharge one or more rechargeable batteries. Such a power circuit may include, for example, a transformer and a rectifier, as well as other circuitry that converts AC line power into DC device and / or recharge power.

[0137] Turning now to Figure 13 , an example keypad 1108 is schematically illustrated. As Figure 11 shown, the keypad 1108 may include at least one processor 1302, volatile memory 1304, non-volatile memory 1308, at least one network interface 1306, a user interface 1314, a battery assembly 1316, and an interconnection mechanism 1318. The non-volatile memory 1308 may store executable code 1310 and, as illustrated, may also include a data store 1312. In some embodiments, the features of the keypad 1108 listed above may be incorporated within or otherwise supported by a housing 1320.

[0138] In some embodiments, the corresponding descriptions of the processor 1202, volatile memory 1204, non-volatile memory 1208, interconnection mechanism 1218, and battery assembly 1216 with reference to the base station 1112 apply to the corresponding descriptions of the processor 1302, volatile memory 1304, non-volatile memory 1308, interconnection mechanism 1318, and battery assembly 1316 with reference to the keypad 1108. Accordingly, these descriptions will not be repeated here.

[0139] By executing code 1310, the processor 1302 of the keypad 1108 can control the operation of the network interface 1306. In some embodiments, the network interface 1306 may include one or more physical interfaces (e.g., radio transceiver devices, Ethernet ports, USB ports, etc.) and a software stack that includes drivers and / or other code 1310 that is configured to communicate with the one or more physical interfaces to support one or more LAN, PAN, and / or WAN standard communication protocols. Such communication protocols may include, for example, TCP and UDP. Thus, the network interface 1306 can enable the keypad 1108 to access and communicate with other computing devices (e.g., other devices arranged at Figure 14 the monitoring location 104) via a computer network (e.g., a LAN established by the router 1114).

[0140] By executing code 1310, the processor 1302 can additionally control the operation of the user interface 1314. In some embodiments, the user interface 1314 may include user input and / or output devices (e.g., physical keys arranged as a keypad, touchscreens, displays, speakers, cameras, biometric scanners, environmental sensors, etc.) and a software stack that includes drivers and / or other code 1310 that is configured to communicate with the user input and / or output devices. Thus, the user interface 1314 can enable the keypad 1108 to interact with a user to receive input and / or present output. Examples of output that may be presented by the user interface 1314 include one or more GUIs that include one or more controls configured to display output and / or receive input. Input received by the user interface 1314 may specify, for example, values to be stored in the data storage 1312. Output provided by the user interface 1314 may also indicate values stored in the data storage 1312. In some embodiments, portions of the user interface 1314 (e.g., one or more LEDs) may be accessible and / or visible as part of or through the housing 1320.

[0141] Now turning to Figure 11 , an example sensor component 1424 is schematically illustrated. Several example embodiments of the sensor component 1424 (e.g., cameras 102 and 102B, the motion sensor component 1110, and the contact sensor component 1106) are illustrated and described above in Figure 14 . As Figure 11As shown, the sensor assembly 1424 may include at least one processor 1402, volatile memory 1404, non-volatile memory 1408, at least one network interface 1406, a battery assembly 1416, an interconnect mechanism 1418, and at least one sensor 1422. The non-volatile memory 1408 may store executable code 1410 and, as illustrated, may also include a data store 1412. In some embodiments, the features of the sensor assembly 1424 listed above may be incorporated within or included as part of a housing 1420. Further, in some embodiments, the sensor assembly 1424 may additionally include a user interface 1414.

[0142] In some embodiments, the corresponding descriptions of the processor 1202, volatile memory 1204, non-volatile memory 1208, interconnect mechanism 1218, and battery assembly 1216 with reference to the reference base station 1112 apply to the corresponding descriptions of the processor 1402, volatile memory 1404, non-volatile memory 1408, interconnect mechanism 1418, and battery assembly 1416 with reference to the sensor assembly 1424. Thus, these descriptions will not be repeated here.

[0143] By executing the code 1410, the processor 1402 may control the operation of the network interface 1406 and the user interface 1414, if present. In some embodiments, the network interface 1406 may include one or more physical interfaces (e.g., radio transceivers, Ethernet ports, USB ports, etc.) as well as a software stack that includes drivers and / or other code 1410 that is configured to communicate with the one or more physical interfaces to support one or more LAN, PAN, and / or WAN standard communication protocols. Such communication protocols may include, for example, TCP and UDP. Thus, the network interface 1406 may enable the sensor assembly 1424 to access other computing devices (e.g., arranged Figure 15other devices in the monitoring location 104) and communicate with them. For example, in some embodiments, when executing code 1410, the processor 1402 may control the network interface to stream (e.g., via UDP) sensor data obtained from the sensor component 1422 to the base station 1112. Further, in some embodiments, by executing code 1410, the processor 1402 may additionally or alternatively control the network interface 1406 to enter a power saving mode, e.g., by powering down the 2.4 GHz radio transceiver device, which is included in the network interface 1406, and powering up the sub-GHz radio transceiver device. In such an embodiment, by executing code 1410, the processor 1402 may additionally control the network interface 1406 to enter a streaming mode, e.g., by powering up the 2.4 GHz radio transceiver device and powering down the sub-GHz radio transceiver device in response to, for example, receiving a wake-up signal from the base station via the sub-GHz radio transceiver device.

[0144] By executing code 1410, the processor 1402 may additionally or alternatively control other operations of the sensor component 1424. In some embodiments, for example, the user interface 1414 of the sensor component 1424 may include user input and / or output devices (e.g., physical buttons, touchscreens, displays, speakers, cameras, accelerometers, biometric scanners, environmental sensors, one or more LEDs, etc.) and a software stack that includes drivers and / or other code 1410 configured to communicate with the user input and / or output devices. Thus, the sensor component 1424 may enable the user interface 1414 to interact with a user to receive input and / or present output. The output presented by the user interface 1314 may include, for example, one or more GUIs that include one or more controls configured to display output and / or receive input. The input received by the user interface 1414 may, for example, specify a value to be stored in the data storage 1412. The output provided by the user interface 94 may also indicate the value stored in the data storage 1412. In some embodiments, portions of the sensor component 1424 may be accessible and / or visible as part of or through the housing 1420.

[0145] As Figure 11 shown, the sensor component 1424 may include one or more types of sensors 1422, such as those referenced above Figure 15One or more of the sensors described for cameras 102A and 102B, motion sensor assembly 1110, and contact sensor assembly 1106, or other types of sensors. In some embodiments, for example, (one or more) sensors 1422 may include a camera and a temperature sensor. Regardless of the (one or more) types of (one or more) sensors XD22 employed, processor 1402 can (e.g., via execution of code 1410) obtain sensor data from (one or more) sensors 1422 and stream the obtained sensor data to processor 1402 for transmission to base station 1112.

[0146] It should be noted that in some embodiments of devices 1302 and 1402, the operations performed by processors 1302 and 1402 under the respective control of code 1310 and 1410 can be hardcoded and / or implemented using hardware rather than as a combination of hardware and software.

[0147] Now turning to Figure 15 , schematically illustrates Figure 3 Aspects of the monitoring center environment 1126, monitoring center environment 1122, a client device 114, (one or more) networks 1120, and multiple monitoring locations 104A through 104N (collectively referred to as monitoring locations 104) shown. As Figure 6 shown, in some embodiments, the monitoring service 1130 may include a location data storage 1502, an image data storage 1504, an artificial intelligence (AI) service 1508, an event listening service 1510, an identity provider service 1512, a customer service 1538, a monitoring service 106, and a camera streaming service 1506. Also as Figure 15 shown, the monitoring center environment 1122 may include multiple monitoring devices 110A through 110M (collectively referred to as monitoring devices 110) that host or are otherwise configured to access corresponding monitoring applications 126A through 126M, and each of the monitoring locations 104A through 104N may include a corresponding monitoring client 1116A through 1116N (collectively referred to as monitoring clients 1116), e.g., at each of the monitoring locations 104A through 1102N within base station 1112 ( Figure 20 not shown). As described above in connection with Figure 11 and Figure 1A , in some embodiments, the monitoring application 126 may be configured such that the monitoring device 110 displays screens 302, 602 that enable the monitoring agent 112 to visually monitor the activities of one or more of the monitoring locations 104 and (e.g., via the microphone and speaker of the camera 102 at the monitoring location 104) engage in an audio conversation with one or more individuals at such locations. Further, as Figure 6As further shown, in some embodiments, (one or more) transmission services 1128 may include a plurality of different transmission services 1128A through 1128D, which are configured to receive location data packets, such as location data packets 1014A through 1014D, from monitoring clients 1116A through 1116N deployed at respective monitoring locations 104A through 1102N. The location data store 1502 of the monitoring service 1130 may be configured to store location data in association with identifiers of customers being monitored at the monitoring location 104 within a plurality of records. For example, the location data may be stored in a record along with an identifier of the customer and / or an identifier of the monitoring location 104 to associate the location data with the customer and the monitoring location 104. The image data store 1504 of the monitoring service 1130 may be configured to store one or more image data frames in association with identifiers of the location and timestamp at which the image data was acquired within a plurality of records.

[0148] The AI service 1508 of the monitoring service 1130 may be configured to process images and / or image sequences to identify semantic regions, movements, faces, and other features within the image or image sequence. The event listening service 1510 of the monitoring service 1130 may be configured to scan received location data for events and, in the event an event is identified, execute one or more event handlers to process the event. In some embodiments, such event handlers may be configured to identify events and transmit messages regarding those events to one or more recipient services (e.g., customer service 1538 and / or monitoring service 106). Operations that the customer service 1538 and / or monitoring service 106 may perform based on events identified by the event listening service 1510 are further described below. In some embodiments, the event listening service 1510 may interoperate with the AI service 1508 to identify events within the image data.

[0149] The identity provider service 1512 may be configured to receive an authentication request including a security certificate from the monitoring client 1116. When the identity provider 1512 can authenticate the security certificate in the request (e.g., via a confirmation function, cross-reference lookup, or some other authentication process), the identity provider 1512 may transmit a security token in response to the request. The monitoring client 1116 may receive, store, and include the security token in subsequent location data (e.g., location data 1014A) packets such that a receiving transmission service (e.g., transmission service 1128A) can securely process (e.g., unpack / parse) the packet to extract the location data before passing the location data to the monitoring service 1130. In some embodiments, for example, the security token may be a JSON web token (JWT), such as the token 2002 described below in connection with Figure 17 Description of the token 2002. One or more transmission services 1128 of the monitoring center environment 1126 may be configured to receive location data packets 1514, authenticate the authenticity of the packets 1514, parse the packets 1514, and extract the location data encoded therein before passing the location data to the monitoring service 1130 for processing. The location data processed as such may include any of the location data types described above with reference to Figure 18 Any of the location data types described above. In some embodiments, each transmission service 1128 may be configured to process location data packets 1514 generated by location-based monitoring devices of a particular manufacturer and / or model. The monitoring client 1116 may be configured to generate a location data packet (e.g., location data packet 1514) based on sensor information received at the monitored location 104 and transmit it to the monitoring service 1130 via, for example, one or more networks 1120.

[0150] The monitoring service 106 may maintain a record of events identified by the event listening service 1510 and may assign each event to each monitoring agent 112 currently online with the monitoring application 126. The monitoring application 126 operated by a given monitoring agent 112 may then add the events assigned to that monitoring agent 112 to an event queue within, for example, Figure 16 The event window 306 shown for review by that monitoring agent 112. In some embodiments, a given monitoring application 126 may use data describing the events within its queue to retrieve location data and / or image data (from the location data storage 1502 and / or the image data storage 1504, respectively) for presentation within or associated with the event window 306.

[0151] In response to a monitoring agent 112 identifying a particular event to review (e.g., by clicking on an event window 306), the monitoring service 106 may interact with the camera streaming service 1506 to obtain an access certificate, enabling the establishment of a peer connection with one or more cameras 102 at the monitored location 104 corresponding to that event and reviewing live video and / or audio streamed from those cameras within, for example, Figure 13 The video feed window 604 and / or the main viewer window 606 shown, as well as communicating verbally in real time with one or more individuals near the one or more cameras 102. The following describes an example interaction between components of the security system 1100 in conjunction with Figure 14 And Figure 11 Which enables the streaming of video and / or audio data between the one or more cameras 102 at the monitored location 104 and the monitoring application 126 operated by the monitoring agent 112.

[0152] Now turning to Figure 11, an example monitoring process 1600 that can be employed by the security system 1100 is illustrated as a sequence diagram. Specifically, in some embodiments, various parts of the process 1600 can be performed by (A) one or more location-based devices (e.g., Figure 11 or Figure 11 either of the processors 1302 or 1402 of Figure 11 under the control of device control system (DCS) code (e.g., code 1310 or 1410) implemented by a processor) such as devices 102A, 102B, 1106, 1108, and 1114; (B) a base station (e.g., Figure 11 of Figure 11 the base station 1112) under the control of a monitoring client (e.g., Figure 16 of Figure 15 the monitoring client 1116); (C) a monitoring center environment (e.g., Figure 12 of Figure 20 the monitoring center environment 1122) under the control of a monitoring application (e.g., Figure 11 of Figures 11 to 15 the monitoring application 126); (D) a monitoring center environment (e.g.,

[0153] as Figure 14 shown, the process 1600 can begin with the monitoring client 1116 authenticating to the monitoring service 1130 by exchanging one or more authentication requests and responses 1604 with the monitoring service 1130. More specifically, in some embodiments, the monitoring client 1116 can transmit an authentication request to the monitoring service 1130 via one or more API calls to the monitoring service 1130. In such an embodiment, the monitoring service 1130 can parse the authentication request to extract a security certificate therefrom and pass such a security certificate to an identity provider (e.g., Figure 3 of the identity provider service 1512) for authentication. In some embodiments, when the identity provider authenticates the security certificate, the monitoring service 1130 can generate a security token and communicate the security token as a payload within the authentication response to the authentication request. In such an embodiment, if the identity provider cannot authenticate the security certificate, the monitoring service 1130 can instead generate an error (e.g., an error code) and transmit the error as a payload within the authentication response to the authentication request. Upon receiving the authentication response, the monitoring client 1116 can parse the authentication response to extract the payload. If the payload includes an error code, the monitoring client 1116 can retry the authentication and / or its user interface with the host device (e.g.,Figure 3 interoperate with the user interface 1214 of the base station 1112 to present an output indicating authentication failure. If the payload includes a security token, the monitoring client 1116 may store the security token for subsequent use in the transmission of location data. It should be noted that in some embodiments, the security token may have a limited lifespan (e.g., one hour, one day, one week, one month, etc.), after which the monitoring client 1116 may be required to re-authenticate with the monitoring service 1130. In some embodiments, for example, the lifespan of the security token (e.g., a token 2002 of the type described below in connection with Figure 6 can be defined in the header 2004 and / or payload 2006 of the token 2002 (e.g., as one or more claims).

[0154] Continuing with process 1600, one or more device control systems 1602 hosted by one or more location-based devices may obtain (1606) sensor data describing a location (e.g., Figure 16 the monitored location 104). The sensor data so obtained can be any of a variety of types, as discussed above with reference to Figure 17 . In some embodiments, one or more device control systems 1602 may obtain sensor data continuously. In other embodiments, one or more DCSs 1602 may additionally or alternatively obtain sensor data in response to an event such as a timer expiration (push event) or receipt of an acquisition poll signal (poll event) transmitted by the monitoring client 1116. In some embodiments, one or more device control systems 1602 may stream the sensor data to the monitoring client 1116 with minimal processing other than acquisition and digitization. In such embodiments, the sensor data may constitute a vector sequence with individual vector members, including, for example, sensor readings and timestamps. In some embodiments, one or more device control systems 1602 may perform additional processing of the sensor data, such as generating one or more summaries of multiple sensor readings. Still further, in some embodiments, one or more device control systems 1602 may perform complex processing of the sensor data. For example, if the (one or more) sensors 1422 of the sensor assembly 1424 ( Figure 18 shown in) include an image capture device, the device control system 1602 may perform image processing routines such as edge detection, motion detection, face recognition, threat assessment, event generation, etc.

[0155] Continuing process 1600, one or more device control components 1602 may transmit sensor data 1608 to monitoring client 1116. Similar to sensor data acquisition, one or more device control systems 1602 may transmit sensor data 1608 continuously or in response to events such as push events (originating from one or more device control systems 1602) or poll events (originating from monitoring client 1116).

[0156] Continuing process 1600, monitoring client 1116 may monitor (1610) monitored location 104 by processing the received sensor data 1608. In some embodiments, for example, monitoring client 1116 may execute one or more image processing routines. Such image processing routines may include any of the image processing routines described above with reference to operation 1606. By distributing at least some of the image processing routines between one or more device control systems 1602 and monitoring client 1116, the power consumption of battery-powered devices can be reduced by offloading processing from line-powered devices. Also, in some embodiments, monitoring client 1116 may execute an overall threat detection process that utilizes sensor data 1608 from multiple different device control systems 1602 as input. For example, in some embodiments, monitoring client 1116 may attempt to confirm an open state received from a contact sensor with motion and facial recognition processing of an image of a scene including a window or door secured by the contact sensor. If two or more of the three processes indicate the presence of an intruder, a score (e.g., a threat score) may be increased and / or a break-in event may be declared, locally recorded, and transmitted. Other processing that monitoring client 1116 may perform includes outputting a local alert (e.g., in response to detecting a specific event and / or meeting other criteria) and detecting maintenance conditions of location-based devices such as the need to replace or recharge a low battery and / or replace / maintain the device hosting one or more device control systems 1602. Any of the above processes within operation 1610 may result in the creation of location data specifying the results of these processes.

[0157] Continuing process 1600, monitoring client 1116 may transmit location data 1612 to monitoring service 1130 (via one or more transmission services 1128). Similar to the transmission of sensor data 1608, monitoring client 1116 may transmit location data 1612 continuously or in response to events such as push events (originating from monitoring client 1116) or poll events (originating from monitoring service 1130).

[0158] Continuing with process 1600, the monitoring service 1130 may process (1614) the received location data. In some embodiments, for example, the monitoring service 1130 may perform one or more of the processes described above with reference to operations 1606 and / or 1610. In some embodiments, the monitoring service 1130 may additionally or alternatively calculate a score (e.g., a threat score) or further refine an existing score using historical information associated with the monitored location 104 identified in the location data and / or other locations that are geographically close to the monitored location 104 (e.g., within the same Zip code). For example, in some embodiments, if multiple break-ins have been recorded for the monitored location 104 and / or other locations within the same Zip code, the monitoring service 1130 may increase the score calculated by the device control system 1602 and / or the monitoring client 1116.

[0159] In some embodiments, the monitoring service 1130 may apply a set of rules and guidelines to the location data 1612 to determine whether the location data 1612 includes any events, and if so, transmit event reports 1616A and / or 1616B to the monitoring application 126 and / or the client application 128. In some embodiments, for example, the monitoring service 106 may assign one or more events to a particular monitoring agent 112 such that the events will be forwarded to the monitoring application 126 in which the monitoring agent 112 operates, e.g., for presentation within the corresponding event window 306 ( Figure 18 as shown). An event may be, for example, a particular type of event (e.g., a break-in) or a particular type of event that meets additional criteria (e.g., movement within a particular area in combination with a threat score that exceeds a threshold). The event reports 1616A and / or 1616B may have a priority based on the same criteria used to determine whether the events reported therein are reportable, or may have a priority based on a different set of criteria or rules.

[0160] Continuing with process 1600, the monitoring application 126 in the monitoring center environment 1122 may interact (1618) with the monitoring agent 112 via, for example, one or more GUIs such as Figure 17 and Figure 3 the screens 302 and 602 shown. Such GUIs may provide details and context regarding one or more events.

[0161] As Figure 20 shown, the client application 128 of the client device 114 (e.g., a smart phone, a personal computer, or other endpoint device) may similarly interact (1620) with at least one client via, for example, one or more GUIs. Such GUIs may provide details and context regarding one or more events.

[0162] Note that the processing of sensor data and / or location data as described above with reference to operations 1606, 1610, and 1614 can be performed by processors disposed within various parts of the security system 1100. In some embodiments, the (one or more) device control systems 1602 may perform minimal processing of the sensor data (e.g., just acquire and stream), and the remainder of the above processing may be performed by the monitoring client 1116 and / or the monitoring service 1130. This approach can help extend the battery runtime of location-based devices. In other embodiments, the (one or more) device control systems 1602 may perform as much sensor data processing as possible such that the monitoring client 1116 and the monitoring service 1130 only perform processes that require sensor data across location-based devices and / or locations. This approach can help improve the scalability of the security system 1100 in terms of adding new locations.

[0163] Figure 20 and Figure 18 illustrates example techniques for establishing a peer-to-peer connection (e.g., for video and / or audio streaming) between the camera 102 at the monitoring location 104 and either or both of (A) a monitoring application 126 hosted on or otherwise accessible by the monitoring device 110 and (B) a client application 128 hosted on or otherwise accessible by the client device 114. In some embodiments, the monitoring application 126 and the client application 128 may be web applications accessed using browsers hosted on the monitoring device 110 and the client device 114, respectively, and the WebRTC capabilities of these browsers may be used to establish peer connections with the camera 102. As described below in connection with Figure 20 the camera streaming service 1506 may provide a signaling channel for establishing peer connections between the camera 102 and the respective browsers. As an example, the camera streaming service 1506 may be implemented using the Amazon Kinesis Video Streams service provided by Amazon Web Services (AWS).

[0164] As r>Figure 17 shown by arrow 1702 in, the monitoring application 126 may provide a user token to the monitoring service 106. The user token may correspond to a monitoring agent 112 that has been authenticated to the monitoring application 126 and may be included in a request for live streaming access to the (one or more) cameras 102 at the monitoring location 104. In some embodiments, for example, in response to the monitoring agent 112 selecting an event window 306 corresponding to a particular camera 102, such a camera access request may be sent from the monitoring application 126 to the monitoring service 106, as described above in connection with Figure 20 the user token may be a JWT, such as described below in connection withFigure 17 The described token 2002.

[0165] The monitoring service 106 can evaluate a user token received from the monitoring application 126 (e.g., by verifying the signature 2008 of the token, as described below in connection with Figure 18 ), and if valid, can communicate with the camera streaming service 1506 to obtain an access token for a signaling channel that the monitoring application 126 can subsequently use to access the camera streaming service 1506. Example processes are described below in connection with Figure 18 by which signaling information can be exchanged between the monitoring application 126 and the camera 102 via a signaling channel established by the camera streaming service 1506 to determine configuration information for a peer-to-peer connection between the monitoring application 126 and the camera 102. In some embodiments, the access token obtained from the camera streaming service 1506 can be a JWT, such as the token 2002 described below in connection with Figure 17 The described token 2002.

[0166] As Figure 18 indicated by arrow 1704 in, in some embodiments, the monitoring service 106 can authenticate on behalf of the monitoring application 126 that provided the user token to the camera streaming service 1506 and request access to the camera streaming service 1506 (as per arrow 1702). In some embodiments, the access request sent by the monitoring service 106 to the camera streaming service 1506 can specify one or more parameters that identify the particular monitoring location 104 in question, the particular camera(s) 102 to which access is to be granted, the particular time window during which access to such a camera 102 is to be granted, and / or any other restrictions or constraints among a plurality of other restrictions or constraints regarding whether and / or how access to the camera(s) 102 is to be allowed. Using these parameters can help ensure that the camera(s) 102 are accessed only by authorized personnel and only when needed to evaluate a particular event. Upon authenticating the access request received from the monitoring service 106, the camera streaming service 1506 can establish a signaling channel between the monitoring application 126 and the camera 102 and generate an access token (e.g., a token of the type described below in connection with Figure 18 ), which the monitoring application 126 can subsequently use to access the signaling channel (e.g., by making a network API call to an API endpoint of the camera streaming service 1506). In some embodiments, the monitoring service 106 can configure the access token to include one or more parameters specified in the access request. For example, in some embodiments, such parameters can be defined in the header 2004 and / or the payload 2006 of the access token (e.g., as one or more claims).

[0167] As Figure 17 indicated by arrows 1706 and 1708 in, camera streaming service 1506 can send the generated access token to monitoring service 106, and then monitoring service 106 can pass the access token to monitoring application 126. In some embodiments, camera streaming service 1506 can also send additional information along with the access token, such as the network address of the signaling channel established by camera streaming service 1506 (e.g., network API endpoint), thereby allowing monitoring application 126 to make network API calls to camera streaming service 1506 for signaling purposes. As previously described, the access token generated by camera streaming service 1506 can be configured based on the parameters included in the access request sent by monitoring service 106 to camera streaming service 1506 to limit the ability of the receiving monitoring application 126 to access the established signaling channel in the manner defined by these parameters. For example, the access token generated by camera streaming service 1506 can be set to expire after a specific period based on a time limit parameter included in the access request.

[0168] As described below in connection with Figure 18 it, when receiving the access token from monitoring service 106, monitoring application 126 can send a Session Description Protocol (SDP) offer to the network address of the signaling channel, and the signaling channel can forward the SDP offer to camera 102, thereby initiating a signaling process to establish a peer connection between monitoring application 126 and the identified camera 102. Finally, also as described below in connection with Figure 18 it, as Figure 19 indicated by arrows 1710 and 1712 in, when suitable Interactive Connectivity Establishment (ICE) candidates are identified, one or more peer connections can be established between monitoring application 126 and camera 102, thereby enabling video data to be streamed from camera 102 to monitoring application 126 and / or audio data to be exchanged between monitoring application 126 and camera 102.

[0169] A similar process can be employed to establish one or more peer connections between client application 128 and one or more cameras 102 at the monitoring location, thereby enabling video data to be streamed from the (one or more) cameras 102 to client application 128 and / or audio data to be exchanged between client application 128 and cameras 102. Thus, this process will not be described further herein. However, it should be understood that the allowed scope provided in the access request sent from client service 1538 to camera streaming service 1506 can be different from (e.g., less restrictive than) the allowed scope provided by the access request sent from monitoring service 106 to camera streaming service 1506, as it may not be desirable to limit the ability of the client to perform live streaming using the camera in the same manner as monitoring agent 112.

[0170] Figure 19 Sequence diagram 1800 illustrates how signaling information (e.g., WebRTC signaling information) can be exchanged between the monitoring application 126 (or alternatively, the client application 128) and the camera 102 via the camera streaming service 1506 to establish a peer-to-peer connection between the monitoring application 126 (or alternatively, the client application 128) and the camera 102. Although Figure 20 the exchange of signaling information between the monitoring application 126 and the camera 102 is depicted, and the following sections describe the exchange of signaling information between these two components, it should be understood that the same process can also be used to exchange signaling information between the client application 128 and the camera 102. As described above, in some embodiments, in response to providing a user token to the monitoring service 106 (see Figure 21 arrow 1702 in), the monitoring application 126 may have received an access token for the camera streaming service 1506 from the monitoring service 106 (see Figure 1A arrow 1708 in), and such an access token may enable the monitoring application 126 to access the signaling channel established by the camera streaming service 1506, thereby allowing the monitoring application 126 to make network API calls to the camera streaming service 1506 for signaling purposes.

[0171] As Figure 1A shown, the signaling process can begin with the monitoring application 126 sending (1802A, 1802B) an SDP offer (via the camera streaming service 1506) to the camera 102 using the received access token. The monitoring application 126 can create an SDP offer, for example, by calling the CreateOffer() function of the WebRTC application programming interface (API) of the browser of the monitoring device 110 or other WebRTC-enabled components. The SDP offer can include information about the type of media to be sent by the monitoring device 110, its format, the transport protocol used, the Internet Protocol (IP) address and port of the monitoring device 110, and / or other information describing the media to be transmitted and / or required by the monitoring device 110.

[0172] When receiving an SDP offer from the monitoring application 126, the camera 102 may send (1804A, 1804B) an SDP answer to the monitoring application 126 via the camera streaming service 1506. The camera 102 may create the SDP answer, for example, by calling the CreateAnswer() function of the WebRTC API of the browser of the camera 102 or other WebRTC-enabled components. The SDP answer may include information about the type of media to be sent by the camera 102, its format, the transport protocol used, the Internet Protocol (IP) address and port of the camera 102, and / or other information describing the media to be transmitted and / or required by the camera 102.

[0173] In addition to sharing information about the media to be exchanged and the corresponding devices that will exchange it, the monitoring application 126 and the camera 102 may share information about the network connections they can use to exchange the media. Specifically, the monitoring application 126 may share one or more ICE candidates with the camera 102, and vice versa, where each ICE candidate is sent by a device describing the available methods that the device can use to communicate (directly or by using a relay traversing a NAT (TURN) server). The monitoring application 126 and the camera 102 may collect ICE candidates, for example, by using the WebRTC API to create an ICE candidate event listener (e.g., by calling the function peerConnection.addEventListener('icecandidate', event =>{…})). In some embodiments, the corresponding devices may first present their best ICE candidates, moving in the direction towards their poorer candidates. Ideally, ICE candidates employ the User Datagram Protocol (UDP) (since it is faster and media streams can recover from interruptions relatively easily), but the ICE standard also allows Transmission Control Protocol (TCP) candidates.

[0174] Possible UDP candidate types include host, peer reflexive (prflx), server reflexive (srflx), and relay. A "host" candidate is one whose IP address is the actual direct IP address of the remote peer. A "peer reflexive" candidate is one whose IP address comes from a symmetric Network Address Translation (NAT) between two peers. A "server reflexive" candidate is generated by a UDP Session Traversal Utilities for NAT (STUN) server. A relay candidate is generated by a TURN server. Possible TCP candidate types include active, passive, and so. An "active" transport will attempt to open an outbound connection but will not receive incoming connection requests. A "passive" transport will receive incoming connection attempts but will not itself attempt to connect. An "so" transport will attempt to open a connection to its peer simultaneously.

[0175] As an example, Figure 1B illustrates how the monitoring application 126 can send (1806A, 1806B) ICE candidate "A" to the camera 102 and how the camera 102 can send (1808A, 1808B) ICE candidate "B" to the monitoring application 126. Different pairs of the identified ICE candidates can be tested, and an endpoint that has been designated as the "control agent" can select one of the identified ICE candidate pairs for establishing (1810) a peer connection between the monitoring application 126 and the camera 102.

[0176] Additional information regarding using WebRTC to establish peer connections can be found on a web page accessible via the Uniform Resource Locator (URL) "webrtc.org", the entire content of which is incorporated herein by reference. Now turning to Figure 1A , the computing device 1900 is schematically illustrated. As Figure 22 shown, the computing device 1900 may include at least one processor 1902, volatile memory 1904, one or more interfaces 1906, non-volatile memory 1908, and an interconnection mechanism 1914. The non-volatile memory 1908 may include executable code 1910 and, as illustrated, may additionally include at least one data store 1912.

[0177] In some embodiments, the non-volatile (non-transitory) memory 1908 may include one or more read-only memory (ROM) chips; one or more hard disk drives or other magnetic or optical storage media; one or more solid-state drives (SSDs), such as flash drives or other solid-state storage media; and / or one or more hybrid magnetic and SSDs. Further, in some embodiments, the code 1910 stored in the non-volatile memory may include an operating system and one or more applications or programs configured to execute under the control of the operating system. In some embodiments, the code 1910 may additionally or alternatively include proprietary firmware and embedded software that is executable without relying on a commercially available operating system. Regardless of its configuration, the execution of the code 1910 may produce manipulated data that can be stored as one or more data structures in the data store 1912. The data structures may have fields that are related by their positions in the data structure. Such an association can also be achieved by allocating storage for fields in positions within the memory that convey the association between the fields. However, other mechanisms can be used to establish associations between the information in the fields of the data structure, including by using pointers, tags, or other mechanisms.

[0178] The processor 1902 of the computing device 1900 may be embodied by one or more processors configured to execute one or more executable instructions, such as a computer program specified by code 1910, to control the operation of the computing device 1900. The functionality, operations, or sequence of operations may be hard-coded into the circuitry or soft-coded by instructions held in a memory device (e.g., volatile memory 1904) and executed by the circuitry. In some embodiments, the processor 1902 may be embodied by one or more application-specific integrated circuits (ASICs), microprocessors, digital signal processors (DSPs), graphics processing units (GPUs), neural processing units (NPUs), microcontrollers, field-programmable gate arrays (FPGAs), programmable logic arrays (PLAs), or multi-core processors.

[0179] Before executing the code 1910, the processor 1902 may copy the code 1910 from the non-volatile memory 1908 to the volatile memory 1904. In some embodiments, the volatile memory 1904 may include one or more static or dynamic random access memory (RAM) chips and / or cache memory (e.g., memory disposed on the silicon die of the processor 1902). The volatile memory 1904 may provide a faster response time than main memory, such as the non-volatile memory 1908. By executing the code 1910, the processor 1902 may control the operation of the interface 1906. The interface 1906 may include a network interface. Such a network interface may include one or more physical interfaces (e.g., radio transceiver devices, Ethernet ports, USB ports, etc.) and a software stack that includes drivers and / or other code 1910 configured to communicate with the one or more physical interfaces to support one or more LAN, PAN, and / or WAN standard communication protocols. Such communication protocols may include, for example, TCP and UDP. Thus, the network interface may enable the computing device 1900 to access and communicate with other computing devices via a computer network. One or more interfaces 1906 may include one or more user interfaces. For example, in some embodiments, one or more user interfaces 1906 may include user input and / or output devices (e.g., keyboard, mouse, touch screen, display, speaker, camera, accelerometer, biometric scanner, environmental sensor, etc.) and a software stack that includes drivers and / or other code 1910 configured to communicate with the user input and / or output devices. Thus, one or more user interfaces 1906 may enable computing device 1900 to interact with a user to receive input and / or present output. The presented output may include, for example, one or more GUIs that include one or more controls configured to display output and / or receive input. The received input may specify values to be stored in data storage 1912. The displayed output may indicate values stored in data storage 1912. The various features of computing device 1900 described above may communicate with each other via an interconnect mechanism 1914. In some embodiments, interconnect mechanism 1914 may include a communication bus.

[0180] Figure 1A An example token 2002 is shown, such as a JSON Web Token (JWT), which may be employed by the various system components described above. As illustrated, token 2002 may include a header 2004, a payload 2006, and a signature 2008. In some embodiments, header 2004 may specify a signing technique for generating signature 2008 based on the content of header 2004 and / or payload 2006 and a private key. In some embodiments, for example, the specified signing technique may involve (A) combining base64url-encoded header and base64url-encoded payload, (B) hashing the combined base64url values using a hashing technique (e.g., SHA256), and (C) encrypting the determined hash using the private key. Thus, by using the private key and the specified signing technique to verify signature 2008, a receiving device can confirm that the content of token 2002 received from another device has not been altered or otherwise compromised. In some embodiments, header 2004 or payload 2006 of token 2002 may additionally include an identifier of the device for which it was generated, thereby enabling the receiving device to confirm that the received token 2002 originated from the same device for which token 2002 was originally generated, thereby restricting the use of token 2002 by other devices to which it may have been transmitted. Figure 3 An example process 2100 is shown in accordance with some embodiments of the present disclosure, which may be performed by edge image processing component 120 and / or remote image processing component 122 ( Figure 4 shown) to implement what was described above in connection with Figures 10A to 10C and Figure 4Certain functions of the described security system 100.

[0181] In step 2110 of process 2100, the first image processing component (e.g., Figure 23 the image processing module of the edge image processing component 120 and / or the remote image processing component 122 as shown) can be made to process the image data corresponding to the event (e.g., the image data acquired by the image sensor 118) to identify the first feature of the image data (e.g., the frame of the image data represents motion, a person, a face, or the recognized face).

[0182] In step 2120 of process 2100, at least partially based on the first image processing component identifying the first feature of the image data, the second image processing component (e.g., another image processing module of the edge image processing component 120 and / or the remote image processing component 122) can be made to further process the image data to identify the second feature of the image data (e.g., the frame of the image data represents motion, a person, a face, or the recognized face).

[0183] In step 2130 of process 2100, at least partially based on the second image processing component identifying the second feature of the image data, the computing device (e.g., the monitoring device 110 or the client device 114) can be made to output an indication related to the event.

[0184] Figure 1A An example process 2200 is shown, which can be executed by one or more components of the security system 100 ( Figure 1A shown therein) to generate the delay bars 310, 1020 of the type described above in connection with Figure 1C 、 Figure 7 and Figures 8A to 8C described.

[0185] In step 2210 of process 2200, one or more components of the computing system (e.g., the monitoring service 106, the monitoring device 110, or the client device 114 of the security system 100) can receive video data acquired by the camera 102 (e.g., via the image sensor 118).

[0186] In step 2220 of process 2200, at least one image processing component of the computing system (e.g., the edge image processing component 120 and / or the remote image processing component 122) can process the video data to determine that the first feature is represented in the first frame of the video data (e.g., the frame of the video data represents motion, a person, a face, or the recognized face).

[0187] In step 2230 of process 2200, one or more components of the computing system can store the first data indicating the identification of the first feature in the first frame (e.g., one or more feature predictions 214).

[0188] In step 2240 of process 2200, one or more components of a computing system may use the first data to generate a latency bar for video data (e.g., latency bar 310 as shown), where the latency bar includes a playback progress indicator (e.g., playback progress indicator 402) indicating a first relative position of the currently displayed frame within the sequence of frames of the video data, and a first feature indicator (e.g., feature indicator 408) indicating a second relative position of a first frame within the frame sequence. Figure 1A

[0189] Figure 24 An example process 2300 is shown, which may be performed by the monitoring application 126 (shown in Figure 9A ) to send a notification about an event to the customer 116 based on the notification preferences 704 of the customer 116 (e.g., as described above in connection with Figure 9A , Figure 25 , Figure 1A and Figure 1A ).

[0190] In step 2310 of process 2300, a first application (e.g., the monitoring application 126) executed on a first computing system operated by a first user (e.g., the monitoring application 126 executed on the monitoring device 110 operated by the monitoring agent 112) may receive first data (e.g., image data acquired by the image sensor 118 as shown) representing at least a first image acquired by a camera at a property. Figure 7

[0191] In step 2320 of process 2300, at least in part based on the first data, the first application (e.g., the monitoring application 126) may cause the first computing system (e.g., the monitoring device 110) to output a first indication of a first event at the property, the first indication including at least the first image.

[0192] In step 2330 of process 2300, the first application (e.g., the monitoring application 126) may receive a first input (e.g., an input identifying a cancellation reason 218 or an event disposition 220) from a first user (e.g., the monitoring agent 112) indicating that the first event is of a first type.

[0193] In step 2340 of process 2300, first preference data associated with the property (e.g., notification preferences 704) may be determined, which indicates that a notification of the first type of event is to be sent to a second user associated with the property (e.g., the customer 116 associated with the monitoring location 104).

[0194] In step 2350 of process 2300, based at least in part on a first input indicating that the first event is of a first type and first preference data indicating that a notification of an event of the first type is to be sent to a second user, a first application (e.g., monitoring application 126) may cause a notification regarding the first event to be sent to an endpoint device (e.g., client device 114) associated with the second user, the notification causing the endpoint device to output a second indication of the first event (e.g., as a push notification, text message, or email).

[0195] ​ An example process 2400 is shown that may be performed by one or more components of security system 100 (shown in FIG. A) to allow customer 116 to specify one or more filters to be applied to events presented on client device 114, e.g., as described in the scenario 902 shown above in connection with ​ the screen 902 shown.

[0196] In step 2410 of process 2400, one or more components of the system (e.g., monitoring service 106 of security system 100 and / or client device 114) may determine first event data for a first event and second event data for a second event at one or more properties (e.g., one or more monitoring locations 104). For example, the first event data may correspond to a first detected motion at one or more properties, and the second event data may correspond to a second detected motion. As indicated, the first event data may include first image data associated with the first event (e.g., image data 212 acquired by image sensor 118), and may also include a first indication of a first characteristic of the first event (e.g., the location of the first event, the camera that detected the first event, the time of the first event, or the type of the first event). Similarly, the second event data may include second image data associated with the second event (e.g., image data 212 acquired by image sensor 118), and may also include a second indication of a second characteristic of the second event (e.g., the location of the second event, the camera that detected the second event, the time of the second event, or the type of the second event), where the second characteristic is different from the first characteristic.

[0197] In step 2420 of process 2400, in response to a first user request to view an event having the first characteristic (e.g., selecting one of the user interface elements 910, 912 shown in ​ ), the computing device (e.g., client device 114) displays at least the first image data based at least in part on the first event data including the first indication.

[0198] In step 2430 of process 2400, in response to a second user request to view an event having the second characteristic (e.g., selecting ​Another one of the user interface elements 910, 912 shown), such that the computing device displays at least second image data based at least in part on second event data including the second indication.

[0199] ​ An example process 2500 is shown, which may be performed by the monitoring application 126 ( ​ shown in) to flag events for review by the customer 116 based on the event review preferences 706 of the customer 116, e.g., as described above in connection with ​ , ​ and Figure 9B described.

[0200] In step 2510 of process 2500, a first application (e.g., the monitoring application 126) executing on a first computing system operated by a first user (e.g., the monitoring device 110 operated by the monitoring agent 112) may receive first data representing at least a first image acquired by a camera at a property (e.g., image data acquired by the Figure 1A image sensor 118 shown).

[0201] In step 2520 of process 2500, at least in part based on the first data, the first application (e.g., the monitoring application 126) may cause the first computing system (e.g., the monitoring device 110) to output a first indication of a first event at the property, the first indication including at least the first image.

[0202] In step 2530 of process 2500, the first application (e.g., the monitoring application 126) may receive a first input from a first user indicating that the first event is of a first type (e.g., an input identifying a cancellation reason 218 or an event disposition 220).

[0203] In step 2540 of process 2500, preference data associated with the property (e.g., event review preferences 706) may be determined, which indicates that events of the first type are to be flagged for review via a second application (e.g., the customer application 128).

[0204] In step 2550 of process 2500, at least in part based on the first input indicating that the first event is of a first type and the preference data (e.g., event review preferences 706) indicating that events of the first type are to be flagged for review via a second application (e.g., the customer application 128), the system may cause second data (e.g., customer review status 222) to be stored in association with the first data, such that the second data causes the second application (e.g., the customer device 114) to output a prompt to review information about the first event (e.g., under the Figure 9B "Events to Review" label 906 shown).

[0205] Figure 26 illustrates an example process 2600 that may be performed by a client application 128 ( Figure 1A shown in) to enable a client 116 to provide feedback for changing one or more functions of an edge image processing component 120 and / or a remote image processing component 122, e.g., as described above in connection with Figure 1A , Figure 2 , Figure 10C and Figure 10D .

[0206] In step 2610 of process 2600, a first application (e.g., client application 128) executed on an endpoint device operated by a first user associated with a property (e.g., client application 128 executed on client device 114 operated by client 116 associated with monitoring location 104) may receive first data that at least represents a first image acquired by a camera at the property (e.g., image data acquired by image sensor 118), and identify a first feature of the first image identified by at least one image processing component of a remote computing system (e.g., one or more image processing modules of edge image processing component 120 and / or remote image processing component 122).

[0207] In step 2620 of process 2600, based at least in part on the first data, the first application (e.g., client application 128) may cause the endpoint device (e.g., client device 114) to display a first indication of the first image and the first feature (e.g., as thumbnail image 1022 and an associated feature identifier, such as Figure 10A shown).

[0208] In step 2630 of process 2600, the first application (e.g., client application 128) may receive at least one input from a first user (e.g., client 116) corresponding to the first image (e.g., as a tap and hold input, such as Figure 10C shown).

[0209] In step 2640 of process 2600, the first application (e.g., client application 128) may send second data to a remote computing system (e.g., monitoring service 106) for changing the function of at least one image processing component (e.g., via a new visitor profile 708 that may be created by edge image processing component 120 and / or remote image processing component 122, or using client feedback 224 to retrain one or more image processing modules of edge image processing component 120 and / or remote image processing component 122), the second data being at least partially based on the at least one input.

[0210] The following clauses describe examples of the inventive concepts disclosed herein.

[0211] Clause 1. A method, comprising: receiving, by a first application executing on a first computing system operated by a first user, first data representing at least a first image acquired by a camera at a property; causing, by the first application and at least partially based on the first data, the first computing system to output a first indication of a first event at the property, the first indication including at least the first image; receiving, by the first application, a first input from the first user indicating that the first event is of a first type; determining that first preference data associated with the property indicates that a notification of an event of the first type will be sent to a second user associated with the property; and causing, at least partially based on the first input indicating that the first event is of the first type and the first preference data indicating that a notification of an event of the first type will be sent to the second user, the first application to send a notification about the first event to an endpoint device associated with the second user, the notification causing the endpoint device to output a second indication of the first event.

[0212] Clause 2. The method according to Clause 1, wherein the first data further represents a first feature of the first image, the first feature having been determined by at least one image processing component.

[0213] Clause 3. The method according to Clause 2, wherein the at least one image processing component includes at least one machine learning (ML) model.

[0214] Clause 4. The method according to Clause 2, wherein the at least one image processing component includes at least a first image processing component and a second image processing component, and the method further comprises: causing the first image processing component to process the first image to identify a second feature of the first image; and causing, at least partially based on the first image processing component identifying the second feature, the second image processing component to further process the first image to identify the first feature of the first image.

[0215] Clause 5. The method according to Clause 4, wherein the first image processing component includes at least one first machine learning model.

[0216] Clause 6. The method according to Clause 4 or Claim 5, wherein the second image processing component includes at least one second machine learning model.

[0217] Clause 7. The method according to any one of Clauses 2 to 6, further comprising: causing, by the first application, the first computing system to include a representation of the first feature in the first indication.

[0218] Clause 8. The method according to any one of Clauses 2 to 7, further comprising: configuring, by the first application, the notification to include a representation of the first feature.

[0219] Clause 9. The method according to any one of Clauses 2 to 8, wherein the first input identifies why the first feature does not present a security issue.

[0220] Clause 10. The method according to Clause 9 further includes: notifying by the first application configuration to include an indication of the reason.

[0221] Clause 11. The method according to any one of Clauses 1 to 10 further includes: notifying by the first application configuration to include a first image.

[0222] Clause 12. The method according to any one of Clauses 1 to 11 further includes: notifying by the first application configuration to include an indication that the first event is of a first type.

[0223] Clause 13. The method according to any one of Clauses 1 to 12 further includes: receiving, by the first application, a second input from a first user corresponding to a request to send a notification to a second user; wherein the first application further causes the notification to be sent to the endpoint device at least in part based on the second input.

[0224] Clause 14. The method according to any one of Clauses 1 to 13, wherein the first data further represents a plurality of frames of video data acquired by a camera in response to detecting a motion event at a property; and the method further includes causing, by the first application, the first computing system to play back the plurality of frames of video data.

[0225] Clause 15. The method according to Clause 14, wherein the first data further represents a second feature of a first frame of the video data, the second feature having been determined by at least one image processing component; and the method further includes generating, using the first data, a time-lapse bar for the video data, wherein the time-lapse bar includes a playback progress indication indicating a first relative position of the currently displayed frame within the sequence of frames of the video data and a first feature indication indicating a second relative position of the first frame within the sequence of frames.

[0226] Clause 16. The method according to any one of Clauses 1 to 15 further includes: receiving, by the endpoint device, a third input indicating a first preference that a notification indicating an event of a first type will be sent to a second user; determining first preference data based on the third input; and causing the first preference data to be stored in association with an identifier of the property.

[0227] Clause 17. The method according to any one of Clauses 1 to 16 further includes: determining that second preference data associated with the property indicates that an event of a first type will be marked for review via a second application executed on the endpoint device; and causing, at least in part based on a first input indicating that the first event is of a first type and the second preference data indicating that an event of a first type will be marked for review via the second application, the second data to be stored in association with the first data, whereby the second data causes the second application to output a prompt to review information about the first event.

[0228] Clause 18. The method according to Clause 17 further includes: causing the second application to output the first image for viewing by the second user.

[0229] Clause 19. The method according to any one of Clauses 17 or 18 further includes: determining an action taken by the first user when viewing the first event; causing third data corresponding to the action to be stored in association with the first data; and causing the second application to output information indicating the action, at least in part based on the third data.

[0230] Clause 20. The method according to any one of Clauses 2 to 10 further includes: determining that second preference data associated with the property indicates that a first type of event will be marked for viewing via a second application executed on an endpoint device; causing the second data to be stored in association with the first data, at least in part based on the first input indicating that the first event is of the first type and the second preference data indicating that the first type of event will be marked for viewing via the second application, whereby the second data causes the second application to output a prompt to view the first image; and causing the second application to output the first image and a representation of the first feature for viewing by the second user.

[0231] Clause 21. The method according to any one of Clause 20 further includes: determining an action taken by the first user when viewing the first event; causing third data corresponding to the action to be stored in association with the first data; and causing the second application to output information indicating the action, at least in part based on the third data.

[0232] Clause 22. A method includes: receiving, by a first application executed on a first computing system operated by a first user, first data representing at least a first image acquired by a camera at a property; causing, by the first application and at least in part based on the first data, the first computing system to output a first indication of a first event at the property, the first indication including at least the first image; receiving, by the first application, a first input from the first user indicating that the first event is of a first type; determining that preference data associated with the property indicates that the first type of event will be marked for viewing via a second application; and causing the second data to be stored in association with the first data, at least in part based on the first input indicating that the first event is of the first type and the preference data indicating that the first type of event will be marked for viewing via the second application, whereby the second data causes the second application to output a prompt to view information about the first event.

[0233] Clause 23. The method according to Clause 22, wherein the first data further represents a first feature of the first image, the first feature having been determined by at least one image processing component.

[0234] Clause 24. The method according to Clause 23, wherein the at least one image processing component includes at least one machine learning (ML) model.

[0235] Clause 25. The method according to Clause 23, wherein at least one image processing component includes at least a first image processing component and a second image processing component, and the method further includes: causing the first image processing component to process a first image to identify a second feature of the first image; and causing the second image processing component to further process the first image to identify a first feature of the first image at least partially based on the second feature identified by the first image processing component.

[0236] Clause 26. The method according to Clause 25, wherein the first image processing component includes at least one first machine learning model.

[0237] Clause 27. The method according to Clause 25 or Claim 26, wherein the second image processing component includes at least one second machine learning model.

[0238] Clause 28. The method according to any one of Clauses 23 to 27, further includes: causing the second application to output a representation of the first feature and the first image.

[0239] Clause 29. The method according to any one of Clauses 23 to 28, wherein the first input identifies why the first feature does not present a security issue.

[0240] Clause 30. The method according to Clause 29, further includes: causing the second application to output an indication of the reason.

[0241] Clause 31. The method according to any one of Clauses 23 to 30, further includes: causing the second application to output the first image for viewing by a second user of the endpoint device; determining that the second user provides a second input to the second application to select the first image; and receiving, from the second application, second data for changing the function of at least one image processing component, the second data being at least partially based on the second input.

[0242] Clause 32. The method according to Clause 31, wherein the second data includes profile data indicating that the person represented in the first image is an authorized visitor to the property.

[0243] Clause 33. The method according to Clause 31 or Claim 32, wherein the second data includes an indication that the first feature is inaccurately identified by at least one image processing component in the first image.

[0244] Clause 34. The method according to any one of Clauses 22 to 30, further includes: causing the second application to output the first image for viewing by a second user of the endpoint device.

[0245] Clause 35. The method according to any one of Clauses 22 to 34, further includes: causing the second application to output an indication that the first event is of a first type.

[0246] Clause 36. A method according to any one of Clauses 22 to 35, wherein the first data further represents a plurality of frames of video data acquired by a camera in response to detecting a motion event at a property; and the method further includes causing a second application to play back the plurality of frames of video data.

[0247] Clause 37. A method according to Clause 36, wherein the first data further represents a second feature of a first frame of the video data, the second feature having been determined by at least one image processing component; and the method further includes generating a time-lapse bar for the video data using the first data, wherein the time-lapse bar includes a playback progress indicator indicating a first relative position of a currently displayed frame within a sequence of frames of the video data and a first feature indicator indicating a second relative position of the first frame within the sequence of frames.

[0248] Clause 38. A method according to any one of Clauses 22 to 37, further including: receiving, by the second application, a second input indicating a first preference that notifications of a first type of event are to be marked for review via the second application; determining preference data based on the second input; and causing the preference data to be stored in association with an identifier of the property.

[0249] Clause 39. A method according to any one of Clauses 22 to 38, further including: determining an action taken by a first user when reviewing a first event; causing third data corresponding to the action to be stored in association with the first data; and causing the second application to output information indicating the action, at least in part based on the third data.

[0250] Clause 40. A method including: receiving, by a computing system, video data acquired by a camera; processing, by at least one image processing component of the computing system, the video data to determine that a first feature is represented in a first frame of the video data; storing, by the computing system, first data indicating that the first feature is recognized in the first frame; and generating, by the computing system and using the first data, a time-lapse bar for the video data, wherein the time-lapse bar includes a playback progress indicator indicating a first relative position of a currently displayed frame within a sequence of frames of the video data and a first feature indicator indicating a second relative position of the first frame within the sequence of frames.

[0251] Clause 41. A method according to Clause 40, wherein the first feature includes detected motion.

[0252] Clause 42. A method according to Clause 40, wherein the first feature includes detected people.

[0253] Clause 43. A method according to Clause 40, wherein the first feature includes detected faces.

[0254] Clause 44. The method according to Clause 40, wherein the first feature includes the recognized face.

[0255] Clause 45. The method according to any one of Clauses 40 to 44, wherein the first feature is of a first feature type, and the method further comprises: processing, by a computing system, video data to determine a second feature representation of a second feature type different from the first feature type in a second frame of the video data; storing, by the computing system, second data indicating that the second feature is recognized in the second frame; and generating, by the computing system and using the second data, a delay bar that includes a second feature indication indicating a third relative position of the second frame within a sequence of frames.

[0256] Clause 46. The method according to Clause 45, wherein the second feature includes a detected motion.

[0257] Clause 47. The method according to Clause 45, wherein the second feature includes a detected person.

[0258] Clause 48. The method according to Clause 45, wherein the second feature includes a detected face.

[0259] Clause 49. The method according to Clause 45, wherein the second feature includes the recognized face.

[0260] Clause 50. The method according to any one of Clauses 45 to 49, further comprising: generating a first feature indication having at least one first characteristic corresponding to the first feature type; and generating a second feature indication having at least one second characteristic corresponding to the second feature type, wherein at least one second characteristic is different from at least one first characteristic.

[0261] Clause 51. The method according to Clause 50, wherein at least one first characteristic includes a first color; and at least one second characteristic includes a second color, wherein the second color is different from the first color.

[0262] Clause 52. The method according to Clause 50 or Claim 49, wherein at least one first characteristic includes a first vertical position relative to a playback progress indication; and at least one second characteristic includes a second vertical position relative to the playback progress indication, wherein the second vertical position is different from the first vertical position.

[0263] Clause 53. The method according to any one of Clauses 40 to 52, wherein at least one image processing component includes at least one machine learning (ML) model.

[0264] Clause 54. A method according to any one of Clauses 40 to 53, wherein at least one image processing component includes at least a first image processing component and a second image processing component, and the method further includes: causing the first image processing component to process a first frame to identify a second feature of the first frame; and causing the second image processing component to further process the first frame to identify a first feature of the first frame, at least in part based on the second feature identified by the first image processing component.

[0265] Clause 55. A method according to Clause 54, wherein the first image processing component includes at least one first machine learning model.

[0266] Clause 56. A method according to Clause 54 or Claim 53, wherein the second image processing component includes at least one second machine learning model.

[0267] Clause 57. A method according to any one of Clauses 40 to 56, wherein the delay bar is configured to cause the playback of video data to jump to a frame of the video data corresponding to a selected position on the playback progress indicator.

[0268] Clause 58. A method includes: determining first event data of a first event corresponding to detected motion at one or more properties and second event data of a second event corresponding to detected motion at one or more properties, wherein the first event data includes first image data associated with the first event and further includes a first indication of a first characteristic of the first event, the second event data includes second image data associated with the second event and further includes a second indication of a second characteristic of the second event, and the second characteristic is different from the first characteristic; in response to a first user request to view an event having the first characteristic, causing a computing device to display at least the first image data at least in part based on the first event data including the first indication; and in response to a second user request to view an event having the second characteristic, causing the computing device to display at least the second image data at least in part based on the second event data including the second indication.

[0269] Clause 59. A method according to Clause 58, wherein the first characteristic includes a first property at which the first event was detected; the first indication includes a first identifier of the first property; the second characteristic includes a second property at which the second event was detected, wherein the second property is different from the first property; and the second indication includes a second identifier of the second property.

[0270] Clause 60. A method according to Clause 58, wherein the first characteristic includes a first date on which the first event was detected; the first indication includes a first identifier of the first date; the second characteristic includes a second date on which the second event was detected, wherein the second date is different from the first date; and the second indication includes a second identifier of the second date.

[0271] Clause 61. The method according to Clause 58, wherein the first characteristic includes a first camera for detecting a first event; the first indication includes a first identifier of the first camera; the second characteristic includes a second camera for detecting a second event, wherein the second camera is different from the first camera; and the second indication includes a second identifier of the second camera.

[0272] Clause 62. The method according to Clause 58, wherein the first characteristic includes a first event type; the first indication includes a first identifier of the first event type; the second characteristic includes a second event type, wherein the second event type is different from the first event type; and the second indication includes a second identifier of the second event type.

[0273] Clause 63. The method according to Clause 62, wherein the first event type includes a detected motion event.

[0274] Clause 64. The method according to Clause 62, wherein the first event type includes a detected person event.

[0275] Clause 65. The method according to Clause 62, wherein the first event type includes a detected facial event.

[0276] Clause 66. The method according to Clause 62, wherein the first event type includes a recognized facial event.

[0277] Clause 67. The method according to any one of Clauses 64 to 66, wherein the second event type includes a detected motion event.

[0278] Clause 68. The method according to Clause 63, 63 or 64, wherein the first event type includes a detected person event.

[0279] Clause 69. The method according to Clause 63, 62 or 64, wherein the second event type includes a detected facial event.

[0280] Clause 70. The method according to any one of Clauses 63 to 65, wherein the second event type includes a recognized facial event.

[0281] Clause 71. The method according to any one of Clauses 62 to 70, further comprising: determining that a user provides an input to the computing device to select an image corresponding to the first event; and receiving, from the computing device, feedback data for changing the function of at least one image processing component for determining the first event type, the feedback data being at least partially based on the input.

[0282] Clause 72. The method according to Clause 71, wherein the feedback data includes profile data indicating that the person represented in the image is an authorized visitor to one or more properties.

[0283] Clause 73. The method according to Clause 71 or Claim 72, wherein the feedback data includes an indication that a first event type is inaccurately recognized in the image by at least one image processing component.

[0284] Clause 74. The method according to any one of Clauses 58 to 73, further comprising: determining an action taken by the monitoring agent when consulting the first event; and causing the computing device to output information indicating the action.

[0285] Clause 75. A method comprising: causing a first image processing component to process image data corresponding to an event to identify a first feature of the image data; causing a second image processing component to further process the image data to identify a second feature of the image data, at least in part based on the first image processing component identifying the first feature of the image data; and causing the computing device to output an indication related to the event, at least in part based on the second image processing component identifying the second feature of the image data.

[0286] Clause 76. The method according to Clause 75, wherein the first image processing component includes at least one first machine learning model.

[0287] Clause 77. The method according to any one of Clause 75 or Claim 76, wherein the second image processing component includes at least one second machine learning model.

[0288] Clause 78. The method according to any one of Clauses 75 to 77, wherein the first image processing component performs edge processing for a camera that captures the image data.

[0289] Clause 79. The method according to any one of Clauses 75 to 77, wherein the first image processing component performs cloud-based processing of at least a portion of the image data.

[0290] Clause 80. The method according to any one of Clauses 75 to 79, wherein the second image processing component performs cloud-based processing of at least a portion of the image data.

[0291] Clause 81. The method according to any one of Clauses 75 to 80, wherein causing the first image processing component to process the image data further includes: causing the first image processing component to identify one or more key frames in the image data that potentially represent the second feature.

[0292] Clause 82. The method according to Clause 81, wherein causing the second image processing component to further process the image data further includes: causing the second image processing component to process the one or more key frames to perform motion detection.

[0293] Clause 83. The method according to Clause 81, wherein causing the second image processing component to further process the image data further includes: causing the second image processing component to process one or more key frames to perform human detection.

[0294] Clause 84. The method according to Clause 81, wherein causing the second image processing component to further process the image data further includes: causing the second image processing component to process one or more key frames to perform face detection.

[0295] Clause 85. The method according to Clause 81, wherein causing the second image processing component to further process the image data further includes: causing the second image processing component to process one or more key frames to perform face recognition.

[0296] Clause 86. The method according to any one of Clauses 75 to 80, wherein causing the first image processing component to process the image data further includes causing the first image processing component to identify one or more first frames representing motion in the image data; causing the second image processing component to further process the image data includes causing the second image processing component to perform human detection on one or more first frames; and causing the computing device to further output an indication at least partially based on detecting a human in one or more first frames.

[0297] Clause 87. The method according to Clause 86, further including: causing the third image processing component to perform face detection on one or more second frames at least partially based on the second image processing component detecting a human in one or more second frames among one or more first frames; and causing the computing device to further output an indication at least partially based on detecting a face in one or more second frames.

[0298] Clause 88. The method according to Clause 87, further including: causing the fourth image processing component to perform face recognition on one or more third frames at least partially based on the third image processing component detecting a face in one or more third frames among one or more second frames; and causing the computing device to further output an indication at least partially based on recognizing a face in one or more third frames.

[0299] Clause 89. The method according to any one of Clauses 75 to 80, wherein causing the first image processing component to process the image data further includes causing the first image processing component to identify one or more first frames representing a human in the image data; causing the second image processing component to further process the image data includes causing the second image processing component to perform face detection on one or more first frames; and causing the computing device to further output an indication at least partially based on detecting a face in one or more first frames.

[0300] Clause 90. The method according to Clause 89 further includes: at least partially based on the second image processing component detecting a face in one or more second frames of one or more first frames, causing the third image processing component to perform face recognition on the one or more second frames; and causing the computing device to further output an indication at least partially based on the face recognized in the one or more second frames.

[0301] Clause 91. The method according to any one of Clauses 75 to 80, wherein causing the first image processing component to process the image data further includes causing the first image processing component to identify one or more first frames of the image data representing a face; causing the second image processing component to further process the image data includes causing the second image processing component to perform face recognition on the one or more first frames; and causing the computing device to further output an indication at least partially based on the face recognized in the one or more first frames.

[0302] Clause 92. A method includes: receiving, by a first application executed on an endpoint device operated by a first user associated with a property, first data that at least represents a first image acquired by a camera at the property and identifies a first feature of the first image recognized by at least one image processing component of a remote computing system; causing, by the first application and at least partially based on the first data, the endpoint device to display the first image and a first indication of the first feature; receiving, by the first application, at least one input from the first user corresponding to the first image; and sending, from the first application to the remote computing system, second data for changing the function of the at least one image processing component, the second data being at least partially based on the at least one input.

[0303] Clause 93. The method according to Clause 92, wherein the second data includes profile data indicating that the person represented in the first image is an authorized visitor to the property.

[0304] Clause 94. The method according to Clause 93 further includes: causing the at least one image processing component to perform face recognition corresponding to the person represented in the first image using the profile data.

[0305] Clause 95. The method according to any one of Clauses 92 to 94, wherein the second data includes a second indication that the first feature is inaccurately recognized by the at least one image processing component in the first image.

[0306] Clause 96. The method according to Clause 95 further includes: using the second indication to retain the at least one image processing component.

[0307] Clause 97. The method according to any one of Clauses 92 to 96, wherein the at least one input includes a tap-and-hold gesture on the first image displayed by the endpoint device.

[0308] Clause 98. A method according to any one of Clauses 92 to 97, wherein at least one image processing component includes at least one machine learning model.

[0309] Clause 99. A method according to any one of Clauses 92 to 98, wherein at least one image processing component includes at least a first image processing component and a second image processing component, and the method further includes: causing the first image processing component to process a first image to identify a second feature of the first image; and causing the second image processing component to further process the first image to identify a first feature of the first image, at least in part based on the second feature identified by the first image processing component.

[0310] Clause 100. The method according to Clause 99, wherein the first image processing component includes at least one first machine learning model.

[0311] Clause 101. The method according to Clause 99 or Claim 100, wherein the second image processing component includes at least one second machine learning model.

[0312] Clause 102. A method including: receiving, by a first application executing on a first computing system operated by a first user, first data representing at least a first image acquired by a camera at a property; causing, by the first application and at least in part based on the first data, the first computing system to output a first indication of a first event at the property, the first indication including at least the first image; receiving, by the first application, a first input from the first user indicating that the first event is of a first type; determining that preference data associated with the property indicates that an event of the first type is to be marked for review via a second application; and causing second data to be stored in association with the first data, at least in part based on the first input indicating that the first event is of the first type and the preference data indicating that an event of the first type is to be marked for review via the second application, such that the second data causes the second application to output a prompt to review information about the first event.

[0313] Clause 103. The method according to Clause 102, wherein the first data further represents a first feature of the first image, the first feature having been determined by at least one image processing component.

[0314] Clause 104. The method according to Clause 103, further including: causing the second application to output a representation of the first feature and the first image.

[0315] Clause 105. The method according to Clause 103 or Clause 104, wherein the first input indicates that there is no security issue with the first feature.

[0316] Clause 106. The method according to any one of Clauses 103 to 105, further including: causing the second application to output an indication that there is no security issue with the first feature.

[0317] Clause 107. The method according to any one of Clauses 102 to 106 further includes: receiving, by a second application, a second input of a first preference indicating that a notification of an event of a first type is to be marked for review via the second application; and determining preference data based on the second input.

[0318] Clause 108. A method includes: determining first event data of a first event corresponding to detected movement at one or more properties and second event data of a second event corresponding to detected movement at one or more properties, wherein the first event data includes first image data associated with the first event and further includes a first indication of a first characteristic of the first event, and the second event data includes second image data associated with the second event and further includes a second indication of a second characteristic of the second event, the second characteristic being different from the first characteristic; in response to a first user request to review an event having the first characteristic, causing a computing device to display at least the first image data based at least in part on the first event data including the first indication; and in response to a second user request to review an event having the second characteristic, causing the computing device to display at least the second image data based at least in part on the second event data including the second indication.

[0319] Clause 109. The method according to Clause 108, wherein the first characteristic includes a first property at which the first event was detected; the first indication includes a first identifier of the first property; the second characteristic includes a second property at which the second event was detected, wherein the second property is different from the first property; and the second indication includes a second identifier of the second property.

[0320] Clause 110. The method according to Clause 108, wherein the first characteristic includes a first date on which the first event was detected; the first indication includes a first identifier of the first date; the second characteristic includes a second date on which the second event was detected, wherein the second date is different from the first date; and the second indication includes a second identifier of the second date.

[0321] Clause 111. The method according to Clause 108, wherein the first characteristic includes a first camera used to detect the first event; the first indication includes a first identifier of the first camera; the second characteristic includes a second camera used to detect the second event, wherein the second camera is different from the first camera; and the second indication includes a second identifier of the second camera.

[0322] Clause 112. The method according to Clause 108, wherein the first characteristic includes a first event type; the first indication includes a first identifier of the first event type; the second characteristic includes a second event type, wherein the second event type is different from the first event type; and the second indication includes a second identifier of the second event type.

[0323] Clause 113. The method according to Clause 112, wherein the first event type includes a detected motion event.

[0324] Clause 114. The method according to Clause 112 or Clause 113, wherein the second event type includes a detected person event.

[0325] Clause 115. A method, comprising: causing a first image processing component to process image data corresponding to an event to identify a first feature of the image data; causing a second image processing component to further process the image data to identify a second feature of the image data at least in part based on the first image processing component identifying the first feature of the image data; and causing a computing device to output an indication related to the event at least in part based on the second image processing component identifying the second feature of the image data.

[0326] Clause 116. The method according to Clause 115, wherein the first image processing component is included in a camera that captures the image data.

[0327] Clause 117. The method according to Clause 116, wherein the second image processing component is remote from the camera that captures the image data.

[0328] Clause 118. The method according to any one of Clauses 115 to 117, wherein causing the first image processing component to process the image data further includes: causing the first image processing component to identify one or more key frames in the image data that potentially represent the second feature.

[0329] Clause 119. The method according to Clause 118, wherein causing the second image processing component to further process the image data further includes: causing the second image processing component to process the one or more key frames to perform person detection.

[0330] Clause 120. The method according to Clause 118, wherein causing the second image processing component to further process the image data further includes: causing the second image processing component to process the one or more key frames to perform face detection.

[0331] Clause 121. The method according to Clause 118, wherein causing the second image processing component to further process the image data further includes: causing the second image processing component to process the one or more key frames to perform face recognition.

[0332] Clause 122. A system, comprising: one or more processors; and one or more non-transitory computer-readable media encoded with instructions that, when executed by the one or more processors, cause the system to perform the method according to any one of Clauses 1 to 121.

[0333] Clause 123. One or more non-transitory computer-readable media encoded with instructions that, when executed by one or more processors of a system, cause the system to perform a method according to any one of Clauses 1 through 121.

[0334] The various inventive concepts may be embodied as one or more methods, examples of which have been provided. The acts performed as part of the method may be ordered in any suitable way. Accordingly, examples may be constructed in which acts are performed in an order different than that illustrated, which may include performing some acts simultaneously, even though shown as sequential acts in illustrative examples.

[0335] The use of ordinal terms such as "first," "second," "third," etc. in the claims to modify the claim elements themselves does not mean any precedence, order, or sequence of one claim element with respect to another, or the temporal order of acts of a method. These terms are merely used as labels to distinguish one claim element having a particular name from another element having the same name (but using an ordinal term).

[0336] The examples of the methods and systems discussed herein are not limited in application to the details of the construction and arrangement of components set forth in the following description or illustrated in the drawings. The methods and systems are capable of being implemented in other examples and of being practiced or carried out in a variety of ways. The examples of the specific implementations provided herein are for illustrative purposes only and are not intended to be limiting. In particular, the acts, components, elements, and features discussed in connection with any one or more examples are not intended to be excluded from a similar role in any other examples.

[0337] Moreover, the language and terminology used herein are for descriptive purposes and should not be regarded as limiting. Any reference herein to examples, components, elements, or acts of a system and method in the singular may also include examples that include a plurality, and any reference to examples, components, elements, or acts in the plural herein may also include examples that include only a singular. References in the singular or plural form are not intended to limit the systems or methods of the present disclosure, their components, acts, or elements.

[0338] The use of "including," "comprising," "having," "containing," "involving," and variations thereof herein means including the items listed thereafter and their equivalents as well as additional items. References to "or" may be construed as inclusive such that any terms described using "or" may indicate any one of the described terms, singular, more than one, and all. Additionally, in the case of inconsistent usage of terms between this document and the documents incorporated herein by reference, the usage of the terms in the incorporated references is supplementary to the usage in this document; for irreconcilable inconsistencies, the usage of the terms in this document prevails.

[0339] Having described several examples in detail, various modifications and improvements will readily occur to those skilled in the art. These modifications and improvements are intended to be within the scope of the present disclosure. Accordingly, the foregoing description is merely exemplary and not restrictive.

[0340] What is claimed in this application is:

Claims

1. A method comprising: receiving, by a first application executed on a first computing system operated by a monitoring agent, first data representing recorded video of a first event, the recorded video having been acquired by a camera at a property; causing, by the first application and based at least in part on the first data, the first computing system to output the recorded video; receiving, by the first application from the monitoring agent, a first input after causing the first computing system to output the recorded video, the first input identifying a first event type category selected from at least three predetermined event type categories, the first event having been classified within the first event type category; as well as The first event type category is identified based at least in part on the first input, causing a second application of a user to output a prompt to review information about the first event.

2. The method according to claim 1, wherein: The first data further represent first characteristics of frames of the recorded video, the first characteristics having been determined by at least one image processing component.

3. The method according to claim 2, further comprising: The second application is caused to output a representation of the first feature and the frame.

4. The method according to claim 2 or 3, wherein: The first input indicates that the first feature does not have a security issue.

5. The method according to any one of claims 2 to 4, further comprising: The second application is caused to output an indication that the first feature has no security issues.

6. The method according to any one of claims 1 to 5, further comprising: determining that preference data associated with the property indicates that events within the first event type category are to be flagged for review via the second application; Wherein causing the second application to output the prompt is also based at least in part on the preference data indicating that events within the first event type category are to be marked for review.

7. The method according to claim 6, further comprising: receiving, by the second application, a second input of a first preference indicating that events classified within the first event type category are marked for review via the second application; as well as The preference data is determined based on the second input.

8. A system comprising: one or more processors; and One or more computer-readable media encoded with instructions that, when executed by the one or more processors, cause the system to: receiving, by a first application executed on a first computing system operated by a monitoring agent, first data representing recorded video of a first event, the recorded video having been acquired by a camera at a property; causing, by the first application and based at least in part on the first data, the first computing system to output the recorded video; receiving, by the first application from the monitoring agent, a first input after having caused the first computing system to output the recorded video, the first input identifying a first event type category selected from at least three predetermined event type categories, the first event having been classified within the first event type category; as well as The first event type category is identified based at least in part on the first input, causing a second application of a user to output a prompt to review information about the first event.

9. The system according to claim 8, wherein: The first data further represent first characteristics of frames of the recorded video, the first characteristics having been determined by at least one image processing component.

10. The system according to claim 9, wherein: The one or more computer readable media are further encoded with additional instructions that, when executed by the one or more processors, further cause the system to: The second application is caused to output a representation of the first feature and the frame.

11. The system according to claim 9 or 10, wherein: The first input indicates that the first feature does not have a security issue.

12. A system according to any one of claims 9 to 11, wherein: The one or more computer readable media are further encoded with additional instructions that, when executed by the one or more processors, further cause the system to: The second application is caused to output an indication that the first feature has no security issues.

13. The system according to claim 8, wherein: The one or more computer readable media are further encoded with additional instructions that, when executed by the one or more processors, further cause the system to: determining that preference data associated with the property indicates that events within the first event type category are to be flagged for review via the second application; as well as The second application is caused to output the prompt further based at least in part on the preference data indicating that events within the first event type category are to be marked for review.

14. The system according to claim 13, wherein: The one or more computer readable media are further encoded with additional instructions that, when executed by the one or more processors, further cause the system to: receiving, by the second application, a second input of a first preference indicating that events classified within the first event type category are marked for review via the second application; as well as The preference data is determined based on the second input.

15. One or more non-transitory computer-readable media encoded with instructions that, when executed by one or more processors of a system, cause the system to: receiving, by a first application executed on a first computing system operated by a monitoring agent, first data representing recorded video of a first event, the recorded video having been acquired by a camera at a property; causing, by the first application and based at least in part on the first data, the first computing system to output the recorded video; receiving, by the first application from the monitoring agent, a first input after having caused the first computing system to output the recorded video, the first input identifying a first event type category selected from at least three predetermined event type categories, the first event having been classified within the first event type category; as well as The first event type category is identified based at least in part on the first input, causing a second application of a user to output a prompt to review information about the first event.

16. One or more non-transitory computer-readable media according to claim 15, wherein: The first data further represent first characteristics of frames of the recorded video, the first characteristics having been determined by at least one image processing component.

17. The one or more non-transitory computer readable media of claim 16, further encoded with additional instructions that, when executed by the one or more processors, further cause the system to: The second application is caused to output a representation of the first feature and the frame.

18. One or more non-transitory computer readable media according to claim 16 or 17, wherein: The first input indicates that the first feature does not have a security issue.

19. The one or more non-transitory computer readable media of any one of claims 16 to 18, further encoded with additional instructions that, when executed by the one or more processors, further cause the system to: The second application is caused to output an indication that the first feature has no security issues.

20. The one or more non-transitory computer readable media of claim 15, further encoded with additional instructions that, when executed by the one or more processors, further cause the system to: determining that preference data associated with the property indicates that events within the first event type category are to be flagged for review via the second application; and The second application is caused to output the prompt further based at least in part on the preference data indicating that events within the first event type category are to be marked for review.

21. The one or more non-transitory computer readable media of claim 20, further encoded with additional instructions that, when executed by the one or more processors, further cause the system to: receiving, by the second application, a second input indicating a first preference that events categorized within the first event type category are marked for review via the second application; and The preference data is determined based on the second input.

Images