Biometric data processing for security systems

Abstract

A computing system may perform image processing on image data acquired by a camera at a property to determine at least a first feature of the image data. Based at least in part on determining that the first feature satisfies a rule, the computing system may generate a vector representing a plurality of characteristics of the face represented in the image data. The vector may then be used to determine that a particular person representation is in the image data.

Description

Background Art

[0001] Some security systems enable the use of cameras and other devices to remotely monitor a location. Brief Description of the Drawings

[0002] Additional examples of the present invention, as well as its features and advantages, will become more apparent by reference to the description taken in conjunction with the drawings that are incorporated herein and form a part of the present invention. The drawings are not necessarily drawn to scale.

[0003] Figure 1 An example security system in accordance with some embodiments of the present invention is shown.

[0004] Figure 2 Shown are example tables that can be used by the security system shown in accordance with some embodiments of the present invention to store data related events Figure 1

[0005] Figure 3 Shown are example tables that can be used by the security system shown in accordance with some embodiments of the present invention to store customer specific settings Figure 1

[0006] Figure 4 Shown is an example table that can be used to store document data of individual persons, and the document data can be used by the security system shown Figure 1 to perform biometric face recognition.

[0007] Figure 5 Shown is a first example screen that can be presented on a monitoring device of the security system shown in accordance with some embodiments of the present invention Figure 1

[0008] Figure 6 Shown is a second example screen that can be presented on a monitoring device of the security system shown in accordance with some embodiments of the present invention Figure 1

[0009] Figure 7A Shown is an example process that can be performed by a terminal application shown in accordance with some embodiments of the present invention to determine biometric settings for a given property Figure 1

[0010] Figure 7B Shown is an example opt-out screen that can be presented on a terminal device during the execution of the process shown in accordance with some embodiments of the present invention Figure 7A during the execution of the process shown Figure 1

[0011] Figure 8 ​​​​​​Shows an example process that can be performed according to some embodiments of the present invention to determine whether to generate and / or store biometric embeddings of image data.

[0012] Figure 9A Shows an example process for creating a visitor document for an individual according to some embodiments of the present invention.

[0013] Figure 9B Shows that according to some embodiments of the present invention, it can be performed by Figure 1 An example screen presented by the terminal device shown to enable input of information for a new visitor document.

[0014] Figure 10 Shows an example embodiment of a security system according to some embodiments of the present invention including Figure 1 The several components shown.

[0015] Figure 11 Is a schematic diagram of a computing device that can be used according to some embodiments of the present invention to implement Figure 1 And Figure 11 The security system shown, a monitoring device, and / or one or more services.

[0016] Figure 12 Shows an example embodiment of a base station of the security system according to some embodiments of the present invention Figure 10 Shown.

[0017] Figure 13 Shows an example embodiment of a keypad of the security system according to some embodiments of the present invention Figure 10 Shown.

[0018] Figure 14 Shows an example embodiment of a security sensor of the security system according to some embodiments of the present invention Figure 10 Shown.

[0019] Figure 15 Shows an example embodiment of a monitoring center environment and a monitoring center environment of the security system according to some embodiments of the present invention Figure 10 Shown.

[0020] Figure 16 Is a sequence diagram of a monitoring process that can be performed by components of the security system shown according to some embodiments of the present invention Figure 10 Shown. Detailed implementation

[0021] Automated facial recognition technology is used for various purposes, such as controlling access to a smart phone. However, the use of biometric data for such purposes is becoming increasingly regulated by, for example, government entities, making it difficult for technology providers to fully utilize facial recognition technology. As used herein, "biometric data" refers to any data representing physical or behavioral characteristics of a person that can be used for identification purposes. In a typical facial recognition system, various features of the face, such as the eyes, nose, and mouth, are precisely determined and measured in a facial image to generate a feature vector. Such a feature vector can include an array of numerical values representing the various measured characteristics of the face. As a simple example, a facial image can be mapped into a feature vector that includes numerical values representing various features, such as: • Facial height (cm) • Facial width (cm) • Average color of the face (R, G, B) • Lip width (cm) • Nose height (cm) In such an example, where the facial height is measured as 23.2 cm, the facial width is measured as 14.9 cm, the average red, green, and blue values of the face are measured as 250, 221, and 173 respectively, the lip width is measured as 5.1 cm, and the nose height is measured as 3.9 cm, the feature vector of the face can be represented as (23.2, 14.9, 250, 221, 173, 5.1, 3.9). There are numerous other features that can also be derived from the image and included in the feature vector of the face, such as features related to eye shape, nose width, lip shape, hair color, eye socket depth, facial hair, glasses, etc.

[0022] Then, the feature vector thus established can be compared with a database of documents containing feature vectors corresponding to the faces of known individuals in an attempt to find a match. Such a feature vector is sometimes referred to as a "biometric embedding" or simply an "embedding".

[0023] Regulations regarding the use of biometric data can vary significantly by location, making it challenging to design a system that can be widely deployed and still comply with such diverse regulations. In existing security systems, devices typically need to be configured for the respective venues in which they may be deployed to ensure compliance with various regulatory regimes. There is an increasing need for devices that can be customized for operation but still enable mass production. Moreover, existing technologies do not enable customers to adjust the operation of their security systems to meet their individual privacy expectations.

[0024] A security system is provided that includes various components and technical improvements designed to ensure that facial recognition technology is adopted in a manner that does not violate regulations applicable to a particular geographic region, while at the same time providing customers with the ability to control whether and how to use facial recognition technology in conjunction with their security systems. In some embodiments, for example, the system may employ one or more components to perform various checks before generating an embedding of a face represented in data acquired by a camera and / or storing data representing such biometric embeddings. Such checks can be used, for example, to identify one or more specific situations in which the system will avoid using biometric face recognition in order to avoid acquiring and / or storing embeddings in such situations. Examples of situations in which the system may avoid using biometric face recognition include (A) situations where local regulations prohibit the use of biometric face recognition, (B) situations where the customer has opted out or not opted in to the use of biometric face recognition, (C) situations where the person (or face) detected in the image is greater than a threshold distance from the camera, (D) situations where the person (or face) detected in the image is outside the boundaries of the customer's property, and (E) situations where the person detected in the image is determined to be below a threshold age.

[0025] In some embodiments, the system may additionally or alternatively employ one or more components to ensure that consent is received from an individual (or, if the individual is a minor, then their guardian) before generating and storing an embedding of the face of those individuals in a visitor document that can be used to perform facial recognition. For example, in some embodiments, the system may allow the customer to enter a unique identifier of the individual for whom a visitor document is being created (e.g., a user identifier, an email address, or a phone number), and may solicit consent from that individual, for example, via email, text message, a push notification to an application (e.g., the terminal application 128 described below in Figure 1 ), to generate a biometric embedding of the individual. Such a message may include, for example, a user interface element that can be selected to provide consent (e.g., a button that says "click here to consent"), or may alternatively include a link to a web page or a user interface (UI) screen of an application (e.g., the terminal application 128) that describes the requested consent and includes a user interface element that can be selected to allow the individual to provide the requested consent. In some embodiments, the system may store biometric data in an individual's visitor document only when consent from the individual has been received.

[0026] In some embodiments, the system may be configured to use a camera to obtain an image of a face and / or to allow the creation of a visitor document including a face image without obtaining consent from the person involved, but the system may avoid generating and / or storing a biometric embedding corresponding to such an image unless certain conditions are met and / or consent is obtained to ensure compliance with applicable regulations. As explained in more detail below, obtaining and / or storing a face image in this manner, even without generating a biometric embedding of such an image, may enable a monitoring agent to visually review the face image and other information from the visitor document when reviewing a recorded or live video of an event at a monitored location to determine whether a particular individual is an authorized visitor.

[0027] To facilitate an understanding of the principles of the present invention, reference will now be made to the examples illustrated in the accompanying drawings, and these examples will be described using specific language. However, it should be understood that no limitation of the scope of the examples described herein is intended.

[0028] Figure 1 An example security system 100 according to some embodiments of the present invention is shown. As shown, the security system 100 may include: one or more cameras 102 disposed at a monitored location 104 (e.g., a residence, a company, a parking lot, etc.); a monitoring service 106 (e.g., including one or more servers 108) located remotely from the cameras 102 (e.g., within a cloud-based service such as the monitoring center environment 1026 described below in connection with Figure 10 and Figure 15 ); one or more monitoring devices 110 operated by respective monitoring agents 112; and one or more terminal devices 114 operated by respective customers 116. Although not shown in Figure 1 it should be understood that the various illustrated components may communicate with each other via one or more networks (e.g., one or more local area networks (LANs) and / or the Internet).

[0029] The camera 102 may include a motion sensor 130, an image sensor 118, an edge image processing component 120, etc. The monitoring service 106 may include a remote image processing component 122, one or more data stores 124 for storing data regarding events detected by the security system 100 (e.g., within the event table 202 described below in connection with Figure 2 ), and one or more data stores 132 for storing setting and document information (e.g., within the setting table 302 and the document table 402 described below in connection with Figure 3 and Figure 4 ). The data store 124 may correspond to, for example, the location data store 1502 and / or the image data store 1504 described below in connection with Figure 15 AlthoughFigure 1 The data storage 132 is illustrated as being located within the monitoring service 106, but it should be understood that in some embodiments, the data storage 132 (or one or more portions of the data storage 132) may be located elsewhere in the security system 100, e.g., within the camera 102 or at another location proximate to the edge image processing component 120. The monitoring device 110 may include a monitoring application 126 (e.g., executed on one or more processors of the monitoring device 110), etc. Finally, the terminal device 114 may include a terminal application 128 (e.g., executed on one or more processors of the terminal device 114), etc.

[0030] As Figure 1 shown, in some embodiments, one or more of the remote image processing component 122, the monitoring application 126, and the terminal application 128 may communicate with the data storage 124, e.g., via one or more networks, such as the network 1020 described below in connection with Figure 10 and Figure 15 Similarly, in some embodiments, one or more of the edge image processing component 120, the remote image processing component 122, the monitoring application 126, and the terminal application 128 may communicate with the data storage 132, e.g., via one or more networks, such as the network 1020 described below in connection with Figure 10 and Figure 15 For example, in some embodiments, the monitoring service 106 or another component within the monitoring center environment 1026 (see Figure 10 and Figure 15 ) may provide one or more application programming interfaces (APIs) that may be used by the edge image processing component 120, the remote image processing component 122, the monitoring application 126, and / or the terminal application 128 to write data to and / or retrieve data from the data storages 124, 132 as needed.

[0031] As Figure 1Note that the image sensor 118 can obtain image data (e.g., digital data representing a frame of one or more acquired pixel values) from the monitoring location 104, and as indicated by the arrow 134, can transfer the image data to the edge image processing component 120 and / or the remote image processing component 122 for processing. In some embodiments, for example, the motion sensor 130 can detect motion at the monitoring location 104 and provide an indication of the detected motion to the image sensor 118. The motion sensor 130 can be, for example, a passive infrared (PIR) sensor. In response to the indication of the detected motion, the image sensor 118 can start acquiring frames of image data from within the field of view of the camera. In some embodiments, the image sensor 118 can continue to collect frames of image data until the motion sensor 130 does not detect motion within a threshold time period (e.g., twenty seconds). Thus, the image data acquired by the image sensor 118 can represent a video clip of the scene within the field of view of the camera, which video clip starts when motion is first detected and ends after the motion has stopped within the threshold time period. In some embodiments, the camera 102 can be configured to emit a flash and / or generate an audible sound to indicate that the camera 102 is recording and processing video and audio.

[0032] In some embodiments, rather than relying on the motion sensor 130 (e.g., PIR sensor) to trigger the collection of frames of image data, the camera 102 can continuously collect frames of image data and rely on one or more image processing modules of the edge image processing component 120 (e.g., machine learning (ML) models and / or other computer vision (CV) processing components) to process the collected frames to detect motion within the field of view of the camera 102. Thus, in such embodiments, rather than relying on the motion indication provided by the motion sensor 130 to determine the start and end of the video clip for further processing, the camera 120 can rely on the motion indication provided by such image processing modules for that purpose.

[0033] The edge image processing component 120 may include one or more first image processing modules (e.g., ML models and / or other CV processing components) configured to identify corresponding features within the image data, and the remote image processing component 122 may include one or more second different image processing modules (e.g., ML models and / or other CV processing components) configured to identify corresponding features within the image data. The first image processing module and / or the second image processing module may be configured, for example, to perform processing on the image data to detect motion, identify people, identify faces, perform face recognition, etc. In some embodiments, for example, motion may be detected by detecting a meaningful difference between frames of the image data using one or more functions of the OpenCV library (accessible at the Uniform Resource Locator (URL) "opencv.org"). An example of an ML model that can be used for person detection is YOLO (accessible via the URL "github.com"). An example of an ML model that can be used for face detection is RetinaFace (accessible via the URL "github.com"). An example of an ML model that can be used for face recognition is AdaFace (accessible via the URL "github.com").

[0034] In some embodiments, the processing power of the server 108 employed by the monitoring service 106 may be significantly higher than the processing power of the processors included in the edge image processing component 120, thereby allowing the monitoring service 106 to employ more complex image processing modules and / or execute a greater number of such image processing modules in parallel.

[0035] The edge image processing component 120 and / or the remote image processing component 122 may include one or more image processing modules configured to perform face recognition on the image data acquired by the image sensor 102. For example, the edge image processing component 120 and / or the remote image processing component 122 may generate an embedding of an image representing a human face and compare the embedding with the embeddings of individual persons stored in the data storage 132 (e.g., the biometric embeddings 416 of the document table 402 as described below). In some embodiments, when two embeddings are within a threshold similarity of each other, the edge image processing component 120 and / or the remote image processing component 122 may determine that the generated embedding matches or otherwise corresponds to the stored embedding.

[0036] As Figure 1 indicated by the arrows 136 and 138 in Figure 3 shown, the edge image processing component 120 and / or the remote image processing component 122 may access the data storage 132 to obtain settings (e.g., customer settings) from the settings table 302 ( Figure 4as shown) to obtain one or more documents (e.g., visitor documents) for such purposes. For example, the documents to be accessed for a particular customer 116 can be identified via the "Visitor Documents" entry 310 in the settings table 302. The data obtained from the settings table 302 can additionally include biometric settings 312 of the customer 116. As explained below in connection with Figure 8 In some embodiments, such biometric settings 312 can be used by the edge image processing component 120 and / or the remote image processing component 122 to determine whether and / or how to perform face recognition using the image data acquired by the image sensor 118. Further, as described below in connection with Figure 7A and Figure 9A In some embodiments, the terminal application 128 can be configured to enable the customer 116 to create and / or modify the visitor documents stored in the document table 402, and / or set or adjust the biometric settings 312 stored in the settings table 302.

[0037] In some embodiments, the edge image processing component 120 can include one or more components configured to determine whether and / or how to perform face recognition processing using the image data based on one or more features of the biometric settings and the acquired image data. An example process 800 that can be performed by the edge image processing component 120 to make such a determination is described below in connection with Figure 8 In such embodiments, when the edge image processing component 120 determines to perform biometric face recognition on one or more images represented in the acquired image data, the edge image processing component 120 can generate biometric embeddings of such images and include these biometric embeddings in the edge processing results sent to the remote image processing component 122 (see arrow 140 in Figure 1 ) One or more components of the remote image processing component 122 can then use the determined biometric embeddings to perform face recognition, such as by comparing them with the biometric embeddings 414 from one or more visitor documents. In some embodiments, the remote image processing component 122 can store the determined biometric embeddings of the event as, for example, biometric embeddings 216 in the event table 202 (by Figure 1the arrow 142) in). In other embodiments, the remote image processing component 122 may compare the determined biometric embedding with the biometric embedding 414 from the visitor document before storing the determined embedding in the event table 202. In some embodiments, the determined biometric embedding may be stored in the event table 202 only if a matching biometric embedding 414 is identified in the document (e.g., the visitor document). In some embodiments, the remote image processing component 122 may discard the determined biometric embeddings after comparing them with the biometric embeddings 414 from the visitor document, thereby avoiding storing them in the event table 202 altogether.

[0038] In some embodiments, the remote image processing component 122 may additionally or alternatively include one or more components configured to determine whether and / or how to perform face recognition processing (e.g., by using biometric embeddings to identify faces in the image) on the image data acquired for the event based on biometric settings and one or more features of the acquired image data. In such embodiments, the process 800 ( Figure 8 shown) may be performed additionally or alternatively by the remote image processing component 122. In such embodiments, when the remote image processing component 122 determines to perform biometric face recognition on one or more images represented in the acquired image data, the remote image processing component 122 may generate biometric embeddings of such images. One or more components of the remote image processing component 122 may then use the determined biometric embeddings to perform face recognition, such as by comparing them with the biometric embeddings 414 from one or more documents. In some embodiments, the remote image processing component 122 may store the determined biometric embeddings of the event as, for example, the biometric embedding 216 in the event table 202 (as per Figure 1 the arrow 142) in). In other embodiments, the remote image processing component 122 may compare the determined biometric embedding with the biometric embedding 414 from the document before storing the determined embedding in the event table 202. In some embodiments, the determined biometric embedding may be stored in the event table 202 only if a matching biometric embedding 414 is identified in the document. In some embodiments, the remote image processing component 122 may discard the determined biometric embeddings after comparing them with the biometric embeddings 414 from the visitor document, thereby avoiding storing them in the event table 202 altogether.

[0039] As Figure 1As indicated by arrow 144 in, in some embodiments, the monitoring application 126 operated by the monitoring agent 112 may also receive customer settings from the settings table 302 and receive at least some data from the document table 402 (e.g., one or more images 412 of the documented person and identifiers 408, 410 of the locations where these people are authorized or unauthorized to access, as described below). The monitoring application 126 may use such information, for example, to display images of authorized (or unauthorized) individuals at the monitoring location 104 and images of faces detected by the edge image processing component 120 and / or the remote image processing component 122, so that the monitoring agent 112 can visually compare the detected faces with the faces of authorized (or unauthorized) individuals, even if the edge image processing component 120 and / or the remote image processing component 122 do not perform face recognition processing on the detected faces.

[0040] As Figure 1 As indicated by arrow 142 in, after the remote image processing component 122 has detected and / or confirmed the presence of one or more relevant features (e.g., motion, person, face, etc.) in the image data acquired by the image sensor 130, the remote image processing component 122 may upload the image data and data reflecting the identified features (collectively referred to as "event data") to the data storage 124 (e.g., to a row of the event table 202) for further review (e.g., by the monitoring agent 112 operating the monitoring application 126 and / or the customer 116 operating the terminal application 128). In other embodiments, the image data (e.g., video corresponding to the motion detected by the motion sensor 130) may be uploaded to the data storage 124 (e.g., to the corresponding row of the event table 202) when it is acquired, and the remote image processing component 122 may simply supplement the uploaded image data with data about the features identified by the remote image processing component 122 (e.g., by populating one or more columns of the event table 202).

[0041] Figure 2 FIG. shows an example event table 202 that can be used to store event data for various events detected by the system 100. As shown, for each event, the event table 202 may be populated with data representing an event identifier (ID) 204, a timestamp 206, a location ID 208, a camera ID 210, image data 212, one or more feature predictions 214, and one or more biometric embeddings 216, etc.

[0042] The event ID 204 may represent different events detected by the security system 100, and the data in the same row as a given event ID 204 may correspond to the same event.

[0043] The timestamp 206 can indicate the date and time when the corresponding event was detected.

[0044] The location ID 208 can identify the monitoring location 104 where the event was detected.

[0045] The camera ID 210 can identify the camera 102 associated with the corresponding detected event.

[0046] The image data 212 can represent one or more images (e.g., snapshots or videos) acquired by the camera 102 identified by the corresponding camera ID 210 when the event was detected.

[0047] The feature prediction 214 can include information about one or more features identified by the edge image processing component 120 and / or the remote image processing component 122. Such information can include, for example, an indication of the motion detected during the event, an indication of the person detected corresponding to the event, an indication of the face detected corresponding to the event, an indication of the face identified corresponding to the event (e.g., using face recognition processing performed by the edge image processing component 120 and / or the remote image processing component 122), the threat level assigned to the event, etc.

[0048] The biometric embedding 216 can represent an embedding generated for the face represented in the image data 212. As described in more detail below, in some embodiments, such an embedding may be generated only in certain cases and / or only when certain conditions are met to ensure compliance with local regulations, etc. Further, in some embodiments, even in cases where an embedding is generated for the acquired image, such an embedding may be stored in the event table 202 only in certain cases, such as when a matching biometric embedding 414 is identified in a visitor document (e.g., by comparing the biometric embedding 216 of the acquired image with the biometric embedding 414 of the visitor document). Storing the biometric embedding 216 of the acquired image in such cases can be useful, for example, as ground truth data for subsequent training or retraining of one or more image processing modules for face recognition.

[0049] Although Figure 2 not illustrated, it should be understood that the event table 202 may additionally include other data, which can be used for various purposes, such as facilitating the assignment of events to specific agents, the description of the event (e.g., "Motion detected by backyard camera"), the alarm status of the monitoring location 104, one or more recorded audio tracks associated with the event, the change in status of one or more sensors (e.g., door lock sensors) at the monitoring location 104, the actions taken or decisions made by the monitoring agent 112 upon receiving the event, etc. As described below in connection with Figure 6More specifically, in some embodiments, some such data can be used to populate one or more event data windows 608 that can be displayed by the monitoring device 110, as well as one or more live video feeds within the video feed window 604 and / or the main viewer window 606.

[0050] Notifications regarding "actionable" events represented in the event table 202 (e.g., events where the remote image processing component 122 identifies one or more features of interest) can be dispatched to the corresponding monitoring applications 126 for review by the monitoring agents 112. In some embodiments, the monitoring service 106 can use the contents of the event table 202 to assign individual events to the respective monitoring agents 112 having the monitoring applications 126. The monitoring application 126 operated by a given monitoring agent 112 can then add the events assigned to that monitoring agent 112 to an event queue for review by that monitoring agent 112.

[0051] Figure 5 An example screen 502 that can be presented on the monitoring device 110 of the security system 100 is shown. As shown, the monitoring device 110 can be operated by the monitoring agent 112, and the screen 502 can include a set of windows 506 corresponding to the respective events currently in the queue of the monitoring agent for review. In some embodiments, for example, each window 506 can be configured to playback a recorded video corresponding to the respective event detected at the respective monitoring location 104. As used herein, a "monitoring location" can correspond to a specific location monitored for security or other purposes (e.g., a house and associated lot, office space, parking lot, etc.). A given monitoring location 104 can be associated with a particular customer, for example, via a customer ID. In some embodiments, the recorded video can be configured and / or played back at an increased rate (e.g., twice the standard speed) at least initially to increase the rate at which the monitoring agent 112 can review the video for potential threats or objects of interest.

[0052] As Figure 5 shown, in some configurations, the screen 502 can include a control interface 508 that includes one or more user interface (UI) elements to allow the monitoring agent 112 to control various aspects of the agent's queue, such as the maximum number of notifications that can be added to the agent's queue for presentation in the respective windows 506. In some embodiments, notifications regarding events can be distributed to the respective monitoring agents 112 included in a pool of available monitoring agents 112 such that all available monitoring agents 112 have approximately the same number of events / notifications in their review queues at a given time.

[0053] When reviewing a window 506, for example, by viewing a recorded video clip corresponding to the detected motion, the monitoring agent 112 may determine that there is no potential security threat and provide an indication to the monitoring application 126 to remove from the agent's review queue a notification indicating the event, thereby freeing the corresponding window 506 to display another event / notification. Such an input may, for example, involve selecting (e.g., clicking) the close element 512 of the window 506.

[0054] Alternatively, when reviewing a window 506, for example, by viewing a recorded video clip corresponding to the detected motion, the monitoring agent 112 may determine that there is a potential threat or other security issue (referred to herein as an "incident") and determine that further review is needed. In such a case, the monitoring agent 112 may click or otherwise select the window 506 that displays the video of the mentioned record. In response to such a selection, the monitoring device 110 may begin receiving live video and / or audio transmitted from one or more cameras at the monitoring location 104 (e.g., video and / or audio corresponding to the currently occurring event), and / or the monitoring agent 112 may otherwise provide additional data (e.g., recorded video and / or audio, still images, sensor data, artificial intelligence (AI) assessment results, etc. from the edge image processing component 120 and / or the remote image processing component 122) such that the monitoring agent 112 can evaluate whether the incident presents an actual security risk. In some embodiments, for example, one or more peer-to-peer connections may be established between one or more cameras 102 at the monitoring location 104 and the monitoring device 110 using the Web Real-Time Communication (WebRTC) function of a browser on the monitoring device 110 to enable the transmission of video data and / or audio data between the cameras 102 and the monitoring device 110.

[0055] Figure 6 An example screen 602 that may be presented by the monitoring device 110 in response to a selection of Figure 5 one of the windows 506 shown is shown. In the illustrated example, the screen 602 includes three video windows 604 configured to display transmitted video feeds from three different cameras 102 at the monitoring location 104 corresponding to the selected window 506. Although Figure 6It is not described in the text, but alternatively or additionally, controls can be provided to allow the monitoring agent 112 to listen to the transmitted audio from the corresponding camera 102 and speak into the microphone, so that one or more speakers of such a camera 102 output audio representing the voice of the monitoring agent. In the illustrated example, the screen 602 also includes a larger main viewer window 606, in which the transmitted video of a window 604 can be optionally played, thereby making it easier for the monitoring agent 112 to view the content of the video. In some embodiments, the monitoring agent 112 can cause the transmitted video from a particular camera 102 to be played in the main viewer window 606 by selecting the window 604 of that camera (e.g., by clicking on it).

[0056] As Figure 6 shown, in some embodiments, the monitoring application 126 can cause the screen 602 of the monitoring device 110 to present one or more other windows 608 containing additional information about the event. Such additional information can include, for example, image frames identified by the edge image processing component 120 and / or the remote image processing component 122 as including features of interest (e.g., motion, people, faces, identified faces, etc.), indications of the output from other sensors at the monitoring location 104 (e.g., door sensors, glass break sensors, motion sensors, smoke detectors, etc.), images of individuals authorized to be at the monitoring location 104, and the like. Further, as also illustrated, in some embodiments, the monitoring application 126 can cause the screen 602 of the monitoring device 110 to present a scroll bar 610 that the monitoring agent 112 can manipulate, for example, to access and review other windows 608 that cannot fit within the display area of the monitoring device 110.

[0057] The monitoring agent 112 can take appropriate actions based on the review of the live video and / or audio from the camera 102. If the monitoring agent 112 determines that there is no security issue, the monitoring agent 112 can cancel the event notification (e.g., by clicking or otherwise selecting a "Cancel" button - not described), thereby removing it from the agent's queue.

[0058] On the other hand, if the monitoring agent 112 continues to believe that there may be a threat or other security issue based on the review of the live video and / or audio from the camera 102, the monitoring agent 112 can instead determine to continue evaluating the event, such as by communicating verbally with one or more individuals at the monitoring location 104, e.g., via the speaker on the camera 102.

[0059] When the monitoring agent 112 further reviews, interacts with one or more individuals at the monitoring location 104, etc., the monitoring agent 112 can determine the disposition of the event and may take one or more remedial measures, such as dispatching the police or fire department to the monitoring location 104.

[0060] Further, in some embodiments, the monitoring application 126 may prompt the monitoring agent 112 to send one or more follow-up communications (e.g., emails, push notifications, text messages, etc.) to the customer 116 that describe the event and its disposition. In some embodiments, the monitoring application 126 may additionally prompt the monitoring agent 112 to select one or more key frames that include features identified by the edge image processing component 120 and / or the remote image processing component 122 (e.g., by using a toggle switch to select such items in the window 608), and may append an indication of the selected frames and features to the notification sent to the customer 116.

[0061] In some embodiments, the monitoring application 126 may also present user interface elements (e.g., a toggle switch) that allow the monitoring agent 112 to mark frames identified by the edge image processing component 120 and / or the remote image processing component 122 as having incorrect or inaccurate feature recognition, and the data thus collected can subsequently be used to retrain the edge image processing component 120 and / or the remote image processing component 122.

[0062] Reference Figure 3 , the settings table 302 may include configuration data for each customer 116 indexed, for example, by the customer ID 304. As illustrated, the settings table 302 may include monitoring preferences 306, system status indicators 308, visitor document identifiers 310, biometric settings 312, and the like.

[0063] The monitoring preferences 306 may represent various settings for controlling the manner in which the security system 100 performs monitoring, such as the property to be monitored, the cameras 102 to be used to perform such monitoring, the type of monitoring to be performed, the times of day at which such monitoring is to occur, and so on. As explained below in connection with Figure 8 , in some embodiments, when determining whether to generate a biometric embedding in a particular situation (e.g., at a particular time of day), the edge image processing component 120 and / or the remote image processing component 122 may refer to the monitoring preferences.

[0064] The system status indicator 308 may identify the current state of the security system 100, such as whether the system is "armed", "disarmed", or "off". As explained below in connection with Figure 8 , in some embodiments, when determining whether to generate a biometric embedding in a particular situation (such as by determining to perform biometric face recognition only when the system is in the "armed" state), the edge image processing component 120 and / or the remote image processing component 122 may refer to the system status indicator 308.

[0065] The identifier 310 can identify one or more visitor documents in the document table 402. As indicated, in some embodiments, such visitor documents can be identified by listing the person ID 404 used to index these visitor documents in the document table 402.

[0066] The biometric setting 312 can be used to indicate whether biometric face recognition is enabled for one or more properties of the customer 116. An example process for determining the biometric setting for a given property is described below in Figure 7A connection with

[0067] reference Figure 4 , the table 402 can include document data of various individuals, which can be used by the edge image processing component 120 and / or the remote image processing component 122 to perform biometric face recognition, or can be used by the monitoring application 126 to allow the monitoring agent 112 to view images of individuals who have authorized or unauthorized access to a specific monitoring location 104. As previously mentioned, the table 402 can be indexed using the corresponding person ID 404. As illustrated, the table 402 can include the name 406 of the individual for whom the visitor document was created, the authorized location ID 408 identifying one or more locations authorized for the individual to access, the unauthorized location ID 410 identifying one or more locations unauthorized for the individual to access, one or more images 412 of the individual's face, and the biometric embedding 414 generated for one or more of the images 412, etc.

[0068] Figure 7A An example process 700 is shown, which can be executed by one or more components (e.g., the terminal application 128) of the security system 100 to determine the biometric setting 312 for a specific monitoring location 104 in the setting table 302 (see Figure 3 ).

[0069] As shown, the process 700 can start at step 702, where the terminal application 128 can determine the geographical location of the property to be monitored. As described below, the application 128 can then use the determined geographical location to apply one or more rules or guidelines related to the use of biometric face recognition at that location (e.g., using a rule engine, logic component, or other). The geographical location can be determined, for example, based on the area improvement plan (ZIP) code, city, and / or state identified by the terminal device 114 during the setup process, and / or using location data determined using the global positioning system (GPS) component of the terminal device 114.

[0070] At decision 704 of process 700, the terminal application 128 may apply a rule to determine whether biometric face recognition is permitted in the geographic region determined at step 702. In some embodiments, for example, the terminal application 128 may consult a database (not shown) that identifies geographic locations (e.g., using ZIP code, city, state, and / or GPS coordinates) where biometric face recognition is prohibited to determine whether the determined geographic region is identified in the database. Alternatively, the terminal application 128 may consult a database (not shown) that identifies geographic locations (e.g., using ZIP code, city, state, and / or GPS coordinates) where biometric face recognition is permitted to determine whether the determined geographic region is identified in the database. In either case, the consulted database may be updated periodically or occasionally (e.g., by a system administrator or using an automated process) to ensure that it accurately identifies geographic locations where biometric face recognition is prohibited or permitted. In some embodiments, the database consulted to make decision 704 may be arranged in the monitoring center environment described in connection with Figure 10 and Figure 15 described.

[0071] When at decision 704 the terminal application 128 determines that biometric face recognition is not permitted in the geographic region, process 700 may proceed to step 714, where the terminal application 128 may set the biometric setting 312 of the monitoring location 104 to "disabled". In some embodiments, for example, the terminal application 128 may be configured to call an API of a component that manages the data store 132 to selectively set the biometric setting 312 in the setting table 302 to "enabled" or "disabled".

[0072] When, at decision 704, the terminal application 128 determines that biometric face recognition is permitted in the geographic region, process 700 can alternatively proceed to decision 706, at which the terminal application 128 can apply another rule to determine whether the customer 116 needs to opt in to using biometric face recognition before the technology can be deployed at the monitoring location 104, as determined at step 702. Similar to decision 704, in some embodiments, the terminal application 128 can make such a determination by consulting a (not shown) database that identifies the geographic location (e.g., using ZIP code, city, state, and / or GPS coordinates) at which a customer opt-in consent is required to determine whether the identified geographic region is recognized in the database. Alternatively, the terminal application 128 can consult a (not shown) database that identifies the geographic location (e.g., using ZIP code, city, state, and / or GPS coordinates) at which a customer opt-in consent is not required to determine whether the identified geographic region is recognized in the database. In either case, the consulted database can be updated periodically or occasionally (e.g., by a system administrator or using an automated process) to ensure that it accurately identifies the geographic locations at which opt-in consent is required or not required. In some embodiments, the database consulted to make decision 706 can be arranged in the monitoring center environment described in conjunction with Figure 10 and Figure 15 described.

[0073] When, at decision 706, the terminal application 128 determines that the geographic location does not require the customer 116 to opt in to using biometric face recognition before the technology can be deployed at the monitoring location 104, process 700 can proceed to step 708, at which the terminal application 128 can pre-set the biometric setting 312 for the monitoring location 104 (in the settings table 302) to “enabled”. After step 708, process 700 can proceed to step 710, at which the terminal application 128 can present an opt-out screen to provide the customer 116 with an option to opt out of using the face recognition technology. Figure 7B An example of an opt-out screen 730 that can be presented on the terminal device 114 via the terminal application 128 is shown in Figure 7B As shown, the screen 730 can include user interface elements (e.g., a toggle switch 732) that indicate that biometric face recognition is enabled for the monitoring location 104, but that can be manipulated to disable biometric face recognition for the monitoring location 104, e.g., by changing the state of the toggle switch 732 or otherwise interacting with one or more user interface elements (e.g., radio buttons, etc.).

[0074] When, at decision 712, the terminal application 128 determines that an opt-out input has been received (e.g., by determining that the customer 116 has changed the state of the toggle switch 732), process 700 can proceed to step 714, where the terminal application 128 can set the biometric setting 312 for the monitoring location 104 to "disabled", e.g., by making an API call to the monitoring service 106 to change the value of the biometric setting 312 for the monitoring location 104 in the settings table 302 of the data store 132 to "disabled".

[0075] When, at decision 706, the terminal application 128 determines that the geographical location requires the customer 116 to opt-in to using biometric facial recognition before the technology can be deployed at the monitoring location 104, process 700 can alternatively proceed to step 716, where the terminal application 128 can pre-set the biometric setting 312 for the monitoring location 104 (in the settings table 302) to "disabled", e.g., by making an API call to the monitoring service 106 to change the value of the biometric setting 312 for the monitoring location 104 in the settings table 302 of the data store 132 to "disabled".

[0076] After step 716, process 700 can proceed to step 718, where the terminal application 128 can present an opt-in screen to provide the customer 116 with the option to opt-in to using the facial recognition technology. Such an opt-in screen can be similar to the above-described opt-out screen 730, except that the toggle switch 732 can be in the opposite position, indicating that biometric facial recognition is not enabled for the monitoring location 104. If desired, the customer 116 can enable the use of biometric facial recognition for the monitoring location 104, e.g., by changing the state of the toggle switch 732.

[0077] When, at decision 720, the terminal application 128 determines that an opt-in input has been received (e.g., by detecting a state change indicating that the customer 116 has changed the position of the toggle switch 732), process 700 can proceed to step 722, where the terminal application 128 can set the biometric setting 312 for the monitoring location 104 to "enabled", e.g., by making an API call to the monitoring service 106 to change the value of the biometric setting 312 for the monitoring location 104 in the settings table 302 of the data store 132 to "enabled".

[0078] Figure 8 An example process 800 is shown, which can be performed by the edge image processing component 120 and / or the remote image processing component 122 ( Figure 1as shown) is performed to determine whether to generate and / or store a biometric embedding of image data acquired by the image sensor 118 of the camera 102 at the monitoring location 104.

[0079] As shown, when the edge image processing component 120 and / or the remote image processing component 122 receive image data from the monitoring location 104, the process can start at step 802.

[0080] At decision 804, the edge image processing component 120 and / or the remote image processing component 122 can determine whether biometric processing has been enabled for the monitoring location 104. The edge image processing component 120 and / or the remote image processing component 122 can make such a determination, for example, by determining whether the biometric setting 312 of the monitoring location 104 (in the setting table 302) is "enabled" (e.g., by making an API call to the monitoring service 106 to read the value of the biometric setting 312 of the monitoring location 104 from the setting table 302 of the data storage 132).

[0081] When at decision 804 the edge image processing component 120 and / or the remote image processing component 122 determine that biometric processing has not been enabled for the monitoring location 104, the process 800 can proceed to step 822, where the edge image processing component 120 and / or the remote image processing component 122 can avoid determining the biometric embedding of the image data, and the process 800 can then terminate.

[0082] On the other hand, when the edge image processing component 120 and / or the remote image processing component 122 determine (at decision 804) that biometric processing has been enabled for the monitoring location 104, the process 800 can alternatively proceed to decision 806, where the edge image processing component 120 and / or the remote image processing component 122 can determine whether the security system 100 at the monitoring location 104 is in an "armed" state. The edge image processing component 120 and / or the remote image processing component 122 can make such a determination, for example, by determining whether the system status 308 of the monitoring location 104 (in the setting table 302) is "armed" (e.g., by making an API call to the monitoring service 106 to read the value of the system status 308 of the monitoring location 104 from the setting table 302 of the data storage 132).

[0083] When at decision 806 the edge image processing component 120 and / or the remote image processing component 122 determine that the security system 100 at the monitoring location 104 is not in an "armed" state, the process 800 can proceed to step 822, where the edge image processing component 120 and / or the remote image processing component 122 can avoid determining the biometric embedding of the image data, and the process 800 can then terminate.

[0084] On the other hand, when the edge image processing component 120 and / or the remote image processing component 122 determine (at decision 806) that the security system 100 at the monitoring location 104 is in the "alert" state, the process 800 can alternatively proceed to decision 808, at which the edge image processing component 120 and / or the remote image processing component 122 can determine whether image data is received during a time window or in another situation where the security system 100 has been enabled to perform face recognition processing. The edge image processing component 120 and / or the remote image processing component 122 can make such a determination, for example, by evaluating the monitoring preference 306 (in the settings table 302) to determine whether the security system 100 has actually been enabled to perform face recognition processing at the mentioned time or in the mentioned situation. For example, if the monitoring preference 306 indicates that face recognition processing will only be performed between 8:00 PM and 6:00 AM, and image data is received outside of these times, the edge image processing component 120 and / or the remote image processing component 122 can determine that no image data has been received during the time window when the security system 100 has been enabled to perform face recognition processing.

[0085] When at decision 808 the edge image processing component 120 and / or the remote image processing component 122 determine that no image data has been received during the time window or in another situation where the security system 100 has been enabled to perform face recognition processing, the process 800 can proceed to step 822, at which the edge image processing component 120 and / or the remote image processing component 122 can avoid determining the biometric embedding of the image data, and the process 800 can then terminate.

[0086] On the other hand, when the edge image processing component 120 and / or the remote image processing component 122 determines (at decision 808) that image data is received during the time window or in another case where the security system 100 has been enabled to perform face recognition processing, process 800 can alternatively proceed to decision 810, where the edge image processing component 120 and / or the remote image processing component 122 can determine whether a person (or face) has been detected within a threshold distance (e.g., five meters) of the camera 102. The edge image processing component 120 and / or the remote image processing component 122 can make such a determination, for example, by using one or more trained machine learning (ML) models or other components configured to estimate the distance between the detected person (or detected face) in the image and the camera 102 used to acquire the image. In some embodiments, for example, an improved YOLO model (such as the model described in "Dist-YOLO: Fast Object Detection with Distance Estimation", M. Vajgl et al., Appl. Sci. 2022, 12(3), 1354 (2022)) can be used for this purpose.

[0087] When, at decision 810, the edge image processing component 120 and / or the remote image processing component 122 determines that a person (or face) has not been detected within the threshold distance of the camera 102, process 800 can proceed to step 822, where the edge image processing component 120 and / or the remote image processing component 122 can avoid determining the biometric embedding of the image data, and process 800 can then terminate.

[0088] On the other hand, when the edge image processing component 120 and / or the remote image processing component 122 determines (at decision 810) that a person (or face) has been detected within the threshold distance of the camera 102, process 800 can alternatively proceed to decision 812, where the edge image processing component 120 and / or the remote image processing component 122 can determine whether the detected person (or detected face) is within the boundaries of the mentioned property. The edge image processing component 120 and / or the remote image processing component 122 can make such a determination, for example, by using automatic image segmentation (e.g., using a semantic segmentation model or technique) and / or customer input to establish the position of the property line relative to the field of view of the camera 102, and using one or more trained machine learning (ML) models or other components to determine whether the detected person (or face) is within these property lines. Examples of models that can be used for object detection and segmentation include Mask R-CNN (which can be accessed, for example, via the URL "github.com").

[0089] When the Edge Image Processing Component 120 and / or the Remote Image Processing Component 122 determines that the detected person (or detected face) is not within the boundaries of the property at decision 812, process 800 can proceed to step 822, where the Edge Image Processing Component 120 and / or the Remote Image Processing Component 122 can avoid determining a biometric embedding of the image data, and process 800 can then terminate.

[0090] On the other hand, when the Edge Image Processing Component 120 and / or the Remote Image Processing Component 122 determines (at decision 812) that the detected person (or detected face) is within the boundaries of the property, process 800 can alternatively proceed to decision 814, where the Edge Image Processing Component 120 and / or the Remote Image Processing Component 122 can determine whether the detected person is older than a threshold age. Examples of models that can be used for age estimation include MiVOLO and CORAL (which can be accessed, for example, via the URL "github.com"). In some embodiments, the threshold age can depend on the geographical location and can be indicated, for example, in the biometric settings 312 or elsewhere. For example, the threshold age for one geographical location can be thirteen, and the threshold age for another geographical location can be eighteen. In some embodiments, the threshold age can be set to be a few years older than the threshold age specified by law or regulation to provide some leeway to account for inaccuracies in the age estimation model.

[0091] When the Edge Image Processing Component 120 and / or the Remote Image Processing Component 122 determines at decision 814 that the detected person is not older than the threshold age, process 800 can proceed to step 822, where the Edge Image Processing Component 120 and / or the Remote Image Processing Component 122 can avoid determining a biometric embedding of the image data, and process 800 can then terminate.

[0092] On the other hand, when the Edge Image Processing Component 120 and / or the Remote Image Processing Component 122 determines (at decision 814) that the detected person is older than the threshold age, process 800 can alternatively proceed to step 816, where the Edge Image Processing Component 120 and / or the Remote Image Processing Component 122 can determine a biometric embedding of the face of the detected person.

[0093] At step 818 of process 800, the Edge Image Processing Component 120 and / or the Remote Image Processing Component 122 can perform biometric face recognition using the embedding determined at step 816, for example, by comparing the determined embedding with the biometric embeddings 414 of one or more documents associated with the monitoring location 104.

[0094] Finally, at step 820, the edge image processing component 120 and / or the remote image processing component 122 may optionally store the biometric embedding determined at step 816 as, for example, a biometric embedding 216 within an event table 202. As described above, storing the biometric embedding 216 of the acquired image in this case may be useful, for example, as ground truth data for subsequently training or retraining one or more image processing modules for face recognition (e.g., in accordance with step 818). In some embodiments, the biometric embedding determined at step 816 may be stored (e.g., within the event table 202) only if a matching biometric embedding 414 is found in the visitor document. Further, as also described above, it should be understood that in some embodiments, these biometric embeddings may be stored (in accordance with step 820) in the event table 202, for example, before performing biometric face recognition (in accordance with step 818) using the biometric embeddings determined at step 816. In other embodiments, the edge image processing component 120 and / or the remote image processing component 122 may discard the determined biometric embeddings after comparing them with the biometric embeddings 414 from the document associated with the monitored location (e.g., in accordance with step 818), thereby avoiding storing them in the event table 202 altogether.

[0095] Figure 9A An example process 900 is shown that may be performed by one or more components of the monitoring service 106 (e.g., the remote image processing component 122 or another component) to create a document for an individual (e.g., a visitor document).

[0096] As shown, when the monitoring service 106 receives a request (e.g., from the terminal application 128) to create a visitor document for a particular individual, the process 900 may begin at step 902. In some embodiments, the terminal application 128 may present a screen 930 (see Figure 9B ) to the customer 116 that enables the customer 116 to specify information for the new visitor document, and the request at step 902 may correspond to submitting this information to the monitoring service 106 for processing (e.g., in response to the customer selecting the "Save" user interface element 938 on the screen 930).

[0097] The screen 930 (or a similar UI mechanism) can be presented in response to any one of a plurality of user inputs. For example, in some embodiments, the customer 116 can select the "Create New Document" option from a menu of available options. In other embodiments, the customer 116 can additionally or alternatively provide a specific input (e.g., press and hold) to select a facial image presented via the terminal application 128, e.g., as an element of an event notification displayed by the terminal device 114, and the terminal application 128 can present the screen 930 (or a similar UI mechanism) or access the UI elements of such a screen in response to such an input. In yet other embodiments, in response to the customer 116 uploading one or more images of the face from the terminal device 114 to the data storage 132 and / or the terminal application 128, the terminal application 128 can present the screen 930 (or a similar UI mechanism) or the UI elements for accessing such a screen. Other mechanisms for accessing the screen 930 (or a similar UI mechanism) are also possible.

[0098] In step 904 of process 900, the monitoring service 106 can determine the name of the individual for whom a new document is to be created. The name can be determined, for example, based on the text entered into the "Name" field 932 on the screen 930 or in other ways of determining the name of the individual.

[0099] In step 906 of process 900, the monitoring service 106 can determine one or more permissions for the individual, such as whether the individual authorizes access to one or more monitoring locations or may not be authorized to access such a property. For example, such permissions can be determined based on the selections made within the "Permissions" area 934 of the screen 930.

[0100] In step 908 of process 900, the monitoring service 106 can determine one or more facial images of the individual. For example, such facial images can be determined based on the images selected by the customer 116 using the "Saved Clips" area 936 of the screen 930. In some embodiments, the terminal application 128 can enable the customer 116 to select images only from one or more specific sources (e.g., from the camera roll of the terminal device 114) to minimize the risk that the customer 116 inadvertently creates a document of the individual without obtaining the individual's consent, e.g., because the customer 116 may actually know and be able to obtain consent from the individuals appearing in the photos from such sources.

[0101] In step 910 of process 900, the monitoring service 106 can create a document that includes the name determined in step 904, the permissions determined in step 906, and the images determined in step 908. The data of the document thus created can be stored, for example, in Figure 4In a new row of Table 402 as shown. In particular, in Table 402, the indicated name can be stored as Name 906, the indicated permission can be stored as Authorized Location ID 408 and / or Unauthorized Location ID 410, and an image of the face of the indicated individual can be stored as Image 412.

[0102] In step 912 of process 900, in some embodiments, the monitoring service 106 can determine a unique identifier (e.g., email address, phone number, user identifier, etc.) of the individual for whom the document is being created. Although not shown in Figure 9B it should be understood that in some embodiments, the terminal application 128 can additionally provide an input field or other mechanism on the screen 930 that enables the customer 116 to input or select (e.g., from the contact list or address book of the terminal device 114) the contact information of the mentioned individual, or can automatically retrieve such contact information from the contact list or address book of the terminal device 114 or obtain such contact information in other ways.

[0103] In step 914 of process 900, the monitoring service 106 can communicate with (e.g., send an email, text message, push notification, etc.) the individual for whom the document is being created to request the individual's consent to use an image of the individual's face to perform biometric face recognition. Such communication can indicate, for example, (1) that the customer 116 initiated the request for consent, (2) the scope of the requested consent, (3) the mechanism or process for revoking consent (if desired), and (4) how to provide consent, such as by typing a name, clicking one or more buttons, etc.

[0104] In decision 916 of the process, the monitoring service 106 can determine whether the individual has provided the requested consent, for example, by receiving an indication that the customer 116 has selected a user interface element in the communication (e.g., email message, text message, push notification, etc.) or in a web page or application screen (e.g., the screen of the terminal application 128) accessed via a link in such communication.

[0105] When in decision 916 the monitoring service 106 determines that the necessary consent has been received from the individual (e.g., by receiving an indication that the customer 116 has typed their name or otherwise taken one or more steps to indicate consent), the process can proceed to step 918, where the monitoring service 106 can generate a biometric embedding of one or more of the face images determined in step 908.

[0106] In step 920 of process 900, the monitoring service 106 can store the biometric embeddings determined in step 918, for example, by writing them as the individual's biometric embeddings 414 into the document table 402.

[0107] When the monitoring service 106 determines at decision 916 that the necessary consent has not been received from the individual, the process may instead proceed to step 922 where the monitoring service 106 may refrain from generating a biometric embedding of the facial image determined at step 908. In some such embodiments, as previously mentioned, the monitoring application 126 may still have access to the facial image stored at step 908 and other information added to the visitor document to enable the monitoring agent 112 to visually review the facial image and other information from the document while reviewing recorded or live video of an event at the monitoring location 104 to determine whether a particular individual is an authorized visitor.

[0108] In some embodiments, terminal application 128 can facilitate deleting some or all biometric embeddings that have been stored in data storage 132, such as biometric embedding 216 in event table 202 and / or biometric embedding 414 in document table 402. For example, in response to customer 116 Figure 7B When the illustrated toggle switch 732 is moved to the “off” position, the terminal application 128 may present a prompt asking the customer 116 to indicate whether certain previously acquired biometric data should be deleted from the security system 100. In some embodiments, such a prompt may explain that even if deletion of the biometric data is requested, an image of the individual's face will be maintained in storage, e.g., as image data 212 in the event table 202 and / or one or more images 412 in the document table 402. Again, maintaining the facial image in this way may, for example, enable the monitoring agent 112 to visually review the facial image and other information from the document when reviewing a recorded or live video of an event at the monitoring location 104 to determine whether a particular individual is an authorized visitor. Figure 10 is a schematic diagram of an example security system 1000 that can be used with various aspects of the present invention. As shown, in some embodiments, the security system 1000 may include multiple monitoring locations 104 (in Figure 10 only one of which is illustrated in ), monitoring center environment 1022, monitoring center environment 1026, one or more terminal devices 114, and one or more communication networks 1020. Monitoring location 104, monitoring center environment 1022, monitoring center environment 1026, one or more terminal devices 114, and communication network 1020 can each include one or more computing devices (e.g., as described below with reference to Figure 11as described). The terminal device 114 may include one or more terminal applications 128, such as applications hosted on or otherwise accessible by the terminal device 114. In some embodiments, the terminal applications 128 may be embodied as web applications that may be accessed via a browser of the terminal device 114. The monitoring center environment 1022 may include one or more monitoring applications 126, such as applications hosted on or otherwise accessible by computing devices within the monitoring center environment 1022. In some embodiments, the monitoring applications 126 may be embodied as web applications that can be accessed via a browser of a computing device operated by the monitoring agent 112 within the monitoring center environment 1022. The monitoring center environment 1026 may include a monitoring service 1030 and one or more transmission services 1028.

[0109] As Figure 10 shown, the monitoring location 104 may include one or more image capture devices (e.g., cameras 102A and 102B), one or more contact sensor assemblies (e.g., contact sensor assembly 1006), one or more keypads (e.g., keypad 1008), one or more motion sensor assemblies (e.g., motion sensor assembly 1010), a base station 1012, and a router 1014. As illustrated, the base station 1012 may host a monitoring client 1016.

[0110] In some embodiments, the router 1014 may be a wireless router configured to communicate with devices (e.g., devices 102A, 102B, 1006, 1008, 1010, and 1012) disposed at the monitoring location 104 via communication consistent with any of the communication standards such as those in the various Institute of Electrical and Electronics Engineers (IEEE) 802.11 standards. As Figure 10 illustrated, the router 1014 may also be configured to communicate with the network 1020. In some embodiments, the router 1014 may implement a local area network (LAN) within or near the monitoring location 104. In other embodiments, other types of networking technologies may be additionally or alternatively used within the monitoring location 104. For example, in some embodiments, the base station 1012 may receive and forward communication packets sent by one or both of the cameras 102A, 102B via a point-to-point personal area network (PAN) protocol such as Bluetooth. Other suitable wired, wireless, and mesh network technologies and topologies will be apparent in light of the present invention and are intended to fall within the scope of the examples disclosed herein.

[0111] Network 1020 may include one or more public and / or private networks that support, for example, Internet Protocol (IP) communications. Network 1020 may include, for example, one or more LANs, one or more PANs, and / or one or more wide area networks (WANs). LANs that may be employed include wired or wireless networks that support various LAN standards such as IEEE 108.11 versions and the like. PANs that may be employed include wired or wireless networks that support various PAN standards such as Bluetooth, ZigBee, and the like. WANs that may be employed include wired or wireless networks that support various WAN standards such as Code Division Multiple Access (CMDA), Global System for Mobile Communications (GSM), and the like. Regardless of the specific networking technology employed, network 1020 may connect components within monitoring location 104, monitoring center environment 1022, surveillance center environment 1026, and terminal device 114 and enable data communication therebetween. In at least some embodiments, both monitoring center environment 1022 and surveillance center environment 1026 may include networking components (e.g., similar to router 1014) configured to communicate with network 1020 and various computing devices within these environments.

[0112] Surveillance center environment 1026 may include physical space, communication, cooling, and power infrastructure to support the networking operations of a large number of computing devices. For example, the infrastructure of surveillance center environment 1026 may include rack space into which computing devices may be installed, uninterruptible power supplies, cooling plenums and equipment, and networking equipment. Surveillance center environment 1026 may be dedicated to security system 1000, may be a non-dedicated, commercially available cloud computing service (e.g., Microsoft Azure, Amazon Web Services, Google Cloud, etc.), or may include a hybrid configuration consisting of dedicated and non-dedicated resources. Regardless of its physical or logical configuration, as Figure 10 shown, surveillance center environment 1026 may be configured to host surveillance service 1030 and transmission service 1028.

[0113] Monitoring center environment 1022 may include multiple computing devices (e.g., desktop computers) and network equipment (e.g., one or more routers) that enable communication between the computing devices and network 1020. Terminal devices 114 may each include personal computing devices (e.g., desktop computers, laptop computers, tablet computers, smart phones, etc.) and network equipment (e.g., routers, cellular modems, cellular radio transceivers, etc.). As Figure 10 illustrated, monitoring center environment 1022 may be configured to host monitoring application 126, and terminal devices 114 may be configured to host terminal application 128.

[0114] Devices 102A, 102B, 1006, and 1010 can be configured to acquire analog signals via sensors incorporated into the devices, generate digital sensor data based on the acquired signals, and transmit the sensor data to a base station 1012 and / or one or more components within a surveillance center environment 1026 (e.g., the aforementioned remote image processing component 122) (e.g., via a wireless link with a router 1014). The type of sensor data generated and transmitted by these devices can vary depending on the characteristics of the sensors they include. For example, image capture devices or cameras 102A and 102B can acquire ambient light, generate one or more frames of image data based on the acquired light, and transmit the frames to a base station 1012 and / or one or more components within a surveillance center environment 1026, although the pixel resolution and frame rate can vary depending on the capabilities of the devices. In some embodiments, cameras 102A and 102B can also receive and store filter zone configuration data and use one or more filter zones (e.g., regions within the camera's FOV from which image data is redacted for various reasons such as to exclude trees that may generate false positive motion detection results on windy days) to filter the frames before transmitting the frames to a base station 1012 and / or one or more components within a surveillance center environment 1026. In Figure 10 In the example shown, camera 102A has a field of view (FOV) starting near the front door at the monitoring location 104 and can acquire images of a sidewalk 1036, a road 1038, and the space between the monitoring location 104 and a road 64A0. On the other hand, camera 102B has a FOV starting near the bathroom at the monitoring location 104 and can acquire images of the living room and dining area at the monitoring location 104. Camera 102B can also acquire images of outdoor areas outside the monitoring location 104, e.g., through windows 1118A and 1118B on the right hand side of the monitoring location 104.

[0115] Each sensor assembly deployed at the monitoring location 104 (e.g., Figure 10 the contact sensor assembly 1006 shown) can include, for example, a sensor that can detect the presence of a magnetic field generated by a magnet when the magnet approaches the sensor. When a magnetic field is present, the contact sensor assembly 1006 can generate boolean sensor data specifying the closed state of a window, door, etc. When no magnetic field is present, the contact sensor assembly 1006 can instead generate boolean sensor data specifying the open state of a window, door, etc. In either case, Figure 10 the contact sensor assembly 1006 shown can transmit sensor data indicating whether the front door at the monitoring location 104 is open or closed to the base station 1012.

[0116] Each motion sensor assembly deployed at the monitoring location 104 (e.g., Figure 10The illustrated motion sensor assembly 1010 can include, for example, a component that can emit high-frequency pressure waves (such as ultrasonic waves) and a sensor that can acquire the reflections of the emitted waves. When the sensor detects a change in the reflected pressure wave, for example, because one or more objects are moving within the space monitored by the sensor, the motion sensor assembly 1010 can generate boolean sensor data specifying an alarm state. When the sensor does not detect a change in the reflected pressure wave, for example, because no objects are moving within the monitored space, the motion sensor assembly 1010 can instead generate boolean sensor data specifying a stationary state. In either case, the motion sensor assembly 1010 can transmit the sensor data to the base station 1012. It should be noted that the above specific sensing modality is not limited to the present invention. For example, merely as an example of an alternative embodiment, the motion sensor assembly 1010 can instead (or additionally) operate based on the detection of changes in reflected electromagnetic waves.

[0117] Although the above describes a specific type of sensor, it should be understood that other types of sensors can be additionally or alternatively employed within the monitoring location 104 to detect the presence and / or movement of a person, or other conditions of interest, such as smoke, elevated carbon dioxide levels, water accumulation, etc., and transmit data indicating these conditions to the base station 1012. For example, although not illustrated in Figure 10 some embodiments, one or more sensors can be employed to detect sudden changes in measured temperature, sudden changes in incident infrared radiation, sudden changes in incident pressure waves (such as, acoustic waves), etc. Still further, in some embodiments, some such sensors and / or the base station 1012 can be additionally or alternatively configured to identify specific signal curves indicative of specific conditions, such as sound curves indicative of glass breakage, footsteps, coughing, etc.

[0118] Figure 10The keypad 1008 shown can be configured to interact with a user and interoperate with other devices arranged at the monitoring location 104 in response to such interaction. For example, in some examples, the keypad 1008 can be configured to receive input from a user specifying one or more commands and transmit the specified commands to one or more addressed devices and / or processes, such as one or more devices arranged at the monitoring location 104, the monitoring application 126, and / or the monitoring service 1030. The transmitted commands can include, for example, codes for authenticating the user as a resident of the monitoring location 104 and / or codes for requesting activation or deactivation of one or more devices arranged at the monitoring location 104. In some embodiments, the keypad 1008 can include a user interface (e.g., a tactile interface, such as a set of physical buttons or a set of "soft" buttons on a touch screen) configured to interact with a user (e.g., receive input from the user and / or present output to the user). Further, in some embodiments, the keypad 1008 can receive a response to the transmitted commands and present such response as a visual or audio output via the user interface.

[0119] Figure 10 The base station 1012 shown can be configured to interoperate with other security system devices arranged at the monitoring location 104 to provide local command and control and / or store-and-forward functionality via the execution of the monitoring client 1016. To implement the local command and control functionality, the base station 1012 can perform various programmed operations via the execution of the monitoring client 1016 in response to various events. Examples of such events include receiving a command from the keypad 1008, receiving a command from either the monitoring application 126 or the terminal application 128 via the network 1020, and detecting the occurrence of a scheduled event. The programmed operations performed by the base station 1012 in response to an event via the execution of the monitoring client 1016 can include, for example, activation or deactivation of one or more of the devices 102A, 102B, 1006, 1008, and 1010; sounding an alarm; reporting the event to the monitoring service 1030; and / or transmitting "location data" to one or more of the transmission services 1028. Such location data can include, for example, data specifying sensor readings (sensor data), image data acquired by one or more cameras 102, configuration data of one or more devices arranged at the monitoring location 104, commands input and received from a user (e.g., via the keypad 1008 or the terminal application 128), or data derived from one or more of the foregoing data types (e.g., filtered sensor data, filtered image data, a summary of sensor data, event data specifying an event detected at the monitoring location 104 via sensor data, etc.).

[0120] In some embodiments, to implement the store-and-forward function, base station 1012 can receive sensor data by monitoring the execution of client 1016, encapsulate the data for transmission, and store the encapsulated sensor data in local memory for subsequent transfer. Such transfer of the encapsulated sensor data can include, for example, transmitting the encapsulated sensor data as a payload of a message to one or more of transmission services 1028 when a communication link to transmission service 1028 via network 1020 is operable. In some embodiments, such encapsulation of the sensor data can include filtering the sensor data using one or more filter zones and / or generating one or more summaries (maximum value, average value, change in value since the previous transmitted value, etc.) of multiple sensor readings.

[0121] The transmission service 1028 that monitors the central environment 1026 can be configured to receive messages from monitoring locations (e.g., monitoring location 104), parse the messages to extract the payloads included therein, and store the payloads and / or data derived from the payloads in one or more data repositories hosted in the monitoring central environment 1026. Examples of such data repositories are described below in conjunction with Figure 11 In some embodiments, the transmission service 1028 can expose and implement one or more application programming interfaces (APIs) that are configured to receive, process, and respond to calls from base stations (e.g., base station 1012) via network 1020. Respective instances of the transmission service 1028 can be associated with certain manufacturers and / or models of location-based monitoring devices (e.g., SIMPLISAFE devices, RING devices, etc.) and be specific to those manufacturers and / or models.

[0122] The APIs of the transport service 1028 can be implemented using various architectural styles and interoperability standards. For example, in some implementations, one or more such APIs can include web service interfaces implemented using the Representational State Transfer (REST) architectural style. In such implementations, the Hypertext Transfer Protocol (HTTP) can be used in conjunction with JavaScript Object Notation (JSON) and / or Extensible Markup Language to encode API calls. Such API calls can be addressed to one or more Uniform Resource Locators (URLs) corresponding to the API endpoints monitored by the transport service 1028. In some implementations, portions of the HTTP communication can be encrypted to increase security. Alternatively (or additionally), in some implementations, one or more APIs of the transport service 1028 can be implemented as.NET web APIs in response to an HTTP data post to a specific URL. Alternatively (or additionally), in some implementations, one or more APIs of the transport service 1028 can be implemented using Simple File Transfer Protocol commands. Thus, the APIs of the transport service 1028 are not limited to any particular implementation.

[0123] The monitoring service 1030 in the monitoring center environment 1026 can be configured to control the overall logical settings and operations of the security system 1000. Thus, the monitoring service 1030 can communicate and interoperate with the transport service 1028, the monitoring application 126, the terminal application 128, and various devices arranged at the monitoring location 104 via the network 1020. In some implementations, the monitoring service 1030 can be configured to monitor data from various sources for events (e.g., break-in events), and when an event is detected, notify one or more of the monitoring application 126 and / or the terminal application 128 of the event.

[0124] In some implementations, the monitoring service 1030 can additionally be configured to maintain status information regarding the monitoring location 104. Such status information can indicate, for example, whether the monitoring location 104 is secure or threatened. In some implementations, the monitoring service 1030 can be configured to change the status information to indicate that the monitoring location 104 is secure only upon receipt of a communication indicating a cleared event (e.g., rather than making such a change solely due to the absence of detection of additional events). This feature can prevent a "smash and grab" theft (e.g., an intruder quickly disabling or destroying the monitoring equipment) from being successfully executed. Additionally, in some implementations, the monitoring service 1030 can be configured to monitor one or more specific areas within the monitoring location 104, such as one or more specific rooms or other distinct areas within and / or around the monitoring location 104 and / or the corresponding image capture devices deployed at the monitoring location (e.g., Figure 10One or more defined areas within the FOV of cameras 102A and 102B as shown.

[0125] The individual monitoring applications 126 of the monitoring center environment 1022 can be configured to enable monitoring personnel to interact with corresponding computing devices to provide monitoring services for corresponding locations (e.g., monitoring location 104), and to perform various programmed operations in response to such interactions. For example, in some embodiments, the monitoring application 126 can control its host computing device to provide information about events detected at a monitoring location (e.g., monitoring location 104) to the personnel operating the computing device. Such events can include, for example, detected movement within a specific area of ​​the monitoring location 104. As described above in conjunction with Figure 5 and Figure 6 As described above, in some embodiments, the monitoring application 126 can enable the monitoring device 110 to present the video of the event in each window 506 of the screen 502, and can also establish a transmission connection with one or more cameras 102 at the monitoring location, and enable the monitoring device 110 to provide streaming video from such camera 102 in the window 604 of the screen 602 and / or the main viewer window 606, and allow audio communication between the monitoring device 110 and the camera 102.

[0126] The terminal application 128 of the terminal device 114 can be configured to enable customers to interact with their computing devices (e.g., their smart phones or personal computers) to access various services provided by the security system 1000 for their personal home or other location (e.g., monitoring location 104), and to perform various programmed operations in response to such interactions. For example, in some embodiments, the terminal application 128 can control the terminal device 114 (e.g., smart phone or personal computer) to provide information about events detected at a monitoring location (such as monitoring location 104) to the customer 116 operating the terminal device 114. Such events can include, for example, detected movement within a specific area of ​​the monitoring location 104. In some embodiments, the terminal application 128 can be additionally or alternatively configured to process input received from the customer 116 to enable or disable one or more devices arranged in the monitoring location 104. Further, as described above in combination with Figure 6 As described above, the terminal application 128 may be additionally or alternatively configured to establish a transmission connection with one or more cameras 102 at the monitoring location, and enable the terminal device 114 to display streaming video from such cameras 102, and to allow audio communication between the terminal device 114 and the cameras 102.

[0127] Now turn Figure 11 , schematically illustrates a computing device 1100. As Figure 11 ​As shown, computing device 1100 may include at least one processor 1102, volatile memory 1104, one or more interfaces 1106, non-volatile memory 1108, and an interconnect mechanism 1114. Non-volatile memory 1108 may include executable code 1110, and as illustrated, may additionally include at least one data store 1112.

[0128] In some embodiments, non-volatile (non-transitory) memory 1108 may include one or more read-only memory (ROM) chips; one or more hard disk drives or other magnetic or optical storage media; one or more solid-state drives (SSDs), such as flash drives or other solid-state storage media; and / or one or more hybrid magnetic and SSDs. Further, in some embodiments, the code 1110 stored in non-volatile memory may include an operating system and one or more application programs or routines configured to execute under the control of the operating system. In some embodiments, the code 1110 may additionally or alternatively include proprietary firmware and embedded software that is executable without relying on a commercially available operating system. Regardless of its configuration, execution of the code 1110 may produce manipulated data that can be stored as one or more data structures in the data store 1112. The data structures may have fields that are associated by virtue of their positions within the data structure. Such association may also be achieved by allocating storage in positions within the memory for the association between the transported fields within the memory. However, other mechanisms may be used to establish associations between the information in the fields of the data structure, including by using pointers, tags, or other mechanisms.

[0129] The processor 1102 of computing device 1100 may be embodied by one or more processors configured to execute one or more executable instructions, such as a computer program specified by the code 1110, to control the operation of computing device 1100. The functions, operations, or order of operations may be hard-coded into the circuitry or soft-coded by instructions held in a memory device (e.g., volatile memory 1104) and executed by the circuitry. In some embodiments, the processor 1102 may be embodied by one or more application-specific integrated circuits (ASICs), microprocessors, digital signal processors (DSPs), graphics processing units (GPUs), neural processing units (NPUs), microcontrollers, field-programmable gate arrays (FPGAs), programmable logic arrays (PLAs), or multi-core processors.

[0130] Before executing code 1110, the processor 1102 may copy code 1110 from the non-volatile memory 1108 to the volatile memory 1104. In some embodiments, the volatile memory 1104 may include one or more static or dynamic random access memory (RAM) chips and / or cache memory (e.g., memory disposed on the silicon die of the processor 1102). The volatile memory 1104 may provide a faster response time than main memory such as the non-volatile memory 1108.

[0131] By executing code 1110, the processor 1102 may control the operation of the interface 1106. The interface 1106 may include a network interface. Such a network interface may include one or more physical interfaces (e.g., radio transceiver devices, Ethernet ports, USB ports, etc.) and a software stack that includes drivers and / or other code 1110 configured to communicate with the one or more physical interfaces to support one or more LAN, PAN, and / or WAN standard communication protocols. Such communication protocols may include, for example, TCP and UDP. Accordingly, the network interface may enable the computing device 1100 to access and communicate with other computing devices via a computer network.

[0132] The interface 1106 may include one or more user interfaces. For example, in some embodiments, the user interface 1106 may include user input and / or output devices (e.g., keyboard, mouse, touch screen, display, speaker, camera, accelerometer, biometric scanner, environmental sensor, etc.) and a software stack that includes drivers and / or other code 1110 configured to communicate with the user input and / or output devices. Accordingly, the user interface 1106 may enable the computing device 1100 to interact with a user to receive input and / or present output. The presented output may include, for example, one or more GUIs that include one or more controls configured to display output and / or receive input. The received input may specify values to be stored in the data storage 1112. The displayed output may indicate values stored in the data storage 1112.

[0133] The various features of the computing device 1100 described above may communicate with each other via an interconnect mechanism 1114. In some embodiments, the interconnect mechanism 1114 may include a communication bus.

[0134] Now turning to Figure 12 FIG., an example base station 1012 is schematically illustrated. As Figure 12As shown, the base station 1012 may include at least one processor 1202, volatile memory 1204, non-volatile memory 1208, at least one network interface 1206, user interface 1214, battery assembly 1216, and an interconnect mechanism 1218. The non-volatile memory 1208 may store executable code 1210 and, as illustrated, may also include a data storage 1212. In some embodiments, the features of the base station 1012 listed above may be contained within a housing 1220 or may otherwise be supported by it. In some embodiments, the user interface 1214 of the base station 1012 may include only: one or more speakers to provide an audio output to the user regarding changes in the operating state of the security system 1000, detected threats, etc.; and / or one or more visual indicators (e.g., light-emitting diode (LED) indicators) to indicate when the base station 1012 is operable in response to user input (e.g., via the keypad 1008), etc. In other embodiments, the user interface may additionally or alternatively include more complex output components (e.g., a display screen) and / or may include one or more user input components, such as one or more microphones (e.g., to receive voice commands) and / or a keypad (e.g., to receive tactile input).

[0135] In some embodiments, the non-volatile (non-transitory) memory 1208 may include one or more read-only memory (ROM) chips; one or more hard disk drives or other magnetic or optical storage media; one or more solid-state drives (SSDs), such as flash drives or other solid-state storage media; and / or one or more hybrid magnetic and SSDs. In some embodiments, the code 1210 stored in the non-volatile memory may include an operating system and one or more application programs or routines configured to execute under the control of the operating system. In some embodiments, the code 1210 may additionally or alternatively include proprietary firmware and embedded software that is executable without relying on a commercially available operating system. In any case, regardless of how the code 1210 is specifically implemented, the execution of the code 1210 may implement Figure 10 the monitoring client 1016 shown and enable the storage and manipulation of data of the monitoring client 1016 within the data storage 1212.

[0136] The processor 1202 of the base station 1012 may include one or more processors configured to execute instructions encoded in a computer-readable medium (such as a computer program implemented by the code 1210) to control the operation of the base station 1012. As used herein, the term "processor" describes a circuit that performs a function, operation, or sequence of operations. The function, operation, or sequence of operations may be hard-coded into the circuit or soft-coded by instructions maintained in a memory device (e.g., volatile memory 1204) and executed by the circuit. In some embodiments, the processor 1202 may be implemented by one or more application-specific integrated circuits (ASICs), microprocessors, digital signal processors (DSPs), graphics processing units (GPUs), neural processing units (NPUs), microcontrollers, field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), and / or multi-core processors.

[0137] Before executing code 1210, processor 1202 may copy at least a portion of code 1210 from non-volatile memory 1208 to volatile memory 1204. In some embodiments, volatile memory 1204 may include one or more static or dynamic random access memory (RAM) chips and / or cache memory (e.g., memory disposed on the silicon die of processor 1202). Volatile memory 1204 may provide a faster response time than main memory (such as non-volatile memory 1208).

[0138] By executing code 1210, processor 1202 may control the operation of network interface 1206. For example, in some embodiments, network interface 1206 may include one or more physical interfaces (e.g., radio transceiver, Ethernet port, universal serial bus (USB) port, etc.) and a software stack, the software stack including drivers and / or other code 1210, which is configured to communicate with one or more physical interfaces to support one or more LAN, PAN and / or WAN standard communication protocols. Such communication protocols may include, for example, transmission control protocol (TCP) and user datagram protocol (UDP), etc. Thus, network interface 1206 may enable base station 1012 to communicate via a computer network (e.g., by Figure 10 The LAN established by router 1014, Figure 10 network 1020 and / or peer-to-peer connections) to access other computing devices (e.g., arranged in Figure 10 Other devices in the monitoring location 104) and communicate with them. For example, in some embodiments, the network interface 1206 can use sub-GHz wireless networking to send a wake-up message to other computing devices to request sensor data streams.

[0139] By executing code 1210, the processor 1202 can additionally control the operation of the hardware and software stack, which includes drivers and / or other code 1210 configured to communicate with other system devices. Thus, the base station 1012 can interact with other system components in response to received inputs. Such an input can specify, for example, a value to be stored in the data storage 1212. The base station 1012 can also provide an output representing the value stored in the data storage 1212. In some embodiments, the base station 1012 can additionally include one or more light emitting diodes (LEDs) or other visual indicators to visually convey information such as system status or alert events. Further, in some embodiments, the base station 1012 can additionally or alternatively include a siren (e.g., a 95 decibel (dB) siren) or other audio output device that can be controlled by the processor 1202 to output an audio indication of a detected intrusion event.

[0140] The various components of the base station 1012 described above can communicate with each other via an interconnection mechanism 1218. In some embodiments, the interconnection mechanism 1218 can include a communication bus. Further, in some embodiments, the battery assembly 1216 can be configured to supply operating power to the various features of the base station 1012 described above. In some embodiments, the battery assembly 1216 can include at least one rechargeable battery (e.g., one or more nickel metal hydride (NiMH) or lithium batteries). In some embodiments, such a rechargeable battery (or rechargeable batteries) can have a runtime capacity sufficient to operate the base station 1012 for twenty-four hours or longer when the base station 1012 is disconnected from line power or otherwise not receiving line power. In some embodiments, the battery assembly 1216 can additionally or alternatively include a power circuit that receives, regulates, and distributes line power to operate the base station 1012 and / or recharge one or more rechargeable batteries. Such a power circuit can include, for example, a transformer and a rectifier and other circuitry that converts AC line power into DC device and / or recharge power.

[0141] Now turning to Figure 13 , an example keypad 1008 is schematically illustrated. As Figure 13 shown, the keypad 1008 can include at least one processor 1302, volatile memory 1304, non-volatile memory 1308, at least one network interface 1306, a user interface 1314, a battery assembly 1316, and an interconnection mechanism 1318. The non-volatile memory 1308 can store executable code 1310 and, as illustrated, can also include a data storage 1312. In some embodiments, the features of the keypad 1008 listed above can be contained within or otherwise supported by a housing 1320.

[0142] In some embodiments, the corresponding descriptions of the reference base station 1012 for the processor 1202, volatile memory 1204, non-volatile memory 1208, interconnect mechanism 1218, and battery assembly 1216 apply to the corresponding descriptions of the reference keypad 1008 for the processor 1302, volatile memory 1304, non-volatile memory 1308, interconnect mechanism 1318, and battery assembly 1316. Accordingly, these descriptions will not be repeated here.

[0143] By executing the code 1310, the processor 1302 of the keypad 1008 can control the operation of the network interface 1306. In some embodiments, the network interface 1306 may include one or more physical interfaces (e.g., radio transceiver devices, Ethernet ports, USB ports, etc.) and a software stack that includes drivers and / or other code 1310 configured to communicate with the one or more physical interfaces to support one or more LAN, PAN, and / or WAN standard communication protocols. Such communication protocols may include, for example, TCP and UDP. Accordingly, the network interface 1306 can enable the keypad 1008 to access and communicate with other computing devices (e.g., other devices arranged at the Figure 10 monitoring location 104) via a computer network (e.g., a LAN established by the router 1014).

[0144] By executing the code 1310, the processor 1302 can additionally control the operation of the user interface 1314. In some embodiments, the user interface 1314 may include user input and / or output devices (e.g., physical keys arranged as a keypad, touch screen, display, speaker, camera, biometric scanner, environmental sensor, etc.) and a software stack that includes drivers and / or other code 1310 configured to communicate with the user input and / or output devices. Accordingly, the user interface 1314 can enable the keypad 1008 to interact with the user to receive input and / or present output. Examples of output that can be presented by the user interface 1314 include one or more GUIs that include one or more controls configured to display output and / or receive input. Input received by the user interface 1314 can specify, for example, values to be stored in the data storage 1312. Output provided by the user interface 1314 can also indicate values stored in the data storage 1312. In some embodiments, portions of the user interface 1314 (e.g., one or more LEDs) may be accessible and / or visible as part of or through the housing 1320.

[0145] Now turning to Figure 14, schematically illustrates an example sensor assembly 1424. Several example embodiments of the sensor assembly 1424 (e.g., cameras 102 and 102B, motion sensor assembly 1010, and contact sensor assembly 1006) are illustrated and described above in Figure 10 . As shown in Figure 14 , the sensor assembly 1424 may include at least one processor 1402, volatile memory 1404, non-volatile memory 1408, at least one network interface 1406, battery assembly 1416, interconnect mechanism 1418, and at least one sensor 1422. The non-volatile memory 1408 may store executable code 1410 and, as illustrated, may also include data storage 1412. In some embodiments, the features of the sensor assembly 1424 listed above may be included within or comprise a part of the housing 1420. Further, in some embodiments, the sensor assembly 1424 may additionally include a user interface 1414.

[0146] In some embodiments, the corresponding descriptions of the processor 1202, volatile memory 1204, non-volatile memory 1208, interconnect mechanism 1218, and battery assembly 1216 with reference to the base station 1012 apply to the corresponding descriptions of the processor 1402, volatile memory 1404, non-volatile memory 1408, interconnect mechanism 1418, and battery assembly 1416 with reference to the sensor assembly 1424. Accordingly, these descriptions will not be repeated here.

[0147] By executing the code 1410, the processor 1402 may control the operation of the network interface 1406 and the user interface 1414 (if present). In some embodiments, the network interface 1406 may include one or more physical interfaces (e.g., radio transceiver, Ethernet port, USB port, etc.) and a software stack that includes drivers and / or other code 1410 configured to communicate with the one or more physical interfaces to support one or more LAN, PAN, and / or WAN standard communication protocols. Such communication protocols may include, for example, TCP and UDP. Accordingly, the network interface 1406 may enable the sensor assembly 1424 to access other computing devices (e.g., arranged in Figure 10other devices in the monitoring location 104) and communicate with it. For example, in some embodiments, when executing code 1410, the processor 1402 may control the network interface to transmit (e.g., via UDP) sensor data obtained from the sensor component 1422 to the base station 1012. Further, in some embodiments, by executing code 1410, the processor 1402 may additionally or alternatively control the network interface 1406 to enter a power saving mode, such as by powering off the 2.4 GHz radio transceiver device, which is included in the network interface 1406, and powering on the sub-GHz radio transceiver device. In such an embodiment, by executing code 1410, the processor 1402 may additionally control the network interface 1406 to enter a transmission mode, such as by powering on the 2.4 GHz radio transceiver device and powering off the sub-GHz radio transceiver device, for example, in response to receiving a wake-up signal from the base station via the sub-GHz radio transceiver device.

[0148] By executing code 1410, the processor 1402 may additionally or alternatively control other operations of the sensor component 1424. In some embodiments, for example, the user interface 1414 of the sensor component 1424 may include user input and / or output devices (e.g., physical buttons, touchscreens, displays, speakers, cameras, accelerometers, biometric scanners, environmental sensors, one or more LEDs, etc.) and a software stack that includes drivers and / or other code 1410 configured to communicate with the user input and / or output devices. Thus, the sensor component 1424 may enable the user interface 1414 to interact with the user to receive input and / or present output. The output presented by the user interface 1314 may include, for example, one or more GUIs that include one or more controls configured to display output and / or receive input. The input received by the user interface 1414 may specify, for example, a value to be stored in the data storage 1412. The output provided by the user interface 94 may also indicate the value stored in the data storage 1412. In some embodiments, a portion of the sensor component 1424 may be accessible and / or visible as part of or through the housing 1420.

[0149] As Figure 14 shown, the sensor component 1424 may include one or more types of sensors 1422, as described above with reference to Figure 10One or more of the sensors described for cameras 102A and 102B, motion sensor assembly 1010, and contact sensor assembly 1006, or other types of sensors. In some embodiments, for example, sensor 1422 can include a camera and a temperature sensor. Regardless of the type of sensor XD22 employed, processor 1402 can (e.g., via execution of code 1410) obtain sensor data from sensor 1422 and transmit the obtained sensor data to processor 1402 for transmission to base station 1012.

[0150] It should be noted that in some embodiments of devices 1302 and 1402, the operations performed by processors 1302 and 1402 under the respective control of code 1310 and 1410 can be hardcoded and / or implemented using hardware rather than as a combination of hardware and software.

[0151] Now turning to Figure 15 , schematically illustrates Figure 10 Aspects of the surveillance center environment 1026, monitoring center environment 1022, one terminal device 114, network 1020, and multiple monitoring locations 104A through 104N (collectively referred to as monitoring locations 104) shown. As Figure 15 shown, in some embodiments, surveillance service 1030 can include location data storage 1502, image data storage 1504, artificial intelligence (AI) service 1508, event listening service 1510, identity provider service 1512, customer service 1538, monitoring service 106, and camera transmission service 1506. Also as Figure 15 shown, monitoring center environment 1022 can include multiple monitoring devices 110A through 110M (collectively referred to as monitoring devices 110), which are hosted or otherwise configured to access respective monitoring applications 126A through 126M, and each of the monitoring locations 104A through 104N can include respective surveillance clients 1116A through 1116N (collectively referred to as surveillance clients 1016), e.g., at each of the monitoring locations 104A through 1102N within base station 1012 ( Figure 15 not shown). As described above in connection with Figure 5 and Figure 6 , in some embodiments, monitoring application 126 can be configured such that monitoring device 110 displays screens 502, 602, which enable monitoring agent 112 to visually monitor the activities of one or more of the monitoring locations 104, and (e.g., via the microphone and speaker of camera 102 at monitoring location 104) have an audio conversation with one or more individuals at such locations. Further, as Figure 15As further shown herein, in some embodiments, the transport service 1028 may include a plurality of different transport services 1128A through 1128D configured to receive location data packets, such as location data packets 1014A through 1014D, from monitoring clients 1116A through 1116N deployed at respective monitoring locations 104A through 1102N.

[0152] The location data store 1502 of the monitoring service 1030 may be configured to store location data associated with an identifier of a customer 116 monitored at a monitoring location 104 in a plurality of records. For example, the location data may be stored in a record together with an identifier of the customer 116 and / or an identifier of the monitoring location 104 to associate the location data with the customer 116 and the monitoring location 104. The image data store 1504 of the monitoring service 1030 may be configured to store one or more frames of image data associated with an identifier of a location and a timestamp at which the image data was acquired in a plurality of records.

[0153] The AI service 1508 of the monitoring service 1030 may be configured to process an image and / or an image sequence to identify semantic regions, movements, faces, and other features within the image or image sequence. The event listening service 1510 of the monitoring service 1030 may be configured to scan received location data for events and, in the event an event is identified, execute one or more event handlers to process the event. In some embodiments, such an event handler may be configured to identify the event and transmit a message regarding the event to one or more recipient services (e.g., the customer service 1538 and / or the monitoring service 106). Operations that the customer service 1538 and / or the monitoring service 106 may perform based on events identified by the event listening service 1510 are further described below. In some embodiments, the event listening service 1510 may interoperate with the AI service 1508 to identify events within the image data.

[0154] The identity provider service 1512 may be configured to receive an authentication request including a security certificate from the monitoring client 1016. When the identity provider 1512 can authenticate the security certificate in the request (e.g., via a confirmation function, cross-reference lookup, or some other authentication process), the identity provider 1512 may transmit a security token in response to the request. The monitoring client 1016 may receive, store, and include the security token in a subsequent location data (e.g., location data 1014A) packet such that a receiving transport service (e.g., the transport service 1128A) can securely process (e.g., unpack / parse) the packet to extract the location data before passing the location data to the monitoring service 1030.

[0155] The transmission service 1028 of the monitoring center environment 1026 can be configured to receive location data packets 1514, verify the authenticity of the packets 1514, parse the packets 1514, and extract the location data encoded therein before passing the location data to the monitoring service 1030 for processing. The location data processed in this way can include any of the location data types described above with reference to Figure 10 Any of the location data types described. In some embodiments, each transmission service 1028 can be configured to process location data packets 1514 generated by location-based monitoring devices of a particular manufacturer and / or model. The monitoring client 1016 can be configured to generate location data packets (e.g., location data packets 1514) based on sensor information received at the monitoring location 104 and transmit them to the monitoring service 1030 via, for example, the network 1020.

[0156] The monitoring service 106 can maintain records of events identified by the event listening service 1510 and can assign each event to each monitoring agent 112 currently online via the monitoring application 126. The monitoring application 126 operated by a given monitoring agent 112 can then add the events assigned to that monitoring agent 112 to an event queue, such as within the window 506 shown, for review by that monitoring agent 112. In some embodiments, a given monitoring application 126 can use data describing the events within its queue to retrieve location data and / or image data (from the location data store 1502 and / or the image data store 1504, respectively) for presentation within or associated with the window 506. Figure 1 In response to a monitoring agent 112 identifying a particular event to review (e.g., by clicking on a window 506), the monitoring service 106 can interact with the camera transmission service 1506 to obtain an access certificate, enabling the establishment of a peer-to-peer connection with one or more cameras 102 at the monitoring location 104 corresponding to the event and reviewing live video and / or audio transmitted from these cameras within, for example, the window 604 and / or the main viewer window 606 shown, as well as communicating verbally in real time with one or more individuals near the camera 102. Example interactions between the components of the security system 1000 to enable the transmission of video and / or audio data between the camera 102 at the monitoring location 104 and the monitoring application 126 operated by the monitoring agent 112 are described below with reference to FIGS. 17 and 18.

[0157] In response to a monitoring agent 112 identifying a particular event to review (e.g., by clicking on a window 506), the monitoring service 106 can interact with the camera transmission service 1506 to obtain an access certificate, enabling the establishment of a peer-to-peer connection with one or more cameras 102 at the monitoring location 104 corresponding to the event and reviewing live video and / or audio transmitted from these cameras within, for example, the window 604 and / or the main viewer window 606 shown, as well as communicating verbally in real time with one or more individuals near the camera 102. Example interactions between the components of the security system 1000 to enable the transmission of video and / or audio data between the camera 102 at the monitoring location 104 and the monitoring application 126 operated by the monitoring agent 112 are described below with reference to FIGS. 17 and 18. Figure 6 In response to a monitoring agent 112 identifying a particular event to review (e.g., by clicking on a window 506), the monitoring service 106 can interact with the camera transmission service 1506 to obtain an access certificate, enabling the establishment of a peer-to-peer connection with one or more cameras 102 at the monitoring location 104 corresponding to the event and reviewing live video and / or audio transmitted from these cameras within, for example, the window 604 and / or the main viewer window 606 shown, as well as communicating verbally in real time with one or more individuals near the camera 102. Example interactions between the components of the security system 1000 to enable the transmission of video and / or audio data between the camera 102 at the monitoring location 104 and the monitoring application 126 operated by the monitoring agent 112 are described below with reference to FIGS. 17 and 18.

[0158] Now turning to Figure 16 , an example monitoring process 1600 that can be employed by the security system 1000 is illustrated as a sequence diagram. Specifically, in some embodiments, the various parts of the process 1600 can be performed by (A) at least one processor (e.g., Figure 13 orFigure 14 One or more location-based devices (e.g., Figure 10 devices 102A, 102B, 1006, 1008, and 1014) under the control of device control system (DCS) code (e.g., code 1310 or 1410) implemented by either processor 1302 or 1402); (B) A base station (e.g., Figure 10 base station 1012) under the control of a monitoring client (e.g., Figure 10 monitoring client 1016); (C) A monitoring center environment (e.g., Figure 10 monitoring center environment 1022) under the control of a monitoring application (e.g., Figure 10 monitoring application 126); (D) A monitoring center environment (e.g., Figure 10 monitoring center environment 1026) under the control of a monitoring service (e.g., Figure 10 monitoring service 1030); and (E) A terminal device (e.g., Figure 10 terminal device 114) under the control of a terminal application (e.g., Figure 10 terminal application 128).

[0159] As Figure 16 shown, process 1600 may begin with monitoring client 1016 authenticating to monitoring service 1030 by exchanging one or more authentication requests and responses 1604 with monitoring service 1030. More specifically, in some embodiments, monitoring client 1016 may transmit an authentication request to monitoring service 1030 via one or more API calls to monitoring service 1030. In such an embodiment, monitoring service 1030 may parse the authentication request to extract a security certificate therefrom and pass such a security certificate to an identity provider (e.g., Figure 15 identity provider service 1512) for authentication. In some embodiments, when the identity provider authenticates the security certificate, monitoring service 1030 may generate a security token and transmit the security token as a payload within the authentication response to the authentication request. In such an embodiment, if the identity provider cannot authenticate the security certificate, monitoring service 1030 may instead generate an error (e.g., an error code) and transmit the error as a payload within the authentication response to the authentication request. Upon receiving the authentication response, monitoring client 1016 may parse the authentication response to extract the payload. If the payload includes an error code, monitoring client 1016 may retry authentication and / or its user interface with the host device (e.g., Figure 12interoperate with the user interface 1214 of the base station 1012 to present an output indicating authentication failure. If the payload includes a security token, the monitoring client 1016 may store the security token for subsequent use in the transmission of location data. It should be noted that in some embodiments, the security token may have a limited lifespan (e.g., one hour, one day, one week, one month, etc.), after which the monitoring client 1016 may be required to re-authenticate with the monitoring service 1030.

[0160] Continuing with process 1600, one or more device control systems 1602 hosted by one or more location-based devices may obtain (1606) sensor data that describes a location (e.g., Figure 10 the monitored location 104). The sensor data so obtained may be any of a variety of types, as discussed above with reference to Figures 10 to 15 . In some embodiments, one or more device control systems 1602 may obtain sensor data continuously. In other embodiments, one or more DCSs 1602 may additionally or alternatively obtain sensor data in response to events such as a timer expiration (push event) or receipt of an acquisition poll signal (poll event) transmitted by the monitoring client 1016. In some embodiments, one or more device control systems 1602 may transmit the sensor data to the monitoring client 1016 with a minimum of processing beyond acquisition and digitization. In such embodiments, the sensor data may constitute a vector sequence with individual vector members, including, for example, sensor readings and timestamps. In some embodiments, one or more device control systems 1602 may perform additional processing of the sensor data, such as generating one or more summaries of multiple sensor readings. Still further, in some embodiments, one or more device control systems 1602 may perform complex processing of the sensor data. For example, if the sensor 1422 of the sensor assembly 1424 ( Figure 14 shown therein) includes an image capture device, the device control system 1602 may perform image processing routines such as edge detection, motion detection, face recognition, threat assessment, event generation, etc.

[0161] Continuing with process 1600, the device control component 1602 may transmit the sensor data 1608 to the monitoring client 1016. Similar to sensor data acquisition, the device control system 1602 may transmit the sensor data 1608 continuously or in response to events such as a push event (originating from the device control system 1602) or a poll event (originating from the monitoring client 1016).

[0162] Continuing process 1600, the monitoring client 1016 can monitor (1610) the monitoring location 104 by processing the received sensor data 1608. In some embodiments, for example, the monitoring client 1016 can execute one or more image processing routines. Such image processing routines can include any of the image processing routines described above with reference to operation 1606. By distributing at least some of the image processing routines between the device control system 1602 and the monitoring client 1016, the power consumed by the battery-powered device can be reduced by offloading processing from the line-powered device. Additionally, in some embodiments, the monitoring client 1016 can execute an overall threat detection process that utilizes sensor data 1608 from multiple different device control systems 1602 as input. For example, in some embodiments, the monitoring client 1016 can attempt to confirm an open state received from a contact sensor with motion and facial recognition processing of an image of a scene including a window or door secured by the contact sensor. If two or more of the three processes indicate the presence of an intruder, a score (e.g., a threat score) can be increased and / or a break-in event can be declared, locally recorded, and transmitted. Other processing that the monitoring client 1016 can perform includes outputting a local alert (e.g., in response to detecting a specific event and / or meeting other criteria) and detecting maintenance conditions of location-based devices, such as the need to replace or recharge a low-battery and / or replace / maintain the device hosting the device control system 1602. Any of the above processes within operation 1610 can result in the creation of location data specifying the results of these processes.

[0163] Continuing process 1600, the monitoring client 1016 can transmit the location data 1612 to the monitoring service 1030 (via the transmission service 1028). Similar to the transmission of the sensor data 1608, the monitoring client 1016 can transmit the location data 1612 continuously or in response to an event, such as a push event (originating from the monitoring client 1016) or a poll event (originating from the monitoring service 1030).

[0164] Continuing with process 1600, the monitoring service 1030 may process (1614) the received location data. In some embodiments, for example, the monitoring service 1030 may perform one or more of the processes described above with reference to operations 1606 and / or 1610. In some embodiments, the monitoring service 1030 may additionally or alternatively calculate a score (e.g., a threat score) or further refine an existing score using historical information associated with the monitored location 104 identified in the location data and / or other locations geographically close to the monitored location 104 (e.g., within the same Zip Code). For example, in some embodiments, if multiple break-ins have been recorded for the monitored location 104 and / or other locations within the same Zip Code, the monitoring service 1030 may increase the score calculated by the device control system 1602 and / or the monitoring client 1016.

[0165] In some embodiments, the monitoring service 1030 may apply a set of rules and criteria to the location data 1612 to determine whether the location data 1612 includes any events, and if so, transmit event reports 1616A and / or 1616B to the monitoring application 126 and / or the terminal application 128. In some embodiments, for example, the monitoring service 106 may assign one or more events to a particular monitoring agent 112 such that the events will be forwarded to the monitoring application 126 in which the monitoring agent 112 operates, e.g., for presentation within the corresponding window 506 ( Figure 5 shown). The events may be, for example, a particular type of event (e.g., a break-in) or a particular type of event that meets additional criteria (e.g., movement within a particular area combined with a threat score above a threshold). The event reports 1616A and / or 1616B may have a priority based on the same criteria used to determine whether the events reported therein are reportable, or may have a priority based on a different set of criteria or rules.

[0166] Continuing with process 1600, the monitoring application 126 in the monitoring center environment 1022 may interact (1618) with the monitoring agent 112 via, for example, one or more GUIs such as the screens 502 and 602 as Figure 5 shown and Figure 6 shown. Such GUIs may provide details and content regarding one or more events.

[0167] As Figure 16 shown, the terminal application 128 of the terminal device 114 (e.g., a smart phone, a personal computer, or other terminal device) may similarly interact (1620) with at least one client via, for example, one or more GUIs. Such GUIs may provide details and content regarding one or more events.

[0168] It should be noted that the processing of sensor data and / or location data as described above with reference to operations 1606, 1610, and 1614 can be performed by processors arranged within various parts of the security system 1000. In some embodiments, the device control system 1602 can perform minimal processing of the sensor data (e.g., only acquire and transmit), and the remainder of the above processing can be performed by the monitoring client 1016 and / or the monitoring service 1030. This approach can help extend the battery runtime of location-based devices. In other embodiments, the device control system 1602 can perform as much sensor data processing as possible, such that the monitoring client 1016 and the monitoring service 1030 only perform processes that require sensor data across location-based devices and / or locations. This approach can help improve the scalability of the security system 1000 in terms of adding new locations.

[0169] The following clauses describe examples of the inventive concepts disclosed herein.

[0170] Clause 1. A method, comprising: performing, by a computing system, a first image processing on image data acquired by a camera at a property to determine at least a first feature of the image data; determining, by the computing system, that the first feature satisfies a first rule; generating, at least in part based on the first feature satisfying the first rule, a first vector representing a plurality of characteristics of a first face represented in the image data; determining a second vector representing a plurality of characteristics of a second face of a person; determining that the first vector corresponds to the second vector; and determining, at least in part based on the first vector corresponding to the second vector, that the person is represented in the image data.

[0171] Clause 2. The method according to clause 1, further comprising: performing, by the computing system, a second image processing on the image data to determine a second feature of the image data, wherein the second feature is different from the first feature; and determining, by the computing system, that the second feature satisfies a second rule, wherein the second rule is different from the first rule; wherein generating the first vector is further at least in part based on the second feature satisfying the second rule.

[0172] Clause 3. The method according to clause 2, further comprising: performing, by the computing system, a third image processing on the image data to determine a third feature of the image data, wherein the third feature is different from the first feature and the second feature; and determining, by the computing system, that the third feature satisfies a third rule, wherein the third rule is different from the first rule and the second rule; wherein generating the first vector is further at least in part based on the third feature satisfying the third rule.

[0173] Clause 4. The method according to any one of clauses 1 to 3, wherein the first feature includes an estimated distance between the camera and the person; and the first rule is that the estimated distance is less than a threshold distance.

[0174] Clause 5. The method according to any one of Clauses 1 to 3, wherein the first feature includes an estimated distance between the camera and the first face; and the first rule is that the estimated distance is less than a threshold distance.

[0175] Clause 6. The method according to any one of Clauses 1 to 3, wherein the first feature includes the position of a person relative to the boundary line of a property; and the first rule is that the person is within the boundary line.

[0176] Clause 7. The method according to any one of Clauses 1 to 3, wherein the first feature includes an estimated age of a person; and the first rule is that the estimated age of the person is higher than a threshold age.

[0177] Clause 8. The method according to Clause 2 or Claim 3, wherein the first feature includes an estimated distance between the camera and the person; the first rule is that the estimated distance is less than a threshold distance; the second feature includes the position of the person relative to the boundary line of the property; and the second rule is that the person is within the boundary line.

[0178] Clause 9. The method according to Clause 2 or Claim 3, wherein the first feature includes an estimated distance between the camera and the first face; the first rule is that the estimated distance is less than a threshold distance; the second feature includes the position of the person relative to the boundary line of the property; and the second rule is that the person is located within the boundary line.

[0179] Clause 10. The method according to Clause 2 or Claim 3, wherein the first feature includes an estimated distance between the camera and the person; the first rule is that the estimated distance is less than a threshold distance; the second feature includes an estimated age of the person; and the second rule is that the estimated age of the person is higher than a threshold age.

[0180] Clause 11. The method according to Clause 2 or Claim 3, wherein the first feature includes an estimated distance between the camera and the first face; the first rule is that the estimated distance is less than a threshold distance; the second feature includes an estimated age of the person; and the second rule is that the estimated age of the person is higher than a threshold age.

[0181] Clause 12. The method according to Clause 3, wherein the first feature includes an estimated distance between the camera and the person; the first rule is that the estimated distance is less than a threshold distance; the second feature includes the position of the person relative to the boundary line of the property; the second rule is that the person is located within the boundary line; the third feature includes an estimated age of the person; and the third rule is that the estimated age of the person is higher than a threshold age.

[0182] Clause 13. The method according to Clause 3, wherein the first feature includes an estimated distance between the camera and the first face; the first rule is that the estimated distance is less than a threshold distance; the second feature includes the position of the person relative to the boundary line of the property; the second rule is that the person is located within the boundary line; the third feature includes an estimated age of the person; and the third rule is that the estimated age of the person is higher than a threshold age.

[0183] Clause 14. The method according to any one of Clauses 1 to 13 further includes: determining that the security system at the property is in an alert state; wherein generating the first vector is at least partially based on the security system being in an alert state.

[0184] Clause 15. The method according to any one of Clauses 1 to 14 further includes: determining that the current time is within a time window configured for the security system to perform biometric face recognition; wherein generating the first vector is at least based on the current time being within the time window.

[0185] Clause 16. The method according to any one of Clauses 1 to 15 further includes: determining that a setting indicates that biometric face recognition is enabled for the property; wherein generating the first vector is at least partially based on the setting indicating that biometric face recognition is enabled for the property.

[0186] Clause 17. The method according to Clause 16 further includes: determining the geographical region where the property is located; determining that biometric face recognition is permitted within the geographical region; and configuring the setting to indicate that biometric face recognition is enabled for the property at least partially based on biometric face recognition being permitted within the geographical region.

[0187] Clause 18. The method according to Clause 17 further includes: causing the terminal device to present a user interface to obtain authorization for using biometric face recognition; and determining that the authorization has been received; wherein configuring the setting to indicate that biometric face recognition is enabled for the property is further based on the receipt of the authorization.

[0188] Clause 19. The method according to any one of Clauses 1 to 18, wherein determining that the first vector corresponds to the second vector includes: determining that the first vector and the second vector are within a threshold similarity.

[0189] Clause 20. The method according to Clause 19 further includes: receiving, by a computing system, a request for a document of the creator; determining the person's email address or phone number; sending, at least partially based on the request, a request for permission to use biometric face recognition of one or more images of the person using the email address or phone number; and generating the second vector at least partially based on a response indicating permission to use biometric face recognition.

[0190] Clause 21. The method according to any one of Clauses 1 to 20, wherein a first image processing is performed in response to determining, by the computing system, that the image data represents a first face.

[0191] Clause 22. The method according to any one of Clauses 1 to 21, wherein generating the first vector includes generating a biometric embedding of the first face.

[0192] Clause 23. A method includes: determining by a computing system that a setting indicates enabling biometric face recognition for a property; generating, by the computing system and at least partially based on the setting indicating enabling biometric face recognition for the property, a first vector that represents multiple characteristics of a first face represented in image data acquired by a camera at the property; determining a second vector that represents multiple characteristics of a second face of a person; determining that the first vector corresponds to the second vector; and determining that the person is in the image data at least partially based on the first vector corresponding to the second vector.

[0193] Clause 24. The method according to clause 23 further includes: determining a geographic region in which the property is located; determining that biometric face recognition is permitted within the geographic region; and configuring the setting to indicate enabling biometric face recognition for the property at least partially based on biometric face recognition being permitted within the geographic region.

[0194] Clause 25. The method according to clause 24 further includes: causing a terminal device to present a user interface to obtain authorization for using biometric face recognition; and determining that the authorization has been received; wherein configuring the setting to indicate enabling biometric face recognition for the property is further based on receipt of the authorization.

[0195] Clause 26. The method according to any one of clauses 23 to 25, wherein determining that the first vector corresponds to the second vector includes: determining that the first vector and the second vector are within a threshold similarity.

[0196] Clause 27. The method according to any one of clauses 23 to 26 further includes: receiving, by the computing system, a request to create a visitor document for a person; determining the person's email address or phone number; sending, at least partially based on the request, a request to use biometric face recognition of one or more images of the person using the email address or phone number; and generating the second vector at least partially based on a response indicating permission to use biometric face recognition.

[0197] Clause 28. The method according to any one of clauses 23 to 27, wherein generating the first vector includes generating a biometric embedding of the first face.

[0198] Clause 29. A method includes: receiving, by a computing system, a request to create a document for a person; determining the person's email address or phone number; sending, at least partially based on the request, a request to use biometric face recognition of one or more images of the person using the email address or phone number; and generating a vector that represents multiple characteristics of the person's face at least partially based on a response indicating permission to use biometric face recognition.

[0199] Clause 30. A system includes: one or more processors; and one or more non-transitory computer-readable media encoded with instructions that, when executed by the one or more processors, cause the system to perform the method according to any one of Clauses 1 to 29.

[0200] Clause 31. One or more non-transitory computer-readable media encoded with instructions that, when executed by one or more processors of a system, cause the system to perform the method according to any one of Clauses 1 to 29.

[0201] Various inventive concepts may be embodied as one or more methods, examples of which have been provided. The acts performed as part of a method may be ordered in any suitable way. Accordingly, examples may be constructed in which acts are performed in an order different from that illustrated, which may include performing some acts simultaneously, even though shown as sequential acts in illustrative examples.

[0202] The use of ordinal terms such as "first," "second," "third," etc. in the claims to modify a claim element itself does not mean any precedence, order, or sequence of one claim element with respect to another, or the temporal order of acts of a method. These terms are merely labels used to distinguish one claim element having a particular name from another element having the same name (but using an ordinal term).

[0203] The examples of methods and systems discussed herein are not limited in application to the details of the construction and arrangement of components set forth in the following description or illustrated in the drawings. The methods and systems are capable of being implemented in other examples and of being practiced or carried out in various ways. The examples of specific implementations provided herein are for illustrative purposes only and are not intended to be limiting. In particular, the acts, components, elements, and features discussed in connection with any one or more examples are not intended to be excluded from a similar role in any other examples.

[0204] Moreover, the language and terminology used herein are for descriptive purposes and should not be regarded as limiting. Any reference herein to an example, component, element, or act in the singular form of a system and method may also include examples that include a plurality, and any reference to the plural form of any example, component, element, or act herein may also include examples that include only a singular. References in the singular or plural form are not intended to limit the systems or methods of the invention, their components, acts, or elements.

[0205] As used herein, "comprising", "including", "having", "containing", "involving" and variations thereof mean including the items listed thereafter and their equivalents as well as additional items. References to "or" may be construed as inclusive such that any term described using "or" may indicate any one of a single, more than one, and all of the described terms. Additionally, in the case of inconsistent usage of terms between this document and the documents incorporated herein by reference, the usage of terms in the incorporated references is supplementary to the usage in this document; for irreconcilable inconsistencies, the usage of terms in this document shall prevail.

[0206] Having described several examples in detail, various modifications and improvements will readily occur to those skilled in the art. These modifications and improvements are intended to be within the scope of the present invention. Accordingly, the foregoing description is merely exemplary and not restrictive.

[0207] What is claimed is:

Claims

1. A method comprising: determining a first identifier of a first geographic area in which a property monitored by the computing system is located; by executing a first application on a first terminal device associated with the property, causing a first identifier to be evaluated against first content of a first data store to determine that facial recognition processing is permitted within a first geographic area, the first content including an identifier of a geographic area in which facial recognition processing is permitted or prohibited; causing the first terminal device to present a user interface element based at least in part on enabling facial recognition processing within the first geographic area, the user interface element being switchable between a first state corresponding to disabling facial recognition for the property and a second state corresponding to enabling facial recognition processing for the property, wherein the user interface element in the first state causes the computing system to process image data acquired by a camera at the property to determine that a person is represented in the image data but to refrain from performing facial recognition processing on the image data to determine the identity of the person; determining, by the first application, that the user interface element is in the second state; as well as Based at least in part on the user interface element being in the second state, causing the computing system to perform facial recognition processing on the first image data acquired by the camera to determine an identity of at least a first person represented in the first image data.

2. The method according to claim 1, further comprising: generating, by a computing system, a first vector representing a plurality of characteristics of a first face represented in the first image data; identifying, by the computing system, a second vector representing a plurality of characteristics of a second face represented in the second image data, the second face belonging to the first person; as well as An identity of the first person is determined, by the computing system, based at least in part on the first vector corresponding to the second vector.

3. The method of claim 2, wherein determining that the first vector corresponds to the second vector comprises: It is determined that the first vector and the second vector are within a threshold similarity.

4. The method according to claim 2 or 3, further comprising: receiving, by a computing system, a first request to create a document for a first person; determining, by the computing system, a second identifier of the first person; causing, by the computing system, a second request to be sent to a second application corresponding to the second identifier based at least in part on the first request, the second request seeking authorization from the first person for use of facial recognition processing using one or more images of the first person; receiving, by the computing system, a response to the second request, the response indicating that the first person has authorized use of facial recognition processing of one or more images of the first person; determining, by the computing system, that a second face represented in the second image data belongs to the first person; as well as Based at least in part on the response that the second face belongs to the first person, a second vector is generated, by the computing system, and using the second image data. The method of claim 4 , wherein the second identifier comprises an email address or a telephone number of the first person.

6. The method of claim 4 or 5, wherein the first request is received from a first application.

7. The method of any one of claims 2 to 6, wherein generating the first vector comprises identifying and measuring facial features represented in an image of the first face.

8. The method according to any one of claims 2 to 7, further comprising: storing, by the computing system and in a second data storage, first data representing an image of a first face; determining, based at least in part on the computing system, an identity of the first person, storing, by the computing system and in a second data store, second data associated with the first data, the second data indicating that the image of the first face corresponds to the first person; as well as Based at least in part on the first data and the second data already being stored in the second data store, the second application is caused to output an indication that the image of the first face corresponds to the first person.

9. The method according to any one of claims 1 to 8, further comprising: receiving, by the computing system, third image data acquired by the camera; processing, by a computing system, the third image data to determine that a second person is represented in the third image data; as well as Based at least in part on the user interface element being in the first state, performing facial recognition processing on the third image data to determine the identity of the second person is avoided.

10. The method according to claim 9, further comprising: determining, by the computing system, that the user interface element has transitioned from the second state to the first state; Based at least in part on the user interface element having transitioned from the second state to the first state, changing a stored setting of the computing system from a first value to a second value; performing facial recognition processing on the first image data based at least in part on the stored setting having a first value; as well as Performing facial recognition processing on the third image data is avoided based at least in part on the stored setting having the second value.

11. The method according to any one of claims 1 to 10, further comprising: causing, by the first application, the first identifier to be evaluated against second content of the second data store to determine that consent of the user is required before performing facial recognition processing within the first geographic area, the second content comprising identifiers of geographic areas for which consent is or is not required before performing facial recognition processing; as well as Based at least in part on requiring user consent before performing facial recognition processing within the first geographic area, causing the first terminal device to present a user interface element in a first state.

12. The method according to claims 1 to 10, further comprising: causing, by the first application, the first identifier to be evaluated against second content of the second data store to determine that consent of the user is not required prior to performing facial recognition processing within the first geographic area, the second content comprising identifiers of geographic areas for which consent is or is not required prior to performing facial recognition processing; as well as Based at least in part on not requiring user consent prior to performing facial recognition processing within the first geographic area, causing the first terminal device to present the user interface element in the second state.

13. A system comprising: one or more processors; and One or more computer-readable media encoded with instructions that, when executed by one or more processors, cause the system to: determining a first identifier of a first geographic area in which a property monitored by the system is located; causing, by a first application executed on a first terminal device associated with the property, to evaluate a first identifier against first content of a first data store to determine that facial recognition processing is permitted within a first geographic area, the first content including an identifier of a geographic area within which facial recognition processing is permitted or prohibited; causing the first terminal device to present a user interface element based at least in part on enabling facial recognition processing within the first geographic area, the user interface element being switchable between a first state corresponding to disabling facial recognition processing for the property and a second state corresponding to enabling facial recognition processing for the property, wherein the user interface element in the first state causes the system to process image data acquired by a camera at the property to determine that a person is represented in the image data but to refrain from performing facial recognition processing on the image data to determine the identity of the person; determining, by the first application, that the user interface element is in a second state; as well as Based at least in part on the user interface element being in the second state, the system is caused to perform facial recognition processing on the first image data acquired by the camera to determine the identity of at least a first person represented in the first image data.

14. The system of claim 13, wherein the one or more computer readable media are further encoded with additional instructions that, when executed by the one or more processors, further cause the system to perform facial recognition processing on the first image data at least in part by: generating a first vector representing a plurality of features representing a first face in the first image data; identifying a second vector representing a plurality of characteristics representing a second face in the second image data, the second face belonging to the first person; and An identity of the first person is determined based at least in part on the first vector corresponding to the second vector.

15. The system of claim 14, wherein the one or more computer readable media are further encoded with additional instructions that, when executed by the one or more processors, further cause the system to: The first vector is determined to correspond to the second vector at least in part by determining that the first vector and the second vector are within a threshold similarity.

16. The system of claim 14 or 15, wherein the one or more computer readable media are further encoded with additional instructions that, when executed by the one or more processors, further cause the system to: receiving a first request to create a document for a first person; determining a second identifier of the first person; causing, based at least in part on the first request, to be sent a second request to a second application corresponding to the second identifier, the second request seeking authorization from the first person for use of facial recognition processing using one or more images of the first person; receiving a response to the second request, the response indicating that the first person has authorized use of facial recognition processing of one or more images of the first person; determining that a second face represented in the second image data belongs to the first person; as well as Based at least in part on the response and the second face belonging to the first person, a second vector is generated using the second image data.

17. The system of claim 16, wherein the second identifier comprises an email address or a telephone number of the first person.

18. The system of claim 16 or 17, wherein the first request is received from a first application.

19. The system of any one of claims 14 to 18, wherein the one or more computer readable media are further encoded with additional instructions that, when executed by the one or more processors, further cause the system to: A first vector is generated at least in part by identifying and measuring facial features represented in an image of a first face.

20. The system of any one of claims 14 to 19, wherein the one or more computer readable media are further encoded with additional instructions that, when executed by the one or more processors, further cause the system to: storing in a second data storage first data representing an image of a first face; Based at least in part on the determined identity of the first person, storing second data associated with the first data in a second data store, the second data indicating that the image of the first face corresponds to the first person; and Based at least in part on the first data and the second data already being stored in the second data store, the second application is caused to output an indication that the image of the first face corresponds to the first person.

21. The system of any one of claims 13 to 20, wherein the one or more computer readable media are further encoded with additional instructions that, when executed by one or more processors, further cause the system to: receiving third image data acquired by the camera; processing the third image data to determine that the second person is represented in the third image data; and Based at least in part on the user interface element being in the first state, performing facial recognition processing on the third image data to determine the identity of the second person is avoided.

22. The system of claim 21, wherein the one or more computer readable media are further encoded with additional instructions that, when executed by the one or more processors, further cause the system to: determining that the user interface element has transitioned from the second state to the first state; changing a stored setting of the system from a first value to a second value based at least in part on the user interface element having transitioned from the second state to the first state; performing facial recognition processing on the first image data based at least in part on the stored setting having a first value; as well as Performing facial recognition processing on the third image data is avoided based at least in part on the stored setting having the second value.

23. The system of any one of claims 13 to 22, wherein the one or more computer readable media are further encoded with additional instructions that, when executed by the one or more processors, further cause the system to: causing, by the first application, the first identifier to be evaluated against second content of a second data store to determine that user consent is required prior to performing facial recognition processing within the first geographic area, the second content comprising identifiers of geographic areas for which consent is or is not required prior to performing facial recognition processing; and Based at least in part on requiring user consent before performing facial recognition processing within the first geographic area, causing the first terminal device to present a user interface element in a first state.

24. The system of any one of claims 13 to 22, wherein the one or more computer readable media are further encoded with additional instructions that, when executed by the one or more processors, further cause the system to: causing, by the first application, the first identifier to be evaluated against second content of a second data store to determine that consent of the user is not required prior to performing facial recognition processing within the first geographic area, the second content comprising identifiers of geographic areas for which consent is or is not required prior to performing facial recognition processing; and Based at least in part on not requiring user consent prior to performing facial recognition processing within the first geographic area, causing the first terminal device to present the user interface element in the second state.

Images