User-defined computer vision event detection

Abstract

Technologies are generally described for user-defined computer vision event detection. A camera device may capture a first frame of image data. The first machine learning model may generate first embedded data for the first frame of image data. The first embedded data may be input into a second machine learning model associated with the camera device. The second machine learning model and the first embedded data may be used to determine that a region of interest represented by the first frame of image data is in a first state. State data stored in the memory may be determined. The state data may indicate that the region of interest was previously in a second state. An alert may be sent to a device associated with the camera device indicating that the region of interest has changed from the second state to the first state.

Description

Background Art

[0001] Computer vision is a field of artificial intelligence that enables computers to perceive and understand visual input. Machine learning computer vision models can allow computers to detect and classify objects and / or other regions of interest that appear in image data and video data. Upon classifying a particular object and / or region of interest, a computer can be used to react to the particular object and / or region of interest in a variety of defined ways. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Figure 1 is a diagram depicting an exemplary system for efficiently providing user-customized computer vision event detection in accordance with various aspects of the present disclosure.

[0003] Figure 2 Described are various aspects of the present disclosure for use in Figure 1 An exemplary process for training and runtime operation of the system described in .

[0004] Figure 3 Depicted are examples of user-defined regions of interest in frames of image data in accordance with various aspects of the present disclosure.

[0005] Figure 4A and Figure 4B A first exemplary interface for enabling a user to name a user-defined state and a second exemplary interface for selecting image data corresponding to the user-defined state according to various aspects of the present disclosure are respectively depicted.

[0006] Figure 5 Depicted are exemplary evaluations of training images to determine whether training should continue in accordance with various aspects of the present disclosure.

[0007] Fig. 6A and Figure 6B Depicted are exemplary training processes that can be used to train a system to provide user-customized computer vision event detection according to various aspects of the present disclosure.

[0008] Figure 7 A cropping operation is shown for generating a cropped frame of image data according to various aspects of the present disclosure.

[0009] Figure 8 Depicted are examples of inputting image data into an encoder model to generate embedded data representing the image data in accordance with various aspects of the present disclosure.

[0010] Fig. 9 Depicted are embedding data that may be used during training of a classifier model in accordance with various aspects of the present disclosure.

[0011] Fig.10Describes various aspects of the present disclosure that may be Fig. 9 Example training operations used during training of a classifier model.

[0012] Fig.11 Depicted is an example of score calculation using a logistic regression classifier according to various examples described herein.

[0013] Fig.12 Depicted are examples of loss functions that may be used in accordance with an exemplary implementation of a logistic regression classifier model.

[0014] Fig.13 Depicted is an exemplary calculation of a portion of a gradient based on a calculated loss according to various examples of the present disclosure.

[0015] Fig.14 Describes the use of various aspects of the present disclosure Fig.13 Example of performing a weight update iteration using a portion of the gradients computed in .

[0016] Fig.15 Depicted are examples of state determination using a trained classifier model in accordance with various aspects of the present disclosure.

[0017] Fig.16 Depicted are exemplary runtime processes that may be used to detect user-defined events based on image data in accordance with various aspects of the present disclosure.

[0018] Fig.17 Another exemplary runtime process that may be used to detect user-defined events based on image data in accordance with various aspects of the present disclosure is depicted.

[0019] Fig.18A is a block diagram illustrating an exemplary architecture of a computing device that can be used in accordance with various aspects of the present disclosure.

[0020] Fig.18B is a block diagram illustrating another exemplary architecture of a computing device that can be used in accordance with various aspects of the present disclosure.

[0021] Fig.19 is a diagram illustrating an exemplary system for transmitting and providing data that can be used in accordance with the present disclosure. DETAILED DESCRIPTION

[0022] In the following description, reference will be made to the accompanying drawings showing several examples of the present invention. It should be understood that other examples may be utilized and various operational changes may be made without departing from the spirit and scope of the present disclosure. The following detailed description should not be considered restrictive, and the scope of the embodiments of the present invention is limited only by the claims of the issued patents.

[0023] Systems, techniques, and interfaces for user-customizable computer vision-based event detection are described herein. For example, a user may define two or more different states associated with an event of interest relative to image data captured by one or more camera devices. An example may be a camera device monitoring a tool shed in a user's backyard. A user may define an event as the opening or closing of a tool shed door. Thus, a first state of the event may be the tool shed door opening. Thus, a plurality of user-provided images may be used to define the first state, wherein the tool shed door is depicted as being in an open state. Similarly, a second state of the event may be the tool shed door closing. Thus, a plurality of user-provided images may be used to define the second state, wherein the tool shed door is depicted as being in a closed state.

[0024] A computer interface (e.g., provided by a mobile application and / or a companion application for a camera device) is provided through which a user can define a custom event with two or more states, as in the example provided above. As described in further detail below, the interface can describe various actions taken by a user in order to provide custom event detection using computer vision. For example, the interface can guide a user through naming an event, naming the various states that define an event, and capturing and / or uploading multiple images representing each state of the event. In addition, the interface can control other systems that can evaluate the data provided by the user to determine whether it is sufficient to provide custom event detection and can provide feedback to the user about custom event detection. The interface can allow a user to customize the resulting actions that occur when a custom event is detected (e.g., when a tool shed door has been transformed from a closed state to an open state, or vice versa). Various examples of such computer interfaces and their functionality will be described in further detail below.

[0025] In response to receiving a request to generate a custom event trigger and monitor the occurrence of a custom event (as described above), the interface may cause a computer vision-based event detection system to be instantiated in response to the request. The computer vision-based event detection system may be executed in whole or in part by a camera device, by one or more computing devices remotely located relative to the camera device, and / or by some combination thereof. The computer vision-based event detection system may include a first machine learning model for encoding input image data (e.g., received from and / or generated by a related camera device) to generate corresponding embedded data for each input frame representing the image data. The embedded data representing each input frame may be used by a second machine learning model. The second machine learning model may be trained using user-provided images to classify each input frame into one of the user-defined states corresponding to the event (e.g., tool_shed_door=open or tool_shed_door=closed). Thus, the second machine learning model may be a custom model for a single event defined by a single user. The second machine learning model may be associated with the camera device and / or the user in some manner. For example, the second machine learning model may be associated with account data of the user and / or the camera device. In some examples, the second machine learning model may be looked up from a storage device using account data and / or other identifier data associated with the camera device.

[0026] The second machine learning model can classify a given image embedding as being associated with a particular state of a user-defined event (e.g., using a score and an adjustable threshold score). The determined state can be stored in a memory. In some cases, the second machine learning model can determine the state using multiple input frames of image data (e.g., by averaging the scores generated by the second machine learning model for multiple input frames (e.g., 5 consecutive frames of video, 10 consecutive frames of video, etc.) and by comparing the average score to an adjustable threshold). Thereafter, when the latest frame evaluated by the second machine learning model is used to determine the current state, the current state can be compared to the latest generated state (stored in the memory) to determine whether a state change has occurred (e.g., whether a tool shed door has transitioned from a closed state to an open state). If a state change has occurred, a user-defined action can be taken. For example, a user can provide instructions through an interface so that when the state changes from tool_shed_door=closed to tool_shed_door=open, an alarm is generated and sent to the user via a short message service (SMS) message. In another example, the user may provide a different set of instructions through the interface such that when the state changes from tool_shed_door=open to tool_shed_door=closed, an in-application alert is generated but an SMS message is not sent (because the user may care more about the tool shed door being open than the tool shed door being closed). In some examples, the thresholds for distinguishing different user-defined states may be determined empirically and / or may be adjusted through the interface to reduce false positive event detection triggers. In at least some examples, the custom machine learning model (e.g., the second machine learning model in the aforementioned example) may be a relatively lightweight model (in terms of the number of required parameters) such that the model can be quickly deployed for each different user and / or user-defined custom event and minimizes computing resource consumption. In addition, lightweight models may be able to be deployed on edge devices, while resource-intensive models may require data center-level computing resources. For example, the custom machine learning model may be implemented as a logistic regression model, a perceptron (or other relatively shallow neural network), a lightweight tree-based model (e.g., LightGBM), and the like.

[0027] Machine learning techniques, such as those described herein, can be used to form predictions, solve problems, identify objects in image data for classification, and the like. For example, machine learning techniques can be used to detect objects represented in image data and / or translate text from one spoken language to another. In various examples, machine learning models can perform better than rule-based systems and can be more adaptable because machine learning models can be improved over time by retraining the models as more data becomes available. Therefore, machine learning techniques are often able to adapt to changing conditions. Deep learning algorithms, such as neural networks, are often used to detect patterns in data and / or perform tasks.

[0028] Generally speaking, in machine learning models such as neural networks, parameters control the activations in neurons (or nodes) within a layer of the machine learning model. The weighted sum of the activations of each neuron in the previous layer can be input to an activation function (e.g., a sigmoid function, a rectified linear unit (ReLu) function, etc.). The result determines the activation of neurons in subsequent layers. In addition, a bias value can be used to shift the output of the activation function to the left or right on the x-axis, and thus can bias the neuron toward activation.

[0029] Generally speaking, in a machine learning model such as a neural network, after initialization, annotated training data can be used to generate a cost or "loss" function that describes the difference between the expected output and the actual output of the machine learning model. The parameters (e.g., weights and / or biases) of the machine learning model can be updated to minimize (or maximize) the cost. For example, the machine learning model can use a gradient descent (or ascent) algorithm to gradually adjust the weights so that the output of the loss function decreases (or increases) as quickly as possible. The method of updating the parameters of a machine learning model is often referred to as backpropagation.

[0030] A transformer model is a machine learning model that typically includes an encoder network and a decoder network. The encoder obtains input and generates a feature representation of the input (e.g., a feature vector, a feature map, etc.). The feature representation can then be fed into a decoder that can generate output embedding data based on the encoding. For example, in natural language processing, a transformer model takes a sequence of words as input. The transformer can receive sentences and / or paragraphs including a sequence of words as input. In some examples, the transformer can receive time series data (e.g., frames of a video) captured at various different times as input. In various examples described herein, a visual transformer (e.g., ViT) can receive frames of two-dimensional image data (e.g., from a video) as input, and the input data can be encoded into a numerical vector (e.g., embedding data).

[0031] The encoder network of the transformer includes a set of encoding layers that process the input data layer by layer. Each encoder layer generates a code (referred to herein as a "token"). These tokens include feature representations (e.g., feature vectors and / or mappings) that include information about which parts of the input data are related to each other. For example, for an input image, the encoder layer can determine which parts of the image are related to other parts of the image (and / or in some cases, related to other images received as part of the input data). Each encoder layer passes its token output to the next encoder layer. The decoder network obtains the tokens output by the encoder network and processes them using the encoded context information to generate an output (e.g., a one-dimensional vector of the tokens mentioned above). The output data can be used to perform task-specific functions (e.g., object detection, speed prediction, the next word in a sentence, etc.). In order to encode context information from other inputs (e.g., a combined feature representation), each encoder layer and decoder layer of the transformer uses an attention mechanism that weights the relevance of every other input for each input and extracts information from other inputs to generate an output. Each decoder layer also has an additional attention mechanism that extracts information from the output of the previous decoder before the decoder layer determines information from the encoding. Both the encoder and decoder layers have feed-forward neural networks for additional processing of the outputs and contain residual connections and layer normalization steps.

[0032] Scaled Dot Product Attention

[0033] The basic building block of the transformer is the scaled dot-product attention unit. When input data is passed into the transformer model, attention weights between each token are calculated simultaneously. The attention unit produces an embedding for each token in the context that contains not only information about the token itself, but also a weighted combination of other related tokens weighted by the attention weight.

[0034] Specifically, for each attention unit, the Transformer model learns three weight matrices; namely, the query weight W Q , key weight W K Sum weight W V For each token, the input embedding x i Multiply by each of the three weight matrices to produce the query vector q i =x i W Q , key vector k i =x i W K Sum value vector v i =x i W V The attention weights are calculated using the query vector and the key vector: the attention weights from the token aij is qi and k j Divide the attention weight by the square root of the key vector dimension This stabilizes the gradients during training, and the weights are then normalized to the sum by the softmax layer. Q and W K are different matrices, so attention can be asymmetric: if token i attends to token h, this does not necessarily mean that token i will attend to token i. The output of the attention unit for token i is the weighted sum of the value vectors of all tokens, with weights a ij , that is, the attention from i to each token.

[0035] The attention calculation for all tokens can be represented as a large matrix calculation, which is useful for training because matrix operations are optimized to make them fast. The matrices Q, K, and V are defined as where the i-th row is the vector q i , k i and v i The matrix of .

[0036]

[0037] Multi-head attention

[0038] A group (W Q , W K , W V ) matrices may be referred to as attention heads, and each layer in a Transformer model may have multiple attention heads. When one attention head focuses on other tokens that are relevant to each token, with multiple attention heads the model can learn to do this for different definitions of "relevance". The relevance encoded by the Transformer can be interpreted by humans. For example, in a natural language context, for each token there is an attention head that focuses primarily on the next word, or an attention head that focuses primarily on going from a verb to its direct object. Because Transformer models have multiple attention heads, they have the potential to capture multi-level and multi-type relevance relationships, from superficial to semantic. Multiple outputs of the multi-head attention layer are cascaded to be passed into the feed-forward neural network layer.

[0039] Each encoder consists of two main components: a self-attention mechanism and a feedforward neural network. The self-attention mechanism takes a set of input encodings from the previous encoder and weights the correlations between them to generate a set of output encodings. The feedforward neural network then further processes each output encoding individually. These output encodings are ultimately passed to the next encoder as its input, as well as the input to the decoder.

[0040] The first encoder takes as its input position information (e.g., macroblock position in the context of image data) and an embedding of the input data, instead of encoding. The transformer uses the position information to exploit the order of the input data. In the various examples described herein, the position embedding can describe the spatial relationship of multiple tokens relative to other tokens. For example, an input token can represent a 16x16 pixel block of an input frame from image data. The position embedding can describe the position within the frame of the 16x16 pixel block (e.g., relative to other tokens representing other parts of the frame). Therefore, the various techniques described herein are not one-dimensional position embeddings (as in the context of natural language, where the position of words in a one-dimensional sentence / paragraph / document is defined), but describe two-dimensional and / or higher-dimensional position embeddings, describing the spatial position of the token within the input data (e.g., a two-dimensional position within a frame, a three-dimensional position within a point cloud, etc.) and / or the temporal position of the token (e.g., a timestamp and / or frame number within a video). In various examples, the first encoder can learn a position embedding for each token in a frame of input data (e.g., 2D input data and / or 3D input data), and a position embedding for each timestamp (e.g., the temporal position of a token when the input data is time series data (such as frames of video and / or image data captured over time)). The sum of the position embedding and the timestamp can then be used to generate an embedding representing the position and temporal information of the token.

[0041] Each decoder layer consists of three components: a self-attention mechanism (e.g., scaled dot-product attention), an encoder-attention mechanism, and a feed-forward neural network. The decoder works in a similar way to the encoder, but with an additional attention mechanism inserted that extracts relevant information from the encoding produced by the encoder. In a self-attention layer, the keys, values, and queries come from the same place - in this case, the output of the previous layer in the encoder. Each position in the encoder can attend to all positions in the previous layer of the encoder. In an "encoder-decoder attention" layer (sometimes called "criss-cross attention"), the query comes from the previous decoder layer, while the keys and values ​​come from the output of the encoder. This allows each position in the decoder to attend to all positions in the input sequence. The decoder is attending to the encoder features.

[0042] In some examples described herein, a pre-trained visual transformer (e.g., ViT described by Dosov itskiy et al. in “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale”; https: / / arxiv.org / abs / 2010.11929) and / or another image-based transformer may be used to generate an embedding representing one or more frames of input image data. As described in further detail below, the resulting embedding may be used to determine whether a particular user-defined event has occurred over time, where the event is represented by the input image. In some examples discussed herein, a transformer-based encoder is used to generate embedding data representing a frame of image data. However, it should be noted that other image encoders may be used instead. For example, a neural network model (such as a convolutional neural network (CNN), a pre-trained CNN, a pre-trained neural network, a pre-trained graph neural network model, a pre-trained capsule neural network model, a pre-trained support vector machine model, a pre-trained model utilizing visual transformations and / or convolutions, and / or some other image encoder network) may be used instead to generate embedding data representing an input image. In some examples, lightweight models such as CNNs may be advantageous when such models need to be deployed on edge devices. However, the embedded data generated using transformer-based models may include richer semantic information, which can provide enhanced capabilities for downstream actions.

[0043] Figure 1 is a diagram depicting an exemplary system 100 for efficiently providing user-customized computer vision event detection in accordance with various aspects of the present disclosure. Figure 1 In the example depicted in , camera device 150 may have a field of view that covers an area of ​​interest for a particular user. Figure 1 In the example of , camera device 150 generates image data 102a, 102b depicting a garage (when camera device 150 captures an image of a physical environment including a garage). The garage may be an area of ​​interest for a user. Thus, the user may have created a custom event (e.g., using an interface provided by a companion application executed by user device 130). The companion application may be associated with camera device 150. In various examples, a user may create a secure account that associates user device 130 and camera device 150. In at least some examples, the companion application may enable the user to view images captured by camera device 150, such as image data 102a, 102b.

[0044] As described in further detail below, a user can create a custom event using an interface of a companion application executed on the user device 130 (or on another network-connected computing device). In various examples, the custom event can determine whether the image data captured by the camera device 150 represents a user-defined state change. As described in further detail below, a user can generate a first plurality of images (e.g., a first plurality of frames of image data) representing a first state associated with the custom event. In the current example, the first state can be a state in which the garage door is open (as shown in the image data 102b). Additionally, a user can generate a second plurality of images (e.g., a second plurality of frames of image data) representing a second state associated with the custom event. In the current example, the second state can be a state in which the garage door is closed (as shown in the image data 102a). Although reference is made to Figure 1 The specific example shown and described describes two states, but more states may be used depending on the specific custom event being created.

[0045] As described in further detail below, the custom classifier model 108 can be a machine learning model trained at least in part based on images representing each state of a custom event. For example, a pre-trained encoder model 104 (e.g., ViT, CNN, or any other model that generates embedded data representing an image) can be used to generate embedded data (e.g., feature maps, vectors, etc.) representing an input image. The input image and / or the embedded data representing the input image can be marked with data identifying the corresponding state represented by the image (e.g., garage door open or garage door closed). For example, the supporting application can prompt the user (via a user interface) with a set of images (e.g., 5 images, 10 images, 2 images, etc.) having a first user-defined state. The user can provide an image corresponding to the first user-defined state (e.g., garage door open). Similarly, the supporting application can prompt the user (via a user interface) with images corresponding to a second user-defined state and / or each other state of a custom event.

[0046] The custom classifier model 108 can be trained in a supervised manner using such labeled image / embedded datasets to classify the input image according to the state represented in the image. For example, if the custom classifier model 108 is implemented as a logistic regression classifier, state_A (garage door closed) can be assigned a ground truth value of 1, while state_B (garage door open) can be assigned a ground truth value of 0. For a given input image, the encoder model 104 can generate embedded data 106 (e.g., a column vector) representing the image. The embedded data 106 can be input to a linear regression model, and the linear regression model can generate a score 112. Generally speaking, in the current example, if the score is closer to 1, the input image can be classified as state A, while a score closer to 0 can cause the input image to be classified as state B (at action 114). In some cases, an F1 optimal threshold score (or another threshold score) can be determined, and the score 112 is used by action 114 to classify the state of the input image. In some examples, multiple thresholds can be used. For example, a first threshold (e.g., a "high" threshold) may be used to determine whether an input image represents a first state (e.g., whether the score 112 of the input image is above the high threshold), while a second threshold (e.g., a "low" threshold) may be used to determine whether the input image represents a second state (e.g., whether the score 112 of the input image is below the low threshold). In some examples, a corresponding threshold may be defined for each state, while in other cases, the threshold may define a score partition between different states. Many different thresholding techniques are possible and may be used depending on the desired implementation. In various examples, the custom classifier model 108 may be implemented as a logistic regression model, a neural network model (such as a convolutional neural network classifier model), a graph neural network model, a graph convolutional neural network model, a support vector machine model, a visual transformer model, and / or another classifier model.

[0047] After training the custom classifier model 108, the system 100 can effectively detect custom events based on changes in the state of the region of interest. In one example, the input image data 102a can represent multiple frames of image data that depict a garage door (the region of interest of the custom event in question) as closed. The input image data 102a can be a video frame captured within a predetermined amount of time (e.g., 5 minutes, 30 seconds, 1 hour, etc.). In some examples, the input image data 102a can be pre-processed before being input into the encoder model 104. For example, the region of interest (e.g., the garage door) can be cropped from the background to generate cropped image data. For example, during the setting of the custom event trigger, the user may be able to define the region of interest within the image frame. For example, the user can select the region of interest by moving the vertices of the polygon and / or adjusting the edges of the polygon so that the polygon covers the region of interest within the sample frame. Thereafter, each input image data 102a, 102b can be cropped to generate cropped image data depicting the user-defined region of interest before being input into the encoder model 104.

[0048] After preprocessing, the input image data 102a can be input into the encoder model 104. The encoder model 104 can generate embedding data 106 for each input frame of the image data. The embedding data 106 can be, for example, a vector representing a given frame of the input image data 102a. The embedding data 106 can be input into the custom classifier model 108. As previously described, the custom classifier model 108 can be trained to generate a score 112 for each input embedding (e.g., embedding data 106). The score 112 can be used to classify each frame according to one of the user-defined states. For example, the score 112 can be compared with a threshold score. In the current example, if the score 112 is higher than the threshold score at action 114, it can be determined that the current state of the input image corresponds to a closed garage door. In various examples, a score 112 can be generated for each image in a group of images (e.g., five frames of image data or other number of frames of image data, depending on the desired implementation). The average of the score 112 can be determined, and the average score can be compared with a threshold at action 114 to determine the state represented by the group of images. In either case, the current state data 122 may represent the current state of the input image data 102 a as predicted by the custom classifier model 108 .

[0049] At action 116, it may be determined whether a state change has occurred between the current state (represented by current state data 122) and the most recent state predicted by the custom classifier model. In an example, the input image data 102a may be an initial input image after the custom classifier model 108 has been trained to detect user-defined events. Therefore, no historical state data may not yet be stored in the non-transitory computer-readable memory 103. In this example, the current state data 122 (indicating that the garage door is closed) may be stored in the non-transitory computer-readable memory 103 as the most recent state data 110 (e.g., previous state data determined at a previous time step).

[0050] The input image data 102b may depict a garage door as open, and may be captured after the input image data 102a (e.g., after the garage door has been opened). The image data 102b may be pre-processed (e.g., a region of interest may be cropped from the input image data 102b to generate cropped image data), and may be input into the encoder model 104 to generate corresponding embedding data 106 for each frame of the input image data 102b. The embedding data 106 may be input into the custom classifier model 108. The custom classifier model 108 may generate a score 112 for each input embedding. The scores 112 may be averaged and compared to a relevant threshold score (at action 114). It may be determined that the new current state data 122 is the garage_door_open state. At act 116, the most recent state data 110 (indicating the garage_door_closed state determined for the input image data 102a) determined at the most recent time step may be retrieved from the non-transitory computer readable memory 103 and compared with the current state data 122 (indicating the garage_door_open state). Thus, it may be determined at act 116 that the state has changed from the garage_door_closed state to the garage_door_open state.

[0051] At action 124, there may be an action associated with the state change of the user-defined event. For example, the user may have provided instructions indicating that an alert should be pushed to the user's user device 130 when an event is detected indicating that the garage door has transitioned from a closed state to an open state. Figure 1In the example of , an alert is shown on the user device 130. It should be noted that different output actions can be configured for different state changes. For example, if there are three related states A, B, and C, there may be different output actions for state changes from state A to state B, from state B to state C, from state A to state C, from state C to state A, from state B to state A, and / or from state C to state B. For example, some output actions may be in-app alerts, while other actions may use push notifications, SMS messages, emails, and the like.

[0052] In some examples, Figure 1 The system 100 depicted in the can be configured via a mobile application to detect states at a user-defined cadence. For example, a user can instruct the system to monitor state changes every minute, every 5 minutes, every hour, every 10 seconds, etc. In some cases, if the user does not specify a specific frequency for state change detection, a default cadence can be used. When a state change needs to be detected, the camera device 150 can capture a predefined number of frames of image data and send it to the system for state classification and state change detection, as described herein.

[0053] It should be noted that the exemplary custom events and user-defined states associated with garage doors are merely illustrative examples. A user can define custom events for various event triggers. For example, a custom event can alert a user when a car enters or leaves a user's driveway (a first state showing that at least a portion of the vehicle is in the driveway, and a second state showing that the driveway is empty). Other examples may include removing or adding a cover to a swimming pool, removing a bicycle from an area of ​​interest, removing trash from an area of ​​interest, and so on. In various examples, the system 100 can prevent a user from tracking a person, and any training images and / or user-defined states that include images of people (or animals) may be rejected for privacy reasons. In some cases, any training images that include humans, cars, and / or any other prohibited categories may be detected and prohibited to prevent the generation of custom events that track such entities (for privacy reasons).

[0054] Figure 2 An exemplary process 200 for training and runtime operation of system 100 performed by user device 130 in accordance with various aspects of the present disclosure is depicted. For clarity and brevity, Figure 2 Previously referenced Figure 1 Actions that have been described may not be described here again. Figure 2The actions of the process depicted in the flowchart may represent a series of instructions including computer-readable machine code that can be executed by one or more processing units of one or more computing devices. In various examples, the computer-readable machine code may be composed of instructions selected from the native instruction set and / or operating system (or multiple systems) of the one or more computing devices. Although the figures and discussion illustrate certain operating steps of the system in a particular order, the steps described may be performed in a different order (and certain steps may be deleted or added) without departing from the purpose of the present disclosure.

[0055] Process 200 may begin with act 202, where user device 130 may receive user input representing a selection of a portion of a frame captured by a camera device (e.g., camera device 150). For example, a user may define a region of interest within a frame by dragging the vertices of a polygon so that the polygon surrounds the region of interest of a user-defined custom event. For example, Figure 3 Depicted is a user's selection of a region of interest 302 (e.g., a tool shed door) in a frame of image data. Figure 3 As shown, the companion application can display instructions for adjusting the region of interest to cover the area that the user wishes to be monitored for the purpose of customized event detection.

[0056] Process 200 may continue at act 204 where user device 130 generates data representing a defined area (e.g., a region of interest) of camera device 150 based on the user input representing a selection of a portion of the frame. For example, the data may represent a region of interest (e.g., a garage door) that may be detected and / or cropped from a subsequently captured frame.

[0057] Process 200 may continue at act 206, where the user device 130 receives user input representing a selection of five frames corresponding to the first state and five frames corresponding to the second state. For example, a companion application of the user device 130 and / or the camera device 150 may prompt the user to input five images (or other number of images, depending on the implementation) that define each state of the custom event. These images may be selected from images captured by the camera device 150 and / or may be viewed via a companion application that is at least partially executed on the user device 130. The user may select five representative images that represent each user-defined state. For example, five images showing a garage door in a closed state may be selected and labeled “Garage door closed.” Similarly, five images showing a garage door in an open state may be selected and labeled (via the companion application) “Garage door open.” For example, if Figure 4A As shown, the companion app can provide fields to allow the user to name each state of a custom event. Figure 4BAs shown, the companion application can display a preview (e.g., a thumbnail) of the user-provided frame representing the first state. The user can selectively delete any of the frames and select a new frame. In some examples, the frame can be selected by the companion application during the setup process of custom event detection.

[0058] Return to Figure 2 , processing may continue at action 208, where the user device 130 sends data representing the defined region (e.g., data representing the region of interest) based on the selected portion and the selected frame corresponding to the user-defined state. For example, the data may be sent to one or more computing devices of the system 100 (e.g., computing devices that implement the encoder model 104, the custom classifier model 108, and / or logic associated with various operations of the system 100 described herein). As previously described, the custom classifier model 108 may be trained based at least in part on the selected frame corresponding to the user-defined state to distinguish between different user-defined states of a given input frame of image data depicting the region of interest.

[0059] Actions 210 and 212 represent runtime actions that may be performed once the encoder model 104 is trained based at least in part on user-provided images representing user-defined states. At action 210, the user device 130 may receive data representing an indication that a state change has been detected between a first state and a second state. For example, the system 100 may determine a current frame of image data and / or a set of current frames of image data representing a second user-defined state. However, the system 100 may determine that in the most recent time step before the current time step, the region of interest was in the first user-defined state. Therefore, the region of interest has transitioned between the first state and the second state.

[0060] At action 212, the user device 130 may take a specific action in response to the detected change in state. The action may be a user-defined action, such as displaying an alert indicating the change in state. For example, the alert may be a push notification that reads "The swimming pool gate is open."

[0061] Figure 5 An exemplary evaluation of training images to determine whether training should continue according to various aspects of the present disclosure is depicted. As described in an exemplary user interface 510 of a companion application, it can be determined whether a user-provided image representing each user-defined state of a custom event is sufficient to train a custom classifier model 108 ( Figure 1 ).

[0062] exist Figure 5In the example depicted in , input frame 502a can be a user-provided image representing a first state, and input frame 502b can be a user-provided image representing a second state. Input frame 502a and input frame 502b can be input into encoder model 104, and encoder model 104 can generate embedding data 504a representing input frame 502a and embedding data 504b representing input frame 502b. For example, a separate embedding can be generated for each frame in input frame 502a, and embedding data 504a can be the average value of these embeddings (e.g., the average value of each element of a vector). Similarly, embedding data 504b can be represented as the average value of the embeddings generated for each input frame 502b.

[0063] The system 100 can evaluate the input frame 502a and the input frame 502b to determine whether they are sufficient to continue training the custom classifier model 108. The evaluation is based on two basic principles. First, the images in the user-provided images representing a single user-defined state should be similar to each other. Therefore, each frame in the input frame 502a should be similar to each other because they are each intended to represent the same user-defined state. Various techniques can be used to determine the similarity between the embeddings of the input frames. For example, cosine similarity and / or standard deviation of values ​​from the mean can be determined. Similarity thresholds can be used to ensure that the frames within the input frame 502a are sufficiently similar to each other and the frames within the input frame 502b are sufficiently similar to each other. Secondly, the image representing one state (e.g., input frame 502a) should be very different from the image representing another state (e.g., input frame 502b). Therefore, the embedded data 504a can be compared with the embedded data 504b to ensure that there is a sufficient distance between the embedding vectors (e.g., greater than a threshold Euclidean distance and / or less than a threshold cosine similarity).

[0064] In some examples, if the intra-frame similarity between user-provided frames corresponding to a single user-defined state satisfies a related similarity threshold, and the aggregated inter-frame differences between frames defined for different states satisfies a related difference threshold, then training of the custom classifier model 108 can continue. Otherwise, the user can be prompted via the companion application for additional frames that meet the above criteria.

[0065] Various other techniques can be used to determine whether a user-provided frame can be used to train a custom classifier model 108. For example, a classifier model (e.g., a logistic regression model) can alternatively be used to determine whether a training image is sufficient for training. For example, ten images of a particular state can be received (e.g., via a companion application). Nine of the ten images can be used to train a classifier model (e.g., a logistic regression model). The trained model can be used to predict the state of the tenth image. This can be done any number of times (e.g., ten times), with a different image retained for testing each time training. If the test is successful (e.g., if a performance metric for a training iteration is met (e.g., a confidence score is greater than a threshold), the image data may pass the comparison check and may be used to train the custom classifier model 108. If the test is unsuccessful (e.g., a performance metric fails, such as when the average confidence score for ten iterations is less than a threshold), the comparison check may fail, and the user may be prompted for additional images of the state before training of the custom classifier model 108 proceeds. Additionally, before such images are used to train the custom classifier model 108, a check may be performed to ensure that the training images do not violate any other rules. For example, a check may be performed to determine whether any person, car, or other prohibited object is detected in the training image. If so, an appropriate alert may be displayed to remind the user that such items are not prohibited in the training image for custom event detection. The alert may prompt the user to select additional images representing one or more states of the custom event.

[0066] Fig. 6A and Figure 6B Depicted are exemplary training processes that can be used to train a system to provide user-customized computer vision event detection according to various aspects of the present disclosure. For clarity and brevity, FIG. 6A to FIG. 6B Previously referenced Figures 1 to 5 Actions that have been described may not be described here again. FIG. 6A to FIG. 6B The actions of the process depicted in the flowchart may represent a series of instructions including computer-readable machine code that can be executed by one or more processing units of one or more computing devices. In various examples, the computer-readable machine code may be composed of instructions selected from the native instruction set and / or operating system (or multiple systems) of one or more computing devices. Although the figures and discussion illustrate certain operating steps of the system in a particular order, the steps described may be performed in a different order (and certain steps may be deleted or added) without departing from the purpose of the present disclosure. Fig. 6A , Figure 6BAs shown, some actions (e.g., those boxes appearing below user device 130) can be performed by user device 130, while other actions (e.g., those boxes appearing below remote system 630) can be performed by a system remote from user device 130 and / or camera device 150.

[0067] Processing may begin at act 602, where user device 130 may receive user input representing a selection of a portion of a frame captured by camera device 150. As previously described, a user may define a region of interest within a frame that defines an area to be monitored for a user-defined custom event. Processing may continue at act 604, where user device 130 may generate data representing a defined area (e.g., a region of interest) of camera device 150 based on the user input representing the selection of a portion of a frame. For example, the data may represent a region of interest (e.g., a garage door) that may be detected and / or cropped from a subsequently captured frame.

[0068] Processing may continue at action 606, where the user device may receive user input representing a selection of five frames corresponding to the first state and five frames corresponding to the second state. For example, the companion application of the user device 130 and / or the camera device 150 may prompt the user to enter five images (or other number of images, depending on the implementation) for each state that defines the custom event. These images may be selected from images captured by the camera device 150 and / or may be viewed through a companion application that is at least partially executed on the user device 130. The user may select five representative images that represent each user-defined state. For example, five images showing a garage door in a closed state may be selected and labeled "Garage door closed". Similarly, five images showing a garage door in an open state may be selected and labeled (by the companion application) as "Garage door open".

[0069] Processing can continue at action 608, where the user device 130 sends data representing a defined area (e.g., data representing a region of interest) based on the selected portion and the selected frame corresponding to the user-defined state to the remote system 630. In various examples, the remote system 630 may include one or more computing devices that implement the encoder model 104, the custom classifier model 108, and / or logic associated with various operations of the system 100 described herein. As previously described, the custom classifier model 108 can be trained based at least in part on the selected frame corresponding to the user-defined state to distinguish different user-defined states for a given input frame of image data depicting a region of interest.

[0070] Processing can continue at act 610, where the remote system can compare frames to determine whether to begin training. For example, the remote system 630 can compare frames corresponding to a single state to each other to ensure similarity, and can compare frames of different states to each other to ensure a sufficient degree of difference, as described above with reference to Figure 5 If training should not yet begin (based on the frames defining a single state being too different and / or based on the frames between two different states being too similar), processing can return to act 606 and the user device 130 can indicate the problem and / or can prompt the user to select additional frames to correct the problem.

[0071] According to one or more preferred embodiments, for each corresponding frame in a set of frames that have been provided by a user with an associated user-defined state (e.g., five frames associated with a first state and five frames associated with a second state), the comparison process may involve using the remaining frames in the set of frames to train a corresponding machine learning model (e.g., a logistic regression model), then using the trained corresponding machine learning model to determine the state of the corresponding frame (e.g., determine a score value and compare it to a threshold value (e.g., 0.5)), and then comparing the determined state with the state identified by the user as being associated with the corresponding frame (e.g., a label of the corresponding frame). The comparison process may involve performing this operation on each corresponding frame in the set of frames, and if a threshold number of frames (e.g., ten frames out of ten frames, or nine frames out of ten frames, or eight frames out of ten frames, etc.) result in a state determination that matches the label provided by the user, then determining that the set of frames passes the comparison test.

[0072] If, on the other hand, it is determined at act 610 that training should begin, processing may continue to act 612 ( Figure 6B ), at which action, the remote system 630 can receive data representing the user-defined region of interest and a user-selected frame of image data representing each state of the custom event. Processing can continue at action 614, where the remote system 630 can crop each of the received image frames corresponding to the user-defined state according to the data defining the region of interest (e.g., the region defined by the user as described above). After cropping, the frame of image data can be described as a "cropped" frame of image data.

[0073] Processing can continue at action 616, where the remote system 630 can use the cropped frames of image data to generate additional frames for training the custom classifier model 108. For example, if the user provides five frames corresponding to the first state and five frames corresponding to the second state, the remote system can generate additional synthetic frames of image data corresponding to each state to generate a sufficient amount of training data to effectively train the custom classifier model 108. Various techniques are known for generating synthetic images for training machine learning models, and any suitable technique can be used as long as the synthetically generated images for each state maintain the visual characteristics associated with the different states. For example, the input frames can be rotated, padded, color enhanced, and / or other image modification techniques applied to generate additional synthetic training images.

[0074] Processing can continue at action 618, where the remote system 630 can use the original user-selected frame for each user-defined state to generate a trained visual transformer (or other instantiation of the encoder model 104) and multiple embeddings for each state. As described above, for a given frame of image data (including a cropped frame representing a region of interest), the encoder model 104 can generate embedding data representing the frame. In various examples, the encoder model 104 can be a pre-trained encoder model (e.g., a pre-trained ViT).

[0075] Processing can continue at action 620, where the remote system 630 can train a logistic regression model (or other instantiation of the custom classifier model 108) to classify the input frame as belonging to the first state or the second state based on the input embedded data. For example, the logistic regression model or other instantiation of the custom classifier model 108 can receive as input the embedded data of the input frame representing the image data, and can classify the embedded data (and, by extension, the input frame) as belonging to the first state or the second state. As previously described, the logistic regression model (or other instantiation of the custom classifier model 108) can be trained in a supervised manner using user-provided labeled training images (and synthetic training images generated therefrom).

[0076] Processing can continue at action 622, where the remote system 630 can determine an optimized threshold for the logistic regression model. For example, an optimal F1 threshold can be determined to distinguish between the first state and the second state. Thereafter, each score output by the logistic regression model can be compared to the optimized threshold score to classify the input image as belonging to a particular state. As previously described, multiple thresholds can be used depending on the desired implementation. For example, a first threshold (e.g., a high threshold) can be used to detect the first state (e.g., if the score output by the custom classifier model 108 meets or exceeds the high threshold), while a second threshold (e.g., a low threshold) can be used to detect the second state (e.g., if the score output by the custom classifier model 108 is equal to or below the low threshold).

[0077] Figure 7 706. The cropping operation of a cropped frame for generating image data according to various aspects of the present disclosure is represented. For example, the remote system 630 may receive an input frame 702 representing a house and trees. The remote system 630 may determine identifier data identifying the camera device 150 from which the input frame 702 was received. The remote system 630 may determine data representing an area of ​​interest 704 (e.g., a polygon around a door) associated with the identifier data. The remote system 630 may crop the input frame 702 by removing image data other than image data within the area of ​​interest 704 to generate cropped image data 706.

[0078] Figure 8 An example of inputting image data into an encoder model to generate embedding data representing the image data according to various aspects of the present disclosure is depicted. As shown, the image data may include RGB pixel values ​​for each pixel captured by an image sensor of camera device 150. Encoder model 104 may be a trained model, such as a trained ViT model, a trained CNN, and / or another model trained to generate a vector embedding (e.g., image 1 embedding) from input image data (e.g., image 1 data representing frame 1).

[0079] Fig. 9Depicted are embedded data 906 that can be used during training of a classifier model 908 in accordance with various aspects of the present disclosure. The classifier model 908 can be an untrained logistic regression model, a neural network, and / or other machine learning model that can be used to classify input image data according to its category. As described in further detail below, each vector of the embedded data 906 can represent a corresponding input image. Each input image (and each corresponding embedding) can be labeled with data identifying a state associated with the image. For example, a first image showing a garage door in a closed state can be labeled with ground truth data indicating that the first image represents a garage_door_closed state. Thus, the embedded data representing the first image can be correspondingly labeled with ground truth data identifying the state of the image. As described in further detail below, the labeled embedded data 906 can be used to train the classifier model 908. Since the labeled embedded data 906 corresponds to user-provided image data representing a user-defined custom event, the classifier model 908 can be customized for a particular user's event, and thus can be a user-specific and event-specific model (after training). After successful training, the user device may output a message or other confirmation data indicating that the custom classifier model 908 has been successfully trained and / or event detection between the first user-defined state and the second user-defined state has been enabled. Optionally, the user can use the companion application to toggle event detection between the active state and the inactive state.

[0080] During training, each image can be embedded into X i Input into the classifier model 908, such as Fig.10 As shown, to generate a score 112. For example, the embedded data representing the input image (e.g., Image 1) can be a vector. Fig.10 Transpose the vector to generate a row vector as shown in Matrix multiplication can be used to convert multiplied by a weight vector (e.g., column vector W) comprising parameters of the classifier model 908 to determine (For example, for Fig.10 , is -0.712). The score 112 may be determined using equation 1002 (for a logistic regression model instantiation of the classifier model 908):

[0081]

[0082] exist Fig.11 In the example depicted in , the score 112 has been calculated to be 0.329. Fig.12 An example of a loss function that can be used according to an exemplary embodiment of a logistic regression classifier model is depicted. In the example, y 实际 represents the digital representation of the ground truth state of each image, and y预测 Represents the score predicted by the model. For example, for the garage_door_closed state, the ground truth score could be 0, while for the garage_door_open state, the score should be 1. Fig.11 In the example, the prediction score for image 1 is 0.329 = y 预测 The ground truth score for image 1 may be 0 (indicating garage_door_closed state).

[0083] Fig.13 Describes the use of Fig.12 The loss function in Fig.11 An example of computing a portion of the gradient for image 1 in Figure 1. Fig.14 Describes the use of various aspects of the present disclosure Fig.13 An example of performing a weight update iteration using a portion of the gradients computed in . Fig.14 As shown, the weight vector W can be updated. Figures 10 to 14 One training iteration is depicted, however, multiple training iterations may be used to iteratively update the weights of the classifier model 908 until Fig.12 The loss defined by the loss function in is minimized. After training, the classifier model 908 can be customized for user-defined events (e.g., resulting in instantiation of the custom classifier model 108). In various examples, a K-fold check can be used during model training. In such examples, a first predefined proportion or number of training images can be used to train the model, while a second predefined proportion or number of training images can be retained for testing the model.

[0084] exist Fig.15 In , Image 1 is input to the custom classifier model 108 (e.g., as test data after training a logistic regression classifier). The score for Image 1 determined using the above techniques after training the custom classifier model 108 is 0.301. Fig.15 In the example, the threshold score is 0.5. Fig.15 In the exemplary code 1502, if the score output by the custom classifier model 108 is less than or equal to the threshold score, a custom status of 0 is returned, and if the score is greater than the threshold score, a custom status of 1 is returned. Fig.15 In the example, the score is less than 0.5, so a custom state of 0 is returned (e.g., garage_door_closed). Fig.15 As shown in the representation of image 1, the garage door is closed. Fig.15 In the example, custom classifier model 108 has predicted the correct user-defined state.

[0085] In various examples, after training, a plurality of scores can be generated using the training corpus (e.g., for cropped image data representing a user-defined region of interest in the training corpus). The scores can be used to determine an optimized threshold score. For example, the initial threshold score can be 0.5. For the first state, the optimized threshold score (e.g., F1 score) can be determined using:

[0086] Precision = True Positive Rate / (True Positive Rate + False Positive Rate)

[0087] Recall rate = true positive rate / (true positive rate + false positive rate)

[0088] F1 score = (2*precision*recall) / (precision+recall)

[0089] Where true positives are scores / images that were predicted to have a first state and did have the first state (as determined using ground truth data), false positives are scores / images that were predicted to have the first state but did not have the first state, and false negatives are scores / images that were predicted not to have the first state but did have the first state. In some examples, separate thresholds may exist for each user-defined state of a custom event.

[0090] Fig.16 An exemplary process 1600 for training and runtime operation of system 100 performed by remote system 630 in accordance with various aspects of the present disclosure is depicted. For clarity and brevity, Fig.16 Previously referenced Figures 1 to 15 Actions that have been described may not be described here again. Fig.16 The actions of the process depicted in the flowchart may represent a series of instructions including computer-readable machine code that can be executed by one or more processing units of one or more computing devices. In various examples, the computer-readable machine code may be composed of instructions selected from the native instruction set and / or operating system (or multiple systems) of the one or more computing devices. Although the figures and discussion illustrate certain operating steps of the system in a particular order, the steps described may be performed in a different order (and certain steps may be deleted or added) without departing from the purpose of the present disclosure.

[0091] Process 1600 can begin at action 1612, where remote system 630 can receive frame data from camera device 150. For example, remote system 630 can receive multiple frames of image data from camera device 150 at a predetermined frequency (e.g., every minute, every 5 minutes, every 30 seconds, etc.) to detect the occurrence of a user-defined event. Depending on the desired implementation, any number of frames of image data can be received. In other examples, a triggering event can cause remote system 630 to receive frames for custom state detection. For example, if camera device 150 detects motion (e.g., using computer vision techniques and / or motion sensors), streaming can be initiated and frames of image data can be sent from camera device 150 to remote system 630.

[0092] Process 1600 may continue at action 1614 where remote system 630 may repeatedly select a set of frames to evaluate state changes. For example, in a series of frames of image data received by remote system 630 from camera device 150, a sliding window of X frames may be evaluated to determine state changes.

[0093] Process 1600 may continue at action 1616, where the remote system 630 may use the visual transformer (or other instantiation of the encoder model 104), the trained logistic regression model (or other instantiation of the custom classifier model 108) to determine the average score for the current set of frames. For example, at action 1618, each frame of image data may be cropped to generate a cropped frame of image data representing a user-defined region of interest based on stored data representing regions in the frame corresponding to the region of interest. At action 1620, the visual transformer and / or other image encoder may be used to generate an embedding representing the input cropped image data (for each frame). At action 1622, the trained logistic regression model (or other custom classifier model) may receive the embedding as input and may determine a score for the embedding. This operation may be performed for each input frame.

[0094] At act 1624, the remote system 630 may determine the state of the set of frames by comparing the calculated average score (e.g., by averaging the scores for each frame in the set of frames (e.g., averaging each score for a sliding window of frames)) to a stored optimized threshold (e.g., optimized as described above). At act 1626, the remote system 630 may determine that the determined state of the set of frames is different from the determined state of a previous set of frames (e.g., a stored state representing the state of a previous set of frames). This may be accomplished in a variety of ways depending on the desired implementation. For example, the remote system 630 may determine a corresponding score 112 for each input frame of image data (e.g., over a window size of X frames). In one example, the corresponding scores 112 may be averaged, and the average score may be used to determine a state prediction for the set of frames. In another example, the state of each frame may be determined using a separate score for each frame. It may be determined whether there are consistent states among the set of frames (e.g., if all frames evaluated have the same state and / or greater than a threshold number of frames have the same state). The determined state of the set of frames can be compared to a most recent state (e.g., stored in memory) to determine whether a state change has occurred. In another example, a state change can be detected if a confidence score associated with an updated state (e.g., a new state relative to a most recently detected state) is determined to be greater than a threshold time period (and / or a threshold number of frames).

[0095] In various examples, the embedding can be stored in memory in association with a state determined about the stored embedding (e.g., door closed). The embedding can be used to verify the current state and / or the previous state. In some examples, the embeddings of different time steps can be compared to determine whether a state change has occurred between time steps (e.g., between motion detection events). Consider an example in which custom event detection is trained to detect a state change between a garage door opening and a garage door closing. Prior to this, the garage door may be closed. Embedding data representing an image of a garage door in a closed state can be stored in association with state data indicating that the garage door is in a closed state. At a later time, the garage door may be opened. However, in this example, a motion detection event may not be detected when the garage door is open (e.g., due to abnormal lighting conditions, network interruption, etc.). At some later time, a motion event can be detected, which can trigger a frame of image data to be evaluated using a custom event detection system. The frame of the image data being evaluated can be compared with the latest embedding and state data stored previously to determine whether a state change has occurred. If so, an alert can be sent to the companion application to notify the user that the state has changed.

[0096] In various examples, the confidence score output by the custom classifier model 108 can be used to determine state change detection. For example, a particular state detected by the custom classifier model 108 can be considered accurate only when the confidence score (generated by the custom classifier model 108 for prediction) and / or the average confidence score over a particular time period exceeds a confidence score threshold. In another example, the confidence score may be required to meet a confidence score threshold for each time step (e.g., each frame of image data) within a given time period (e.g., N frames) in order to detect a state change event. It should be noted that the specific thresholding technique to be used can be adjusted and / or selected according to the desired implementation. For example, requiring a state change confidence score to be higher than a threshold confidence score for each of the N frames can be a more stringent requirement for state change detection than determining that the average confidence score of the N frames (or another number of frames) exceeds a confidence score threshold. Therefore, different thresholding techniques can be selected according to the precision / recall requirements of a specific use case and / or implementation.

[0097] At action 1628, remote system 630 may perform an action in response to the detected state change. The action may be determined based on a particular state change (configured by the user when defining a custom event). For example, alert data indicating a state change may be sent to a user device associated with camera device 150.

[0098] Fig.17 An example process 1700 for training and runtime operation of the system 100 performed by the camera device 150 in accordance with various aspects of the present disclosure is depicted. For clarity and brevity, Fig.16 Previously referenced Figures 1 to 16 Actions that have been described may not be described here again. Fig.17 The actions of the process depicted in the flowchart may represent a series of instructions including computer-readable machine code that can be executed by one or more processing units of one or more computing devices. In various examples, the computer-readable machine code may be composed of instructions selected from the native instruction set and / or operating system (or multiple systems) of the one or more computing devices. Although the figures and discussion illustrate certain operating steps of the system in a particular order, the steps described may be performed in a different order (and certain steps may be deleted or added) without departing from the purpose of the present disclosure. Fig.17 is an example of various processing steps being performed by camera device 150 (e.g., encoding by encoder model 104 and / or classification by custom classifier model 108). However, as previously discussed with respect to Fig.16As described, these processing steps (and / or some of these processing steps) may alternatively be performed by remote system 630. The selection of a particular device and / or a set of particular devices to perform a particular processing step may depend on the particular implementation of the particular device employed and / or hardware-based constraints.

[0099] Process 1700 may begin at action 1712, where camera device 150 may receive data for a logistic regression model from remote system 630. For example, camera device 150 may receive instructions for executing a logistic regression model and / or may receive parameters of a trained logistic regression model that has been trained to detect user-defined events. In various other examples, camera device 150 may receive parameters of encoder model 104 (e.g., a trained instantiation of encoder model 104).

[0100] Processing may continue at action 1714, where camera device 150 may select a set of captured frames for state change detection. For example, in a series of frames of image data captured by camera device 150, a sliding window of X frames may be evaluated to determine a state change. In various examples, state change detection (e.g., user-defined custom event detection) may be triggered by an event, such as motion detection in a scene. For example, if motion is detected within a scene or within a region of interest defined by a user for custom event detection, a state change may be evaluated based on a user-defined custom event.

[0101] Processing may continue to act 1716, where the camera device 150 may use the visual transformer (or other instantiation of the encoder model 104), the trained logistic regression model (or other instantiation of the custom classifier model 108) to determine an average score for the current set of frames. For example, at act 1718, each frame of image data may be cropped to generate a cropped frame of image data representing a user-defined region of interest based on stored data representing regions in the frame corresponding to the region of interest. At act 1720, the visual transformer and / or other image encoder may be used to generate an embedding representing the input cropped image data (for each frame). At act 1722, the trained logistic regression model (or other custom classifier model) may receive the embedding as input and may determine a score for the embedding. This operation may be performed for each input frame.

[0102] At act 1724, camera device 150 may determine the state of the set of frames by comparing the calculated average score (e.g., by averaging the scores for each frame in the set of frames (e.g., averaging each score for a sliding window of frames)) to a stored optimized threshold (e.g., optimized as described above). At act 1726, camera device 150 may determine that the determined state of the set of frames is different from the determined state of a previous set of frames (e.g., a stored state representing the state of the previous set of frames).

[0103] At action 1728, camera device 150 may perform an action in response to the detected state change. The action may be determined based on a particular state change (configured by the user when defining a custom event). For example, camera device 150 may send alert data indicating that the state change may be output by user device 130 associated with camera device 150.

[0104] Fig.18A 1 is a block diagram of an exemplary architecture 1800 of a computing device that can be used to instantiate various components of system 100 (such as camera device 150) according to various aspects of the present disclosure. It should be understood that not all devices include all components of architecture 1800, and some user devices may include additional components not shown in architecture 1800. Architecture 1800 may include one or more processing elements 1804 for executing instructions and retrieving data stored in storage element 1802. Processing element 1804 may include at least one processor. Any suitable one or more processors may be used. For example, processing element 1804 may include one or more digital signal processors. Storage element 1802 may include one or more different types of memory, data storage, or computer-readable storage media dedicated to different uses within architecture 1800. For example, storage element 1802 may include flash memory, random access memory, disk-based storage, and the like. For example, different portions of storage element 1802 may be used for program instructions executed by processing element 1804, storage of images or other digital works, and / or removable storage for transferring data to other devices, and the like. In addition, the storage element 1802 can store parameters and / or machine learning models generated using the various techniques described herein. For example, the storage element 1802 can store a custom classifier model 108 (and / or encoder model 104) so ​​that event classification can be performed locally on a device including the image sensor 1832.

[0105] The storage element 1802 may also store software for execution by the processing element 1804. The operating system 1822 may provide a user with an interface for operating the computing device and may facilitate communication and commands between applications executing on the architecture 1800 and its various hardware. The transmission application 1824 may be configured to receive images, audio, and / or video from another device (e.g., a mobile device, an image capture device, and / or a display device) or from an image sensor 1832 and / or a microphone 1870 included in the architecture 1800.

[0106] When implemented in some user devices, architecture 1800 may also include display component 1806. Display component 1806 may include one or more light emitting diodes (LEDs) or other suitable display lights. In addition, in some examples, display component 1806 may include, for example, one or more devices such as a cathode ray tube (CRT), a liquid crystal display (LCD) screen, a gas plasma-based flat panel display, an LCD projector, a raster projector, an infrared projector, or other types of display devices, etc. As described herein, display component 1806 can effectively display image data and / or video data representing custom user-defined events (when detected), as described herein.

[0107] The architecture 1800 may also include one or more input devices 1808 operable to receive input from a user. The input devices 1808 may include, for example, buttons, touch pads, touch screens, scroll wheels, joysticks, keyboards, mice, trackballs, keypads, light guns, game controllers, or any other such devices or elements by which a user may provide input to the architecture 1800. These input devices 1808 may be incorporated into the architecture 1800 or operably coupled to the architecture 1800 via a wired or wireless interface. In some examples, the architecture 1800 may include a microphone 1870 or microphone array for capturing sounds, such as voice requests. In various examples, the audio captured by the microphone 1870 may be streamed to an external computing device via the communication interface 1812.

[0108] When display component 1806 includes a touch-sensitive display, input device 1808 may include a touch sensor that operates with display component 1806 to allow a user to use touch input (e.g., with a finger or a stylus) to interact with images displayed by display component 1806. Architecture 1800 may also include a power source 1814, such as a wired alternating current (AC) converter, a rechargeable battery operable to be charged by conventional plug-in methods, or by other means such as by capacitive or inductive charging.

[0109] The communication interface 1812 may include one or more wired or wireless components operable to communicate with one or more other computing devices. For example, the communication interface 1812 may include a wireless communication module 1836 configured to communicate on a network (such as network 605) according to any suitable wireless protocol (such as IEEE 802.11 or another suitable wireless local area network (WLAN) protocol). The short-range interface 1834 may be configured to communicate using one or more short-range wireless protocols (e.g., near field communication (NFC), Bluetooth, Bluetooth LE, etc.). The mobile interface 1840 may be configured to communicate using a cellular or other mobile protocol. The global positioning system (GPS) interface 1838 may communicate with one or more earth-orbiting satellites or other suitable location determination systems to identify the location of the architecture 1800. The wired communication module 1842 may be configured to communicate according to a USB protocol or any other suitable protocol.

[0110] The architecture 1800 may also include one or more sensors 1830 , such as, for example, one or more position sensors, image sensors, and / or motion sensors. Fig.18A 1800. Some examples of architecture 1800 may include multiple image sensors 1832. For example, a panoramic camera system may include multiple image sensors 1832 to produce multiple images and / or video frames that can be stitched and blended to form a seamless panoramic output. An example of image sensor 1832 may be a camera configured to capture color information, image geometry information, and / or ambient light information. In some additional examples, image sensor 1832 may include a depth sensor and / or multiple depth sensors. For example, image sensor 1832 may include a TOF sensor, a stereo depth sensor, a lidar sensor, a radar, and the like.

[0111] As described above, multiple devices may be employed in a single system. In such a multi-device system, each of the devices may include different components to perform different aspects of system processing. Multiple devices may include overlapping components. As described herein, the components of a computing device are exemplary and may be located as independent devices or may be included in whole or in part as components of a larger device or system. Fig.18B Another exemplary architecture 1801 is depicted. The architecture 1801 can be an example of, for example, a camera device 150, where the encoder model 104 and / or the custom classifier model 108 are executed remotely relative to the architecture 1801 (eg, by a remote system 630).

[0112] An exemplary system for sending and providing data will now be described in detail. Specifically, Fig.19 An exemplary computing environment in which the embodiments described herein may be implemented is shown. For example, Fig.19 The computing environment can be used to provide the various machine learning models described herein as services over a network, where one or more of the techniques described herein can be requested by a first computing device and can be executed by a different computing device configured to communicate with the first computing device over the network. Fig.19 6 is a diagram schematically illustrating an example of a data center 65, which can provide computing resources to users 60a and 60b (which can be referred to as users 60 in the singular or users 60 in the plural) via user computers 62a and 62b (which can be referred to as user computers 62 in the singular or user computers 62 in the plural) via a network 605. The data center 65 can be configured to provide computing resources for executing applications permanently or as needed. The computing resources provided by the data center 65 can include various types of resources, such as gateway resources, load balancing resources, routing resources, networking resources, computing resources, volatile and non-volatile memory resources, content delivery resources, data processing resources, data storage resources, data communication resources, and the like. Each type of computing resource can be available in a variety of specific configurations. For example, data processing resources can be available as virtual machine instances that can be configured to provide various web services. In addition, a combination of resources can become available via a network and can be configured as one or more web services. The instances may be configured to execute applications, including web services, such as application services, media services, database services, processing services, gateway services, storage services, routing services, security services, encryption services, load balancing services, application services, etc. In various examples, the instances may be configured to execute one or more of the various machine learning techniques described herein.

[0113] These services can be configured by set or customized applications, and can be configured in size, execution, cost, latency, type, duration, accessibility, and any other dimension. These web services can be configured as infrastructure for one or more clients, and may include one or more applications configured as a system or software for one or more clients. These web services can become available via one or more communication protocols. These communication protocols may include, for example, hypertext transfer protocol (HTTP) or non-HTTP protocols. These communication protocols may also include, for example, more reliable transport layer protocols, such as transmission control protocol (TCP) and less reliable transport layer protocols, such as user datagram protocol (UDP). Data storage resources may include file storage devices, block storage devices, and the like.

[0114] Each type or configuration of computing resources may have different sizes, such as large resources (consisting of many processors, large amounts of memory, and / or large storage capacity) and small resources (consisting of fewer processors, smaller amounts of memory, and / or smaller storage capacity). For example, a customer may choose to allocate multiple small processing resources as web servers and / or allocate a large processing resource as a database server.

[0115] Data center 65 may include servers 66a and 66b (which may be referred to herein as server 66 in the singular or in the plural) that provide computing resources. These resources may be available as bare metal resources or as virtual machine instances 68a-d (which may be referred to herein as virtual machine instance 68 in the singular or in the plural). In at least some examples, server manager 67 may control the operation of server 66 and / or maintain the server. Virtual machine instances 68c and 68d are reproduction switching virtual machine ("RSVM") instances. RSVM virtual machine instances 68c and 68d may be configured to perform all or any portion of the techniques for improved reproduction switching according to the present disclosure and described in detail above and / or any other techniques in the disclosed techniques. It should be understood that although Fig.19 The specific example shown in includes one RSVM virtual machine in each server, but this is only an example. A server may include more than one RSVM virtual machine, or may not include any RSVM virtual machine.

[0116] The emergence of computing hardware virtualization technology is conducive to providing customers with large-scale computing resources and allowing multiple customers to share computing resources efficiently and securely. For example, virtualization technology can allow multiple users to share physical computing devices by providing each user with one or more virtual machine instances hosted by the physical computing device. A virtual machine instance can be a software simulation of a specific physical computing system that acts as a different logical computing system. Such a virtual machine instance provides isolation between multiple operating systems that share a given physical computing resource. In addition, some virtualization technologies can provide virtual resources that span one or more physical resources, such as a single virtual machine instance with multiple virtual processors that span multiple different physical computing systems.

[0117] refer to Fig.19 , network 605 may be, for example, a publicly accessible network of linked networks, and may be operated by different parties such as the Internet. In other embodiments, network 605 may be a private network, such as a company or university network that is completely or partially inaccessible to non-privileged users. In other embodiments, network 605 may include one or more private networks that can access and / or be accessed from the Internet.

[0118] The network 605 may provide access to a user computer 62. The user computer 62 may be a computer used by the user 60 or other client of the data center 65. For example, the user computer 62a or 62b may be a server, a desktop or laptop personal computer, a tablet computer, a wireless phone, a personal digital assistant (PDA), an e-book reader, a game console, a set-top box, or any other computing device capable of accessing the data center 65. The user computer 62a or 62b may be directly connected to the Internet (e.g., via a cable modem or a digital subscriber line (DSL)). Although only two user computers 62a and 62b are depicted, it should be understood that multiple user computers may exist.

[0119] The user computer 62 may also be used to configure various aspects of the computing resources provided by the data center 65. In this regard, the data center 65 may provide a gateway or web interface through which various aspects of its operation may be configured using a web browser application executing on the user computer 62. Alternatively, a standalone application executing on the user computer 62 may access an application programming interface (API) exposed by the data center 65 to perform configuration operations. Other mechanisms for configuring the operation of the various web services available at the data center 65 may also be utilized.

[0120] The server 66 shown in Figure 6 can be a server appropriately configured to provide the above-mentioned computing resources, and can provide computing resources for executing one or more web services and / or applications. In one embodiment, the computing resources can be virtual machine instances 68. In the example of virtual machine instances, each of the servers 66 can be configured to execute an instance manager 63a or 63b (which can be referred to as instance manager 63 in the singular or instance manager 63 in the plural) capable of executing virtual machine instances 68. For example, instance manager 63 can be a virtual machine monitor (VMM) or another type of program configured to enable execution of virtual machine instances 68 on server 66. As described above, each virtual machine instance in virtual machine instances 68 can be configured to execute all or part of an application.

[0121] It should be understood that although the embodiments disclosed above discuss the context of virtual machine instances, other types of implementations can be implemented using the concepts and technologies disclosed herein. For example, the embodiments disclosed herein can also be used with computing systems that do not use virtual machine instances.

[0122] exist Fig.19In the exemplary data center 65 shown, a router 61 can be used to interconnect a server 66a and a server 66b. The router 61 can also be connected to a gateway 64, which is connected to a network 605. The router 61 can be connected to one or more load balancers, and can manage communications within the network in the data center 65 individually or in combination, such as by appropriately forwarding packets or other data communications based on the characteristics of such communications (e.g., header information including source and / or destination addresses, protocol identifiers, sizes, processing requirements, etc.) and / or the characteristics of the dedicated network (e.g., routing based on network topology, etc.). It should be understood that for simplicity, various aspects of the computing systems and other devices of this example are shown without showing certain conventional details. In other embodiments, additional computing systems and other devices can be interconnected, and can be interconnected in different ways.

[0123] exist Fig.19 In the exemplary data center 65 shown in FIG. 1 , the data center 65 is also used to at least partially direct various communications to and from the server 66a and the server 66b and / or between the two. Fig.19 Router 61 is depicted as being located between gateway 64 and data center 65, but this is merely an exemplary configuration. In some cases, for example, data center 65 may be located between gateway 64 and router 61. In some cases, data center 65 may examine portions of incoming communications from user computer 62 to determine one or more appropriate servers 66 to receive and / or process the incoming communications. Data center 65 may determine appropriate servers to receive and / or process incoming communications based on factors such as the identity, location, or other attributes associated with user computer 62, the nature of the task associated with the communication, the priority of the task associated with the communication, the duration of the task associated with the communication, the size of the task associated with the communication, and / or estimated resource usage, as well as many other factors. Data center 65 may, for example, collect or otherwise access state information and other information associated with various tasks to, for example, help manage communications and other operations associated with such tasks.

[0124] It should be understood that Fig.19 The network topology shown in has been greatly simplified, and more networks and networking devices can be used to interconnect the various computing systems disclosed herein. These network topologies and devices should be obvious to those skilled in the art.

[0125] It should also be understood that Fig.19The data center 65 described in the description is merely illustrative, and other implementations may be used. It should also be understood that the server, gateway, or other computing device may include any combination of hardware or software that can interact and perform the types of functions described, including but not limited to: desktop or other computers, database servers, network storage devices and other network devices, PDAs, tablet computers, cellular phones, wireless phones, pagers, electronic organizers, Internet appliances, television-based systems (e.g., using set-top boxes and / or personal / digital video recorders), and various other consumer products with appropriate communication capabilities.

[0126] A network established by an entity (such as a company or public sector organization) for providing one or more web services (such as various types of cloud-based computing or storage) accessible via the Internet and / or other networks to a set of distributed clients may be referred to as a provider network. Such a provider network may include numerous data centers hosting various resource pools, such as a collection of physical and / or virtualized computer servers, storage devices, networking equipment, etc., for implementing and distributing the infrastructure and web services provided by the provider network. In some embodiments, resources may be provided to clients in various units related to web services, such as an amount of storage capacity for storage, processing power for processing, instances, as a collection of related services, and the like. For example, a virtual computing instance may include one or more servers with a specified computing capacity (which may be specified by indicating the type and number of CPUs, the size of main memory, etc.) and a specified software stack (e.g., a specific version of an operating system, which in turn may run on top of a hypervisor).

[0127] In different embodiments, the resources of the provider network may be implemented using a plurality of different types of computing devices, such as computer servers, storage devices, network devices, etc., either alone or in combination. In some embodiments, a client or user may be provided with direct access to resource instances, such as by providing the user with an administrator login name and password. In other embodiments, the provider network operator may allow the client to specify execution requirements for a given client application and schedule the application on behalf of the client to be executed on an execution system suitable for the application (such as an application server instance, a Java JVM, etc.). TM The execution of a resource instance on a virtual machine (JVM), a general-purpose or special-purpose operating system that supports various interpreted or compiled programming languages ​​(such as Ruby, Perl, Python, C, C++, etc.), or a high-performance computing system) without, for example, requiring the client to directly access the instance or execution system. In some embodiments, a given execution system can utilize one or more resource instances; in other embodiments, multiple execution systems can be mapped to a single resource instance.

[0128] In many environments, operators of provider networks that implement different types of virtualized computing, storage, and / or other network-accessible functions may allow customers to reserve or purchase access to resources in various resource acquisition modes. Computing resource providers may provide customers with facilities for selecting and launching required computing resources, deploying application components to computing resources, and maintaining applications executed in the environment. In addition, as the demand or capacity requirements of applications change, computing resource providers may provide customers with additional facilities to quickly and easily increase or decrease the number and type of resources allocated to applications manually or through automatic scaling. The computing resources provided by computing resource providers may be available in discrete units, which may be referred to as instances. An instance may represent a physical server hardware system, a virtual machine instance executed on a server, or some combination of the two. Instances of various types and configurations may be provided, including resources of different sizes executing different operating systems (OS) and / or virtual machine hypervisors, and installing various software applications, runtimes, and the like. Instances may also be available in specific availability zones representing, for example, logical regions, fault-tolerant regions, data centers, or other geographic locations of underlying computing hardware. Instances may be replicated within or across availability zones to improve the redundancy of instances, and instances may also be migrated within or across availability zones. As an example, the latency for client communications with a particular server in an availability zone can be less than the latency for client communications with a different server. Therefore, instances can be migrated from servers with higher latency to servers with lower latency to improve the overall client experience.

[0129] In some embodiments, the provider network can be organized into multiple geographic regions, and each region may include one or more availability zones. Availability zones (also referred to as availability containers) may then include one or more different locations or data centers, which are configured so that the resources in a given availability zone can be isolated or insulated from faults in other availability zones. That is, a fault in one availability zone may not cause a fault in any other availability zone. Therefore, the availability profile of a resource instance is intended to be independent of the availability profile of the resource instance in different availability zones. The client may be able to protect its application from failures in a single location by starting multiple application instances in the corresponding availability zone. At the same time, in some embodiments, a cheap and low-latency network connection can be provided between resource instances residing in the same geographic region (and network transmission between resources in the same availability zone may be even faster).

[0130] Terms

[0131] In the first clause of the present disclosure: a computer-implemented method, the method comprising: receiving a first frame of image data from a first camera device; determining first account data associated with the first camera device; generating a first embedding for the first frame of image data by a first transformer model; determining a first logistic regression model associated with the first account data; generating a first score by the first logistic regression model using the first embedding; determining a first state of a region of interest represented by the first frame of image data by comparing the first score with a first threshold score; determining previous state data stored in a computer-readable memory, the previous state data indicating that a second frame of image data captured before the first frame of image data is associated with a second state of the region of interest; and sending an alert to a first user device associated with the first account data in response to the state of the region of interest changing from the second state to the first state.

[0132] In a second clause of the present disclosure, the computer-implemented method as described in the first clause also includes: determining the region of interest associated with the first account data; and generating a cropped frame of image data by cropping the region of interest from the first frame of image data, wherein the first embedding represents the cropped frame of image data.

[0133] In a third clause of the present disclosure, the computer-implemented method as described in the first clause also includes: receiving a second frame of image data from a second camera device; determining second account data associated with the second camera device; generating a second embedding of the second frame of image data by the first transformer model; determining a second logistic regression model associated with the second account data; generating a second score by the second logistic regression model using the second embedding; determining a second state of a second region of interest represented by the second frame of image data by comparing the second score to a second threshold score; and determining that the second state of the second region of interest has not changed since the state of the second region of interest was previously determined.

[0134] In a fourth clause, a method includes: receiving first image data representing a first frame from a camera device; generating first embedded data for the first frame based at least in part on the first image data and using a first machine learning model; determining a first score based on the first embedded data and using a second machine learning model trained based on the image data generated by the camera device; determining a first state depicted in the first frame based at least in part on the first score; determining a second state depicted in a second frame based on data associated with the camera device and stored in a memory; determining that the first state is different from the second state; and sending a message to a device associated with the camera device based on determining that the first state and the second state are different.

[0135] The fifth clause may include a method as described in the fourth clause, wherein the second machine learning model includes one or more parameters determined based on training the second machine learning model using image data generated by the camera device.

[0136] Clause 6 may include the method of clause 4, wherein determining the first state comprises comparing the first score to a first threshold.

[0137] Clause 7 may include the method as recited in clause 6, wherein the method further comprises receiving first data, and based on receiving the first data, changing the first threshold.

[0138] Clause 8 may include the method of clause 4, wherein determining the first state comprises determining whether the first score is above a first threshold or below a second threshold.

[0139] Clause 9 may include a method as described in clause 4, wherein the method comprises: receiving second image data representing a third frame from a camera device; generating second embedded data for the third frame based on the second image data using the first machine learning model; determining a second score based on the second embedded data using the second machine learning model; determining a third score based at least in part on the first score and the second score; and wherein the first state is determined based on the third score value.

[0140] Clause ten may include the method of clause nine, wherein determining the third score comprises determining an average of a plurality of scores, the plurality of scores comprising the first score and the second score.

[0141] The eleventh clause may include a method as described in the fourth clause, further comprising generating second image data representing a cropped version of the first frame based on the first image data and boundary data of a region of interest; wherein the generation of the first embedded data of the first frame is based on the second image data.

[0142] Clause 12 may include the method of clause 4, further comprising wherein determining, based at least in part on the first score, that the first state is depicted in the first frame comprises determining that the first state is depicted in a region of interest in the first frame.

[0143] Clause 13 may include a method as described in clause 4, further comprising: determining identifier data associated with the camera device; and selecting the second machine learning model from a plurality of machine learning models based at least in part on the identifier data associated with the second machine learning model.

[0144] Clause fourteen may include a method as described in clause four, wherein the method includes, before receiving the first image data: receiving second image data representing the second frame; determining the second state based at least in part on the second image data and using the first machine learning model and the second machine learning model; and storing the second state.

[0145] Clause 15 may include the method of clause 14, wherein the first frame represents a frame of a first video stream and the second frame represents a previous frame of the first video stream.

[0146] Clause 16 may include the method of clause 14, wherein the first frame represents a frame of a second video stream and the second frame represents a frame of a first video stream.

[0147] Clause seventeen may include the method as recited in clause four, wherein the message represents an alert including an indication of a change of state.

[0148] Clause 18 may include the method as recited in clause 4, wherein the message includes second image data representative of the first frame.

[0149] Clause 19 may include the method as recited in clause 4, wherein the message includes an indication of the first state.

[0150] Clause 20 may include the method as described in clause 4, wherein the first state and the second state are user-defined states.

[0151] Clause 21 may include a method as described in clause 4, wherein the method comprises, before receiving the first image data, receiving: first tag data, the first tag data indicating a first group of frames associated with the first state, each frame in the first group of frames having been captured by a camera device; and second tag data, the second tag data indicating a second group of frames associated with the second state, each image in the second group of one or more images having been captured by the camera device.

[0152] Clause 22 may include the method as described in clause 4, wherein the message includes second image data representing the most recent frame for which image data is available.

[0153] In clause 23, a device may include: one or more processors; and at least one non-transitory computer-readable memory storing instructions, which, when executed by the one or more processors, are effective for: receiving first image data representing a first frame from a camera device; generating first embedded data for the first frame based at least in part on the first image data and using a first machine learning model; determining a first score based on the first embedded data and using a second machine learning model trained based on the image data generated by the camera device; determining a first state depicted in the first frame based at least in part on the first score; determining a second state depicted in the second frame based on data associated with the camera device and stored in a memory; determining that the first state is different from the second state; and sending a message to a device associated with the camera device based on the determination that the first state is different from the second state.

[0154] Clause 24 may include a device as described in clause 23, wherein the device further comprises the camera device.

[0155] Clause 25 may include an apparatus as described in clause 24, wherein the second machine learning model comprises one or more parameters determined based on training the second machine learning model using image data generated by the camera device.

[0156] Clause twenty-six may include an apparatus as described in clause twenty-three, wherein the at least one non-transitory computer-readable memory stores additional instructions, which, when executed by the one or more processors, are further effectively used to: receive second image data representing a third frame from a camera device; generate second embedded data for the third frame using the first machine learning model based on the second image data; determine a second score using the second machine learning model based on the second embedded data; and determine a third score based at least in part on the first score and the second score; and wherein the first state is determined based on the third score value.

[0157] Clause twenty-seven may include a computer-implemented method comprising: receiving first label data indicating a first image and a first user-defined state from a user device associated with a user, the first image having been captured by a first camera device associated with the user; receiving second label data indicating a second image associated with a second user-defined state from the user device, the second image having been captured by the first camera device associated with the user; generating corresponding embedding data using a first machine learning model for each corresponding image in a first group of images including the first image and the second image; and training a second machine learning model to classify an image as corresponding to the first user-defined state or the second user-defined state based at least in part on the first label data, the second label data, and the embedding data of the image of the first group of images.

[0158] Clause 28 may include clause 27, wherein the method further comprises receiving boundary data indicating boundaries of an area of ​​interest from the user device associated with the user; and wherein generating embedded data of the first group of one or more images and the second group of one or more images comprises, for each corresponding image, generating cropped image data representing a cropped version of the corresponding image based on the corresponding image and the boundary data.

[0159] Clause 29 may include clause 27, wherein the first machine learning model is a pre-trained visual transformer model, a pre-trained convolutional neural network model, a pre-trained neural network model, a pre-trained graph neural network model, a pre-trained graph convolutional neural network model, a pre-trained capsule neural network model, a pre-trained support vector machine model, or a pre-trained model utilizing visual transformation and convolution; and the second machine learning model is a logistic regression model, a convolutional neural network model, a neural network model, a graph neural network model, a graph convolutional neural network model, a support vector machine model, or a visual transformer model trained using data associated with the camera device.

[0160] Clause 30 may include clause 27, wherein the second machine learning model comprises a logistic regression model.

[0161] Clause thirty-one may include clause twenty-seven, wherein the method comprises determining a first threshold associated with one or more of the user-defined states based on a plurality of images in the first set of images.

[0162] Clause thirty-two may include clause twenty-seven, wherein the method comprises determining a first threshold associated with the first user-defined state and a second threshold associated with the second user-defined state based on a plurality of images in the first set of images.

[0163] Clause 33 may include Clause 31, wherein the method comprises: for each value of a plurality of possible threshold values: determining a corresponding precision based on the corresponding number of true positives determined and the corresponding number of false positives determined, determining a corresponding recall based on the corresponding number of true positives determined and the corresponding number of false negatives determined, and determining a corresponding f-score value based on the corresponding precision determined and the corresponding recall determined; and determining the first threshold value based on the determined f-score value.

[0164] Clause thirty-four may include clause twenty-seven, wherein the first tag data indicates a second group of images including the first image, and wherein the first group of images includes the second group of images.

[0165] Clause thirty-five may include clause twenty-seven, wherein the first set of images includes one or more images generated based on the first image or the second image.

[0166] Clause thirty-six may include clause twenty-seven, wherein the first set of images includes one or more images generated based on modifying the first image or the second image using a data augmentation technique.

[0167] Clause thirty-seven may include clause twenty-seven, wherein the first label data indicates a second group of images including the first image; wherein the first group of images includes the second group of images; wherein the second label data indicates a third group of images including the second image; wherein the first group of images includes the third group of images; and wherein the method includes, before using the first machine learning model to generate the embedded data: selecting a first plurality of images from the second group of images and the third group of images; training a third machine learning model to classify images as corresponding to the first user-defined state or the second user-defined state based at least in part on the first label data, the second label data and the first plurality of images; using the third machine learning model to determine a first state of a first image in the first group of one or more images that is not part of the first plurality of images; and comparing the first state with a state associated with the first image based on the first label data.

[0168] In clause 38, a device may include: one or more processors; and one or more computer-readable storage memories storing instructions, which, when executed by the one or more processors, are effective for: generating display data for a user interface that displays one or more images generated by a camera device; receiving first input data, the first input data defining a user-defined area of ​​interest regarding the one or more images generated by the camera device; receiving second input data associating a first image generated by the camera device with a first user-defined state; receiving third input data associating a second image generated by the camera device with a second user-defined state different from the first user-defined state; and sending first configuration data indicating the user-defined area of ​​interest, second configuration data indicating the first image is associated with the first user-defined state, and third configuration data indicating the second image is associated with the second user-defined state to a remote computing system.

[0169] In clause thirty-nine, one or more computer-readable storage memories as described in clause thirty-eight store additional instructions, which, when executed by the one or more processors, are further effectively used to: receive alarm data from the remote computing system indicating that a user-defined event including a change from the first user-defined state to the second user-defined state has occurred.

[0170] In clause 40, one or more computer-readable storage memories as described in clause 38 store additional instructions, which, when executed by the one or more processors, are further effectively used to: receive first data from the remote computing system, the first data indicating that the first image and the second image are too similar to each other so that the remote computing system cannot detect two different states.

[0171] In clause 41, one or more computer-readable storage memories as described in clause 38 store additional instructions, which, when executed by the one or more processors, are further effectively used to: receive first data from the remote computing system, the first data indicating that the second image and a fourth image marked as belonging to the first user-defined state are too different from each other to represent the first user-defined state.

[0172] In clause 42, one or more computer-readable storage memories as described in clause 38 store additional instructions, which, when executed by the one or more processors, are further effectively used to: receive first alarm data from the remote computing system at a first time indicating that a first user-defined event including a change from the first user-defined state to the second user-defined state has occurred; and receive second alarm data from the remote computing system at a second time after the first time indicating that a second user-defined event including a change from the second user-defined state to the first user-defined state has occurred.

[0173] In clause 43, a device includes: one or more processors; one or more computer-readable storage memories storing instructions, which, when executed by the one or more processors, are effectively used to: generate display data for a first user interface, the first user interface including a first group of images and a first prompt, the first prompt being used to enable the user to select at least a first image in the first group of images as a representation of a first user-defined state; generate display data for a second user interface, the second user interface including a second group of images and a second prompt, the second prompt enabling the user to select at least a second image in the second group of images as a representation of a second user-defined state different from the first user-defined state; and generate a first message indicating that event detection between the first user-defined state and the second user-defined state of an area of ​​interest has been enabled; and send first configuration data indicating that the first image is associated with the first user-defined state, and second configuration data indicating that the second image is associated with the second user-defined state to a remote computing system.

[0174] In clause 44, one or more computer-readable storage memories as described in clause 43 store additional instructions, which, when executed by the one or more processors, are further effectively used to: generate a second message indicating that a user-defined event has been detected, and the user-defined event represents a state change from the first user-defined state to the second user-defined state.

[0175] In clause 45, one or more computer-readable storage memories as described in clause 43 store additional instructions, which, when executed by the one or more processors, are further effectively used to: receive a selection of a first user-selected frame in the first set of frames; and generate a second message requesting selection of a different frame as a representation of the first user-defined state of the region of interest.

[0176] In clause 46, one or more computer-readable storage memories as described in clause 43 store additional instructions, which, when executed by the one or more processors, are further effectively used to: generate a third prompt associated with the second user interface, the third prompt requesting the name of the first user-defined state; and generate a fourth prompt associated with the third user interface, the third prompt requesting the name of the second user-defined state.

[0177] Although the various systems described herein may be embodied in software or code executed by general purpose hardware as described above, as an alternative, they may also be embodied in dedicated hardware or a combination of software / general purpose hardware and dedicated hardware. If embodied in dedicated hardware, each may be implemented as a circuit or state machine using any one or combination of a variety of technologies. These technologies may include, but are not limited to, discrete logic circuits having logic gates for implementing various logic functions when one or more data signals are applied, application specific integrated circuits or other components having appropriate logic gates, and the like. These technologies are generally well known to those of ordinary skill in the art and are therefore not described in detail herein.

[0178] The flowcharts and methods described herein illustrate the functionality and operation of various embodiments. If embodied in software, each box or step may represent a module, segment or portion of a code including program instructions for implementing a specified logical function. The program instructions may be embodied in the form of source code, which includes human-readable statements written in a programming language, or in the form of machine code, which includes digital instructions that can be recognized by a suitable execution system (such as a processing unit in a computer system). If embodied in hardware, each box may represent a circuit or multiple interconnected circuits to implement a specified logical function.

[0179] Although the flow chart and method described herein can describe a specific execution order, it should be understood that the execution order may be different from the described order. For example, the execution order of two or more frames or steps may be out of order relative to the described order. In addition, two or more frames or steps may be performed simultaneously or partially simultaneously. In addition, in some embodiments, one or more frames or steps in a frame or step may be skipped or omitted. It should be understood that all these changes are within the scope of the present disclosure.

[0180] In addition, any logic or application including software or code described herein may be embodied in any non-transient computer-readable medium or memory for use by an instruction execution system (such as a processing unit in a computer system) or in combination with an instruction execution system. In this sense, logic may include, for example, statements including instructions and declarations, which may be obtained from a computer-readable medium and executed by an instruction execution system. In the context of the present disclosure, "computer-readable medium" may be any medium that may contain, store, or maintain logic or applications described herein for use by an instruction execution system or in combination with an instruction execution system. Computer-readable media may include any of many physical media, such as magnetic media, optical media, or semiconductor media. More specific examples of suitable computer-readable media include, but are not limited to, tapes, magnetic floppy disks, magnetic hard drives, memory cards, solid-state drives, USB flash drives, or optical disks. In addition, computer-readable media may be random access memory (RAM), including, for example, static random access memory (SRAM) and dynamic random access memory (DRAM), or magnetic random access memory (MRAM). Additionally, the computer readable medium may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or other types of memory devices.

[0181] It should be emphasized that the above-mentioned embodiments of the present disclosure are only possible examples of the embodiments proposed for a clear understanding of the principles of the present disclosure. Many changes and modifications may be made to the above-mentioned examples without substantially departing from the spirit and principles of the present disclosure. In this article, all such modifications and variations are intended to be included within the scope of the present disclosure and are protected by the following claims.

Claims

1. A computer-implemented method, the method comprising: receiving a first frame of image data from a first camera device; determining first account data associated with the first camera device; generating, by a first transformer model, a first embedding for the first frame of image data; determining a first logistic regression model associated with the first account data; generating, by the first logistic regression model, a first score using the first embedding; determining a first state of a first region of interest represented by the first frame of image data by comparing the first score to a first threshold score; determining previous state data stored in a computer readable memory, the previous state data indicating that a second frame of image data captured prior to the first frame of image data was associated with a second state of the first region of interest; as well as In response to the state of the first area of ​​interest changing from the second state to the first state, sending an alert to a first user device associated with the first account data.

2. The computer-implemented method of claim 1 , further comprising: determining the first area of ​​interest associated with the first account data; as well as A cropped frame of image data is generated by cropping the first region of interest from the first frame of image data, wherein the first embedding represents the cropped frame of image data.

3. The computer-implemented method of claim 1 , further comprising: receiving a second frame of image data from a second camera device; determining second account data associated with the second camera device; generating, by the first transformer model, a second embedding for the second frame of image data; determining a second logistic regression model associated with the second account data; generating, by the second logistic regression model, a second score using the second embedding; determining a first state of a second region of interest represented by the second frame of image data by comparing the second score to a second threshold score; as well as It is determined that the first state of the second region of interest has not changed since a previous determination of a state of the second region of interest.

4. A method comprising: receiving first image data representing a first frame from a camera device; generating first embedding data for the first frame based at least in part on the first image data and using a first machine learning model; determining a first score based on the first embedded data and using a second machine learning model trained based on image data generated by the camera device; determining a first state depicted in the first frame based at least in part on the first score; determining a second state depicted in a second frame based on data associated with the camera device and stored in a memory; determining that the first state is different from the second state; as well as Based on determining that the first state is different from the second state, a message is sent to a device associated with the camera device.

5. The method of claim 4, wherein the second machine learning model comprises one or more parameters determined based on training the second machine learning model using image data generated by the camera device. The method of claim 4 , wherein determining the first state comprises comparing the first score to a first threshold. 7 . The method of claim 6 , wherein the method further comprises receiving first data, and changing the first threshold based on receiving the first data. 8 . The method of claim 4 , wherein determining the first state comprises determining whether the first score is above a first threshold or below a second threshold.

9. The method of claim 4, wherein the method comprises receiving second image data representing a third frame from the camera device; generating second embedding data for the third frame using the first machine learning model based on the second image data; determining a second score using the second machine learning model based on the second embedded data; as well as determining a third score based at least in part on the first score and the second score; Wherein the first state is determined based on the third score.

10. The method of claim 9, wherein determining the third score comprises determining an average of a plurality of scores, the plurality of scores comprising the first score and the second score.

11. The method of claim 4, wherein the method comprises generating second image data representing a cropped version of the first frame based on the first image data and boundary data of the region of interest; Wherein said generating of said first embedded data of said first frame is based on said second image data.

12. The method of claim 4, wherein determining, based at least in part on the first score, that the first state is depicted in the first frame comprises determining that the first state is depicted in a region of interest in the first frame.

13. A device, comprising: one or more processors; as well as at least one non-transitory computer readable memory storing instructions that, when executed by the one or more processors, are effective to: receiving first image data representing a first frame from a camera device; generating first embedding data for the first frame based at least in part on the first image data and using a first machine learning model; determining a first score based on the first embedded data and using a second machine learning model trained based on image data generated by the camera device; determining a first state depicted in the first frame based at least in part on the first score; determining a second state depicted in a second frame based on data associated with the camera device and stored in a memory; determining that the first state is different from the second state; as well as Based on determining that the first state is different from the second state, a message is sent to a device associated with the camera device.

14. The apparatus of claim 13, further comprising the camera device.

15. The device of claim 14, wherein the second machine learning model comprises one or more parameters determined based on training the second machine learning model using image data generated by the camera device.

Images