Video monitoring using neural networks

Abstract

Example methods, apparatus, systems, and articles of manufacture (e.g., physical storage media) for implementing video surveillance with neural networks are disclosed. An example system disclosed herein includes a database for storing records of operator-labeled video segments (e.g., as records of operator-labeled video segments). The operator-labeled video segments include reference video segments and corresponding reference event labels that describe the video segments. The disclosed example system also includes a neural network that includes a first instance of an inference engine and a training engine for training the first instance of the inference engine based on a training set of operator-labeled video segments obtained from the database, the first instance of the inference engine for inferring events from the operator-labeled video segments included in the training set. The disclosed example system also includes a second instance of the inference engine for inferring events from a monitored video feed, the second instance of the inference engine based on the first instance of the inference engine.

Description

[0001] This application is a divisional application. The original application was filed in China on January 5, 2021 (international filing date was July 5, 2019), with application number 201980045392.4 and invention title "Video Surveillance Using Neural Networks". Technical Field

[0002] This disclosure relates generally to video surveillance, and more specifically to video surveillance utilizing neural networks. Background Technology

[0003] Traditionally, video surveillance systems have been used by large organizations (e.g., large commercial enterprises, government agencies, educational institutions, etc.) for security monitoring. Such systems typically use cameras to cover the area to be monitored and feed video feeds to a central monitoring facility, such as a security office. The central monitoring facility usually includes one or more monitoring stations operated by one or more human operators who review the monitored video feeds and flag events of interest. In some examples, the monitoring stations allow human operators to record events of interest and take appropriate remedial action, such as activating alarms or contacting emergency services.

[0004] Recently, with the decrease in the cost of video surveillance cameras, the use of video surveillance systems has increased in other environments. For example, homes, small businesses, parks, and public areas equipped with video surveillance systems for monitoring have become commonplace. Such video surveillance systems may rely on low-cost cameras and / or any other imaging sensors to monitor areas of interest. These cameras typically include network interfaces, enabling them to connect to a network, which allows them to transmit their respective video feeds to one or more remote monitoring facilities. These remote monitoring facilities again rely on monitoring stations operated by human operators to view the monitored video feeds, flag events of interest, and take appropriate action in response to those events. Attached Figure Description

[0005] Figure 1 This is a block diagram of an example video surveillance system.

[0006] Figure 2 This is a block diagram of a first example video surveillance system based on the teachings of this disclosure, the system including a trainable neural network configured to support video surveillance.

[0007] Figure 3 This is a block diagram of a second example video surveillance system based on the teachings of this disclosure, the system including a trainable neural network and a trained inference engine configured to support video surveillance.

[0008] Figure 4This is a block diagram of a third example video surveillance system based on the teachings of this disclosure, the third example video surveillance system including a trainable neural network and a trained inference engine configured to support video surveillance.

[0009] Figure 5-8 A flowchart representing an example machine-readable instruction is shown, which can be executed to achieve... Figure 2-4 Example video surveillance system.

[0010] Figure 9 This is a block diagram of an example processor platform configured to execute commands from... Figure 5-8 One or more example machine-readable instructions are included to implement the implementation of the instructions included in Figure 2-4 Example monitoring station in an example video surveillance system.

[0011] Figure 10 This is a block diagram of an example processor platform configured to execute commands from... Figure 5-8 One or more example machine-readable instructions are included to implement the implementation of the instructions included in Figure 2-4 Example database in an example video surveillance system.

[0012] Figure 11 This is a block diagram of an example processor platform configured to execute commands from... Figure 5-8 One or more example machine-readable instructions are included to implement the implementation of the instructions included in Figure 2-4 Example neural network in an example video surveillance system.

[0013] The figures are not drawn to scale. Where possible, the same reference numerals will be used throughout the figures and the accompanying written description to denote the same or similar parts, elements, etc. Detailed Implementation

[0014] This document discloses example methods, apparatuses, systems, and artifacts (e.g., physical storage media) for implementing video surveillance using neural networks. The example video surveillance system disclosed herein includes a database for storing operator-tagged video clips (e.g., records as operator-tagged video clips). The operator-tagged video clips include reference video clips and corresponding reference event tags describing the reference video clips. The disclosed example video surveillance system also includes a neural network comprising a first instance of an inference engine and a training engine for training the first instance of the inference engine based on a training set of operator-tagged video clips obtained from the database. In the disclosed example, the first instance of the inference engine is used to infer events from the operator-tagged video clips included in the training set. The disclosed example video surveillance system also includes a second instance of the inference engine for inferring events from monitored video feeds, wherein the second instance of the inference engine is based on the first instance of the inference engine (e.g., initially a copy thereof).

[0015] In some publicly available examples, the reference event label indicates whether the corresponding reference video clip depicts a defined event.

[0016] In some publicly available examples, the first reference event label in the reference event label corresponding to the first reference video segment in the reference video clip indicates (i) the event type, and (ii) whether the first reference video segment in the reference video clip depicts that event type.

[0017] Some publicly disclosed example video surveillance systems include a monitoring station. In some publicly disclosed examples, the monitoring station includes a display for presenting a first monitored video feed in the monitored video feed, and a monitoring interface for generating operator event tags based on operator decisions corresponding to the monitored video segments in the first monitored video feed.

[0018] In some of these publicly disclosed examples, the database communicates with the monitoring station to receive a first monitored video segment from the monitored video clips and a first operator event tag in the operator event tag corresponding to the first monitored video segment from the monitored video clips. In some of these examples, the database stores the first monitored video segment from the monitored video clips and the corresponding first operator event tag in the operator event tag as a first reference video segment from the reference video clips included in the first operator-tagged video segment from the operator-tagged video clips and a corresponding first reference event tag in the reference event tags.

[0019] Additionally or alternatively, in some of these disclosed examples, the monitoring station also implements a second instance of the inference engine. For example, the second instance of the inference engine can output inferred events for a second monitored video segment within a first monitored video feed presented by the monitoring station's display. In some such examples, the monitoring interface of the monitoring station generates a second operator event label in the operator event label based on an operator decision detected for the second monitored video segment within the monitored video feed. In some such examples, the monitoring station also includes a comparator for comparing the inferred events with the second operator event label in the operator event label to obtain updated training data. In some disclosed examples, the neural network communicates with the monitoring station to receive updated training data, and the neural network's training engine retrains the first instance of the inference engine based on the updated training data.

[0020] These and other example methods, apparatuses, systems, and artifacts (e.g., physical storage media) for utilizing neural networks to achieve video surveillance will be disclosed in more detail below.

[0021] Video surveillance systems are typically used for security monitoring by large organizations such as large commercial enterprises, government agencies, and educational institutions. Recently, with the decrease in the cost of video surveillance cameras, the use of video surveillance systems has increased in other environments. For example, homes, smaller businesses, parks, and public areas equipped with video surveillance systems for monitoring have become commonplace. Such newer video surveillance systems may rely on low-cost cameras and / or any other imaging sensors capable of connecting to networks such as the Internet, or one or more cloud services accessible via the Internet and / or other networks. Such network access enables the cameras and / or other imaging sensors to transmit their respective video feeds to one or more remote monitoring facilities. These remote monitoring facilities typically rely on monitoring stations operated by human operators to view the monitored video feeds, flag events of interest, and take appropriate action in response to those events. As the cost of video surveillance cameras / sensors and associated network technologies continues to decrease, the use of video surveillance is expected to continue to grow, potentially exponentially. However, relying on monitoring stations operated by human operators to monitor the video feeds generated by video surveillance systems limits the scalability of video surveillance and can affect the accuracy of event detection, especially when human operators are overloaded by the sheer number of video feeds to monitor.

[0022] The video surveillance system implemented using neural networks disclosed herein provides a technical solution to the scalability and accuracy problems associated with existing video surveillance systems that rely on human operators for monitoring video feeds. The example video surveillance system disclosed herein includes a neural network with an inference engine trained to detect or infer events in the monitored video feed. As disclosed in further detail below, the neural network is trained using a training set of reference video clips, and reference event labels are determined based on decisions made by a human operator. For example, a reference event label may describe whether the corresponding reference video clip depicts a defined event (e.g., security breach, presence of an individual, arrival of a package, any designated / pre-planned event, etc.). In some examples, the reference event label may also include a description of the type of event of interest (e.g., security breach, presence of an individual, arrival of a package, etc.) and an indication of whether the corresponding reference video clip depicts the described event.

[0023] As described further in detail below, some publicly available example video surveillance systems deploy instances of trained inference engines to one or more monitoring locations to infer events from the corresponding monitored video feeds. For example, the trained inference engine can operate (e.g., in parallel, asynchronously, collaboratively, etc.) to infer whether one or more trained events (e.g., security breaches, individual presence, package arrival, etc.) are represented in the corresponding video feeds they monitor. In some such examples, the trained inference engine can be deployed to perform or operate in conjunction with a monitoring station operated by a human operator to enhance human-operated surveillance. In some such examples, the inferred events from the output of the trained inference engine combined with the operation of a human-operated monitoring station can be compared with corresponding decisions made by a human operator to determine updated training data that can be used to refine the operation of the inference engine. For example, a neural network in a video surveillance system can receive updated training data, retrain its inference engine, and then redeploy instances of the retrained inference engine to one or more monitoring locations. In some examples, instances of trained / retrained inference engines can be deployed to unmanned monitoring locations or to provide additional capacity at manned locations, allowing the capacity of the video surveillance system to easily scale as demand increases. These and other aspects of video surveillance utilizing neural networks, as disclosed herein, are described in further detail below.

[0024] Turn to the attached diagram. Figure 1 A block diagram of an example video surveillance system 100 is shown. Figure 1 The example video surveillance system 100 includes example imaging sensors 105A-D, example network 110, and example monitoring stations 115A-B. Figure 1In the illustrated example, the imaging sensor 105A-D is configured to monitor areas of interest, such as, but not limited to, areas in one or more commercial establishments, small businesses, government offices, educational institutions, homes, parks, public areas, etc., and / or any combination thereof. The example imaging sensor 105A-D can include any number, type, and / or combination of imaging sensors. For example, the imaging sensor 105A-D can be implemented by one or more cameras, smartphones, photodiodes, photodetectors, etc.

[0025] exist Figure 1 In the example video surveillance system 100, the imaging sensor 105A-D includes a network interface capable of communicating with the example network 110. The example network 110 can be implemented by any number, type, and / or combination of networks. For example, network 110 can be implemented by the Internet and / or one or more cloud services accessible via the Internet. In some examples, the imaging sensor 105A-D includes a network interface capable of accessing network 110 via one or more wireless access points (e.g., cellular access points / base stations, wireless LAN access points, Bluetooth access points, etc.), wired access points (e.g., Ethernet access points, wired communication links, etc.), or any combination thereof.

[0026] exist Figure 1 In the example video surveillance system 100, imaging sensors 105A-D transmit their respective video feeds to monitoring stations 115A-B, which include corresponding network interfaces capable of communicating with network 110. Thus, monitoring stations 115A-B communicate with imaging sensors 105A-D via network 110. As used herein, the phrase “communicate with” includes variations covering direct communication and / or indirect communication via one or more intermediate components, and does not require direct physical (e.g., wired) communication and / or continuous communication, but also includes selective communication at regular or irregular intervals and one-off events.

[0027] The monitoring station 115A-B shown in the example can be implemented by any system / device capable of presenting the monitored video feed and accepting user input related to the monitored video feed. For example, monitoring station 115A-B can be implemented by any number, type, and / or combination of computing systems / devices (e.g., one or more computers, workstations, smartphones, tablets, personal digital assistants (PDAs), etc.). In some examples, monitoring station 115A-B is implemented by a processor platform, such as... Figure 9 The example processor platform 900 shown below will be described in further detail.

[0028] exist Figure 1In the illustrated example, monitoring station 115A-B includes corresponding example displays 120A-B and corresponding example monitoring interfaces 125A-B, enabling video feeds reported by imaging sensors 105A-D to be monitored by operator 130A-B. In the illustrated example, the displays 120A-B of the corresponding monitoring station 115A-B are configured to display one or more video feeds reported by imaging sensors 105A-D. For example, displays 120A-B may display one video feed, cycle through multiple video feeds one at a time, display multiple video feeds simultaneously in a tiled manner, and so on.

[0029] The monitoring interface 125A-B of the corresponding monitoring station 115A-B is configured to accept input from human operator 130A-B, reflecting the human operator 130A-B's decision regarding whether to depict an event in the video feed presented by display 120A-B. For example, monitoring interface 125A-B may include input buttons or keys (in... Figure 1 The monitoring interface 125A-B can be labeled "Y" and "N" to allow human operators 130A-B to indicate whether a video segment fed by the monitored video depicts or otherwise represents an event of interest (e.g., where "Y" indicates that the event of interest is depicted / represented, and "N" indicates that the event of interest is not depicted / represented). For example, human operators 130A-B can be trained to detect events of interest, such as, but not limited to, security breaches, the presence of an individual, the arrival of a package, etc. In some such examples, human operators 130A-B can use the "Y" and "N" inputs of the monitoring interface 125A-B to indicate whether an event of interest is depicted or otherwise represented in a video segment fed by the monitored video. In some examples, the monitoring interface 125A-B may include other input / output functions (in... Figure 1 The interface is labeled "I / O" to provide greater flexibility by allowing human operators 130A-B to input descriptions of event types. In such examples, human operators 130A-B are not limited to monitoring only one or more events of interest, but can instead input descriptions of any type of event of interest. This additional capability allows events of interest to change over time. In some examples, human operators 130A-B can use the "I / O" interface to input descriptions of the type of monitored event in a specific video segment of the monitored video feed, and can use "Y" and "N" inputs to indicate whether the event is indeed depicted or otherwise represented in the monitored video segment.

[0030] although Figure 1The example video surveillance system 100 is illustrated as including four imaging sensors 105A-D, one network 110, and two monitoring stations 115A-B, but the video surveillance system 100 is not limited thereto. Rather, the example video surveillance system 100 may include any number of imaging sensors 105A-D, any number of networks 110, and any number of monitoring stations 115A-B.

[0031] exist Figure 2 The diagram shows a block diagram of a first example video surveillance system 200 that utilizes a neural network to implement video surveillance in accordance with the teachings of this disclosure. Figure 2 Example video surveillance system 200 includes Figure 1 The video surveillance system 100 includes example imaging sensors 105A-D, example network 110, and example monitoring stations 115A-B. Thus, the above is combined... Figure 1 The description describes various aspects of these elements of the video surveillance system 200.

[0032] Figure 2 The example video surveillance system 200 also includes an example neural network 205 and an example database 210 implemented according to the teachings of this disclosure. The example database 210 includes one or more network interfaces capable of communicating with the example network 110. Figure 2 In the illustrated example, database 210 communicates with monitoring stations 115A-B via network 110 and receives video segments from monitoring stations 115A-B with corresponding event tags describing the video segments. The video segments received from monitoring stations 115A-B are obtained from monitored video feeds. The event tags, based on decision information input by human operators 130A-B, indicate whether the corresponding video segment depicts or otherwise represents an event of interest. In some examples, the event tags for corresponding video segments indicate whether the video segment depicts or otherwise represents a defined event. In some examples, the event tags for corresponding video segments include a description of the event type or otherwise indicates the event type, and also indicate whether the video segment depicts or otherwise represents that event type.

[0033] In the example shown, database 210 treats video clips with corresponding event tags received from monitoring stations 115A-B as operator-tagged video clips for training neural network 205. For example, database 210 treats video clips received from monitoring stations 115A-B as example reference video clips 215 for operator-tagged video clips, and corresponding event tags received from monitoring stations 115A-B as reference event tags 218 corresponding to the operator-tagged video clips 215. Furthermore, database 210 creates example records 220 of operator-tagged video clips, which include the reference video clip 215 represented by record 220 and the corresponding reference event tag 218 of the operator-tagged video clip. Therefore, each record 220 of operator-tagged video clips includes the reference video clip 215 and the corresponding reference event tag 218 represented by that record 220. Thus, in some examples, database 210 implements a method for obtaining operator-tagged video clips, wherein the operator-tagged video clips include reference video clips and corresponding reference event tags describing the reference video clips. Other methods for obtaining operator-tagged video clips may include, but are not limited to, computing devices, servers, cloud-based services, websites, etc., which are configured to collect video clips and combine them with corresponding event tags describing the video clips to form operator-tagged video clips.

[0034] The illustrated database 210 also includes an example recording storage device 225, which stores records 220 of video clips marked by operators, created by the database 210 based on video clips received from monitoring stations 115A-B and corresponding event tags. The recording storage device 225 of the illustrated database 210 can be implemented using any number and / or type of storage technology, memory technology, etc. For example, the database 210 can be implemented using any computing system / device, such as... Figure 10 The example processor platform 1000 shown is described in further detail below. In such an example, the recording storage device 225 may be implemented by one or more of the example mass storage device 1028 and / or volatile memory 1014 of the example processor platform 1000.

[0035] The neural network 205 shown in the example includes an example inference engine 230 for inferring events from video clips, such as video clips fed from monitored video obtained from imaging sensors 105A-D. In the example shown, the inference engine 230 is implemented by a convolutional neural network (CNN) inference engine, which includes one or more layers of weights, also called neurons, trained to infer events from video clips. For example, the inference engine 230 may be configured to include: an input layer for accepting one or more input video clips 235 as input data; one or more hidden layers for processing the input data; and an output layer for providing one or more outputs indicating whether a given input video clip 235 depicts or otherwise represents one or more events that the inference engine 230 has been trained to detect. In some examples, the output of the inference engine 230 may additionally or alternatively provide a probability (e.g., a likelihood) that a given input video clip 235 depicts or otherwise represents one or more events that the inference engine 230 has been trained to detect. Although the inference engine 230 of the example neural network 205 shown is implemented by a CNN, other neural network solutions can also be used to implement the inference engine 230.

[0036] The neural network 205 illustrated also includes an example training engine 240, an example comparator 245, and an example training data retrieval unit 250 to train the inference engine 230 to infer events from the input video clip 235. In the illustrated example, the training data retrieval unit 250 retrieves a set 220 of operator-marked video clips from a database 210 to use as training data to train the inference engine 230. For example, the training data retrieval unit 250 may send a request to the database 210 for the set 220 of operator-marked video clips. In some examples, the request includes the number of records 220 requested. In some examples, the request additionally or alternatively includes the type of event to be depicted or otherwise represented by the records 220 of operator-marked video clips included in the training set. In such an example, the database 210 may respond to the request by retrieving the requested set of records 220 of operator-marked video clips from the recording storage device 225 and sending the retrieved set of records 220 to the training data retrieval unit 250.

[0037] In the example shown, after obtaining the requested set of training records 220 from database 210, training data retrieval unit 250 applies the training records 220 to inference engine 230 to train inference engine 230 to infer events from operator-tagged video clips included in the set of training records 220. For example, for a given training record 220, training data retrieval unit 250 applies a reference video clip of that training record 220 as input video clip 235 to inference engine 230. Training data retrieval unit 250 also applies the corresponding reference event label of that training record as example training event label 255 for comparator 245 to compare with the inferred event decision output from inference engine 230. Ideally, when trained, the inferred event decision output from inference engine 230 will match the training event label 255 corresponding to the input video clip 235 (e.g., producing zero error). However, when training inference engine 230, comparator 245 may detect an error between the training event label 255 and the inferred event decision output from inference engine 230. In the example shown, the output of comparator 245 is provided to training engine 240, which feeds back errors in any suitable manner to update the layers of weights in inference engine 230 to reflect the accuracy with which inference engine 230 infers events from input segment 235. For example, training engine 240 is shown as a backpropagator that performs backpropagation to train inference engine 230. However, training engine 240 can implement any other suitable training mechanism. Thus, in some examples, training engine 240 implements a unit for training a first instance of the inference engine based on a training set of operator-tagged video segments, wherein the first instance of the inference engine is used to infer events from operator-tagged video segments included in the training set. Other units for training the first instance of the inference engine based on a training set of operator-tagged video segments can include, but are not limited to, computing devices, servers, cloud-based services, websites, etc., configured to obtain a training set of operator-tagged video segments and apply the training set to any type of machine learning inference engine to train the inference engine.

[0038] In some examples, the training data retrieval unit 250 continues to apply different training records from the training record 220 to the inference engine 230 until the comparator 245 indicates that the desired inference accuracy has been reached. For example, the inference accuracy can be specified as a threshold ratio for correct event detection (e.g., the percentage of input reference video clips for which the inference engine 230 correctly infers the existence of the corresponding event (as indicated by the reference event label), a threshold ratio for incorrect event detection (e.g., the percentage of input reference video clips for which the inference engine 230 incorrectly infers the existence of the corresponding event when the event does not actually exist (as indicated by the reference event label corresponding to the reference video clip), a threshold for missed event detection (e.g., the percentage of input reference video clips for which the inference engine 230 incorrectly infers the non-existence of the corresponding event when the event actually exists (as indicated by the reference event label corresponding to the reference video clip), and so on.

[0039] although Figure 2 The example video surveillance system 200 is shown to include four imaging sensors 105A-D, a network 110, two monitoring stations 115A-B, a neural network 205, and a database 210, but the video surveillance system 200 is not limited thereto. Instead, the example video surveillance system 200 may include any number of imaging sensors 105A-D, any number of networks 110, any number of monitoring stations 115A-B, any number of neural networks 205, and any number of databases 210.

[0040] exist Figure 3 A block diagram of a second example video surveillance system 300 that utilizes a neural network to implement video surveillance in accordance with the teachings of this disclosure is shown. Figure 3 Example video surveillance system 300 includes Figure 1 and Figure 2 The video surveillance systems 100 and 200 include example imaging sensors 105A-D, example network 110, and example monitoring stations 115A-B. Thus, the above is combined... Figure 1 and Figure 2 The description describes various aspects of these elements of the video surveillance system 300. Figure 3 The example video surveillance system 300 also includes Figure 2 The video surveillance system 200 includes an example neural network 205 and an example database 210. Thus, the above is combined... Figure 2 The description describes various aspects of these elements of the video surveillance system 300.

[0041] exist Figure 3In the illustrated example video surveillance system 300, the neural network 205 also includes an example deployer 305 for deploying instances of the trained inference engine 230 to infer events from monitored video feeds. Thus, in the illustrated example, the inference engine 230 corresponds to a first instance or reference instance of the inference engine 230, and the deployer deploys other instances of the inference engine 230 based on the first instance or reference instance (e.g., an initial copy). For example, the deployer 305 deploys a second example instance 310A of the inference engine 230 to be executed by or in conjunction with example monitoring station 115A, and deploys a third example instance 310B of the inference engine 230 to be executed by or in conjunction with example monitoring station 115B.

[0042] In some examples, deployer 305 deploys instances of inference engine 230, such as second instance 310A and third instance 310B, as data representing trained weight layers obtained by training a first instance of inference engine 230 included in neural network 205. In such examples, deployer 305 (e.g., via network 110) downloads data representing trained weight layers to instances of inference engine 230 (e.g., second instance 310A and third instance 310B) already present at the target monitoring location. In some examples, deployer 305 deploys instances of inference engine 230 (e.g., second instance 310A and third instance 310B) as downloadable executables (e.g., downloaded via network 110) executed by computing devices (e.g., monitoring stations 115A-B). Thus, in some examples, deployer 305 implements a unit for deploying instances of inference engine to infer events from monitored video feeds, wherein the deployed instances of inference engine are based on trained instances of inference engine (e.g., initially copies thereof). Other units used to deploy inference engine instances may include, but are not limited to, computing devices, servers, cloud-based services, websites, etc., which are constructed as copies of trained instances of the inference engine for obtaining and deploying.

[0043] exist Figure 3In the example video surveillance system 300, a second instance 310A and a third instance 310B of inference engine 230 are part of corresponding enhanced inference engines 315A and 315B executed or otherwise implemented by the respective monitoring stations 115A-B to operate in conjunction with the respective monitoring stations 115A-B. In the illustrated example, enhanced inference engine 315A is executed or otherwise implemented by monitoring station 115A to infer events from monitored video feeds being processed by monitoring station 115A. For example, the second instance 310A of inference engine 230 included in enhanced inference engine 315A receives video segments from video feeds monitored by monitoring station 115A and outputs inferred events for the monitored video segments (e.g., an indication of whether a specific event is depicted in a corresponding monitored video segment within the monitored video segments). The enhanced inference engine 315A also includes an example comparator 320A, which determines updated training data by comparing the inferred events output by the second instance 310A of the inference engine 230 for the corresponding monitored video segment with the corresponding operator event labels generated by the monitoring station 115A based on operator decisions detected by each monitoring station 115A for the corresponding monitored video segment (e.g., input by human operator 130A via monitoring interface 125A, as described above). The comparator 320A reports the updated training data to the neural network 205 (e.g., via network 110).

[0044] Similarly, in Figure 3 In the illustrated example, the enhanced inference engine 315B is executed by or otherwise implemented by the monitoring station 115B to infer events from the monitored video feed being processed by the monitoring station 115B. For example, a third instance 310B of the inference engine 230 included in the enhanced inference engine 315B receives video segments from the video feed monitored by the monitoring station 115B and outputs inferred events for the monitored video segments (e.g., an indication of whether a particular event is depicted in a corresponding monitored video segment). The enhanced inference engine 315B also includes an example comparator 320B to determine updated training data by comparing the inferred events output by the third instance 310B of the inference engine 230 for the corresponding monitored video segment with corresponding operator event labels generated by the monitoring station 115B based on operator decisions detected by the corresponding monitoring station 115B for the corresponding monitored video segment (e.g., input by human operator 130B via monitoring interface 125B, as described above). Comparator 320B (e.g., via network 110) reports the updated training data to neural network 205.

[0045] exist Figure 3In the illustrated example, neural network 205 retrains a first instance of inference engine 230 included in neural network 205 using updated training data received from enhanced inference engines 315A-B implemented by the respective monitoring stations 115A-B, to improve event inference accuracy. For example, neural network 205 may retrain the first instance of inference engine 230 at periodic intervals based on one or more events (e.g., when updated training data of a threshold amount has been received from the enhanced inference engines 315A-B implemented by the respective monitoring stations 115A-B), based on operator input, etc., or any combination thereof. In some examples, the deployer 305 of the illustrated example neural network 205 then redeploys the retrained instance of inference engine 230 to one or more target monitoring locations. For example, deployer 305 may redeploy the retrained instance of inference engine 230 to update / replace a second instance 310A of inference engine 230 implemented by monitoring station 115A and / or a third instance 310B of inference engine 230 implemented by monitoring station 115B.

[0046] although Figure 3 The example video surveillance system 300 includes four imaging sensors 105A-D, a network 110, two monitoring stations 115A-B implementing two enhanced inference engines 315A-B, a neural network 205, and a database 210, but the video surveillance system 200 is not limited to this. Instead, the example video surveillance system 200 may include any number of imaging sensors 105A-D, any number of networks 110, any number of monitoring stations 115A-B implementing any number of enhanced inference engines 315A-B, any number of neural networks 205, and any number of databases 210.

[0047] exist Figure 4 The diagram shows a block diagram of a third example video surveillance system 400 that utilizes a neural network to implement video surveillance in accordance with the teachings of this disclosure. Figure 4 Example video surveillance system 400 includes Figure 1-3 Example imaging sensors 105A-D, example network 110, and example monitoring stations 115A-B are provided for video surveillance systems 100, 200, and 300. Thus, various aspects of these elements of video surveillance system 400 are combined above. Figure 1-3 The description was provided. Figure 3 The example video surveillance system 400 also includes the example neural network 205 and the example database 210 of the video surveillance systems 200 and 300 shown in Figures 2-3. Thus, aspects of these elements of the video surveillance system 400 are combined above. Figure 2-3 The description was provided.

[0048] exist Figure 4In the example video surveillance system 400 shown, the deployer 305 of the neural network 205 also deploys instances of a trained inference engine 230 for monitoring video feeds without being integrated with monitoring stations 115A-B. For example, in the video surveillance system 400, the deployer 305 of the neural network 205 can deploy instances of the trained inference engine 230 to unattended monitoring locations. Additionally or alternatively, the deployer 305 of the neural network 205 can deploy instances of the trained inference engine 230 to monitoring locations with monitoring stations (e.g., monitoring stations 115A-B), but operate independently of the monitoring stations. For example, in Figure 4 In the video surveillance system 400, the deployer 305 of the neural network 205 deploys a fourth example instance 410A and a fifth example instance 410B of the inference engine 230 to monitor the video feed independently of the monitoring stations 115A-B. In this way, additional instances of the inference engine 230 can be deployed to cost-effectively increase the monitoring capabilities of the video surveillance system 400.

[0049] although Figure 4 The example video surveillance system 400 is shown as including four imaging sensors 105A-D, a network 110, two monitoring stations 115A-B implementing two enhanced inference engines 315A-B, a neural network 205, a database 210, and two separate instances 410A-B of the inference engine 230, but the video surveillance system 200 is not limited thereto. Instead, the example video surveillance system 200 may include any number of imaging sensors 105A-D, any number of networks 110, any number of monitoring stations 115A-B implementing any number of enhanced inference engines 315A-B, any number of neural networks 205, any number of databases 210, and any number of instances 410A-B of the inference engine 230.

[0050] Furthermore, although the illustrated example video surveillance systems 200, 300, and 400 include imaging sensors 105A-D, the surveillance and monitoring utilizing neural networks disclosed herein are not limited to video surveillance. For example, the neural network techniques disclosed herein can be adapted for use with other monitoring sensors. For instance, video surveillance systems 100, 200, and / or 300 may include other sensors, attached to or replacing imaging sensors 105A-D. Such other sensors may include, but are not limited to, motion sensors, thermal / temperature sensors, acoustic sensors (e.g., microphones), electromagnetic sensors, etc. In such an example, these sensors transmit their respective data feeds via network 110 for monitoring via one or more inference engines 230, 310A, 310B, 410A, and / or 410B, possibly in conjunction with one or more monitoring stations 115A-B.

[0051] Despite Figure 1-4 The document illustrates example implementations of video surveillance systems 100, 200, 300, and 400, but... Figure 1-4 One or more of the elements, processes, and / or devices shown may be combined, divided, rearranged, omitted, eliminated, and / or implemented in any other way. Furthermore, Figure 1-4 Example video surveillance systems 100, 200, 300 and / or 400, example imaging sensors 105A-D, example network 110, example monitoring station 115A-B, example neural network 205, example database 210, example instance enhanced inference engines 315A-B and / or inference engine 230, and example instances 310A-B and / or 410A-B can be implemented by hardware, software, firmware and / or any combination of hardware, software and / or firmware. Therefore, for example, any of the example video surveillance systems 100, 200, 300 and / or 400, example imaging sensors 105A-D, example network 110, example monitoring station 115A-B, example neural network 205, example database 210, example enhanced inference engines 315A-B and / or inference engine 230, example instances 310A-B and / or 410A-B, can be implemented by one or more analog or digital circuits, logic circuits, programmable processors, application-specific integrated circuits (ASICs), programmable logic devices (PLDs) and / or field-programmable logic devices (FPLDs). When reading any device or system claim of this patent used to cover purely software and / or firmware implementations, at least one of the example video surveillance systems 100, 200, 300 and / or 400, example imaging sensors 105A-D, example network 110, example monitoring station 115A-B, example neural network 205, example database 210, example enhanced inference engines 315A-B and / or inference engine 230, and example instances 310A-B and / or 410A-B are hereby explicitly defined as including non-transitory computer-readable storage devices or storage disks, such as memory, digital versatile discs (DVDs), compact discs (CDs), Blu-ray discs, etc., including software and / or firmware. Furthermore, the example video surveillance systems 100, 200, 300 and / or 400 may include one or more elements, processes and / or devices to be attached to or replace... Figure 1-4 Those shown, and / or may include any or more of the elements, processes and devices shown.

[0052] exist Figure 5-8The flowchart shown illustrates example machine-readable instructions for implementing example video surveillance systems 100, 200, 300, and / or 400, example imaging sensors 105A-D, example network 110, example monitoring stations 115A-B, example neural network 205, example database 210, example enhanced inference engines 315A-B, and / or inference engine 230, as well as example instances 310A-B and / or 410A-B. In these examples, the machine-readable instructions comprise one or more programs executed by a processor, such as the following combination Figure 9-11 The processors 912, 1012, and / or 1112 shown in the example processor platforms 900, 1000, and 1100 discussed are examples of processors. One or more programs, or portions thereof, may be embodied in software stored on a non-transitory computer-readable storage medium such as a CD-ROM, floppy disk, hard disk drive, digital versatile disc (DVD), Blu-ray disk™, or memory associated with processors 912, 1012, and / or 1112. However, the entire one or more programs and / or portions thereof may alternatively be executed by a device other than processors 912, 1012, and / or 1112, and / or embodied in firmware or dedicated hardware (e.g., implemented by ASIC, PLD, FPLD, discrete logic, etc.). Furthermore, although references are made to... Figure 5-8 The flowcharts shown describe example programs that can be alternatively implemented using many other methods, such as example instances 310A-B and / or 410A-B, implementing example video surveillance systems 100, 200, 300 and / or 400, example imaging sensors 105A-D, example network 110, example monitoring stations 115A-B, example neural network 205, example database 210, example enhanced inference engines 315A-B and / or inference engine 230. For example, refer to Figure 5-8 The flowchart shown can change the execution order of the blocks, and / or can change, eliminate, combine some of the described blocks and / or subdivide some of the described blocks into multiple blocks. Additionally or alternatively, any or all blocks can be implemented by one or more hardware circuits (e.g., discrete and / or integrated analog and / or digital circuits, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), comparators, operational amplifiers (op-amps), logic circuits, etc.) configured to perform the corresponding operations without executing software or firmware.

[0053] As mentioned above, Figure 5-8Example processes can be implemented using encoded instructions (e.g., computer and / or machine-readable instructions) stored on non-transitory computer and / or machine-readable media (e.g., hard disk drives, flash memory, read-only memory, compact disks, digital multifunction disks, caches, random access memory, and / or any other storage device or storage disk in which information is stored for any duration (e.g., extended time periods, permanent, transient, such as temporary buffering, and / or for information caching)). As used herein, the term non-transitory computer-readable media is explicitly defined to include any type of computer-readable storage device and / or storage disk, excluding propagation signals and transmission media.

[0054] "Comprising" and "including" (and all their forms and tenses) are used herein as open-ended terms. Therefore, whenever a claim lists anything following any form of "comprising" or "including" (e.g., including, containing, comprising, including, etc.), it should be understood that additional elements, terms, etc., may be present that do not fall outside the scope of the corresponding claim. As used herein, when the phrase "at least" is used as a transitional term in the preamble of a claim, it is open-ended in the same way that the terms "comprising" and "including" are open-ended. Furthermore, as used herein, unless otherwise indicated, the terms "computer-readable" and "machine-readable" are considered equivalent.

[0055] Figure 5 It shows that it can be executed to achieve Figure 1-4 The example program 500 includes one or more of the example monitoring stations 115A-B in the example video surveillance systems 100, 200, 300, and 400. For convenience and without loss of generality, the example program 500 is derived from... Figure 4 The execution of example procedure 500 is described from the perspective of example monitoring station 115A operating in example video surveillance system 400. Referring to the preceding figures and associated written description, Figure 5Example program 500 begins execution at block 505, where monitoring station 115A accesses a video feed received from one of the imaging sensors 105A-D via network 110 and presents the accessed video feed via display 120A of monitoring station 115A. At block 510, monitoring station 115A detects an operator decision input by human operator 130A via monitoring interface 125A of monitoring station 105A. As described above, the operator decision detected at block 510 indicates whether an event of interest is depicted or otherwise represented in the monitored video feed. For example, monitoring station 115A can segment the accessed video feed into monitored video segments based on any segmentation criteria. For example, monitoring station 115A can segment the monitored video feed into continuous video segments with a given duration (e.g., 15 seconds, 30 seconds, 60 seconds, or any duration) based on changes detected in the video feed and based on the characteristics of the imaging sensor 105A-D providing the feed (e.g., the sensor's scan rate, image capture rate, etc.). In such an example, monitoring interface 125A associates an input operator decision with a specific monitored video segment currently displayed on display 120A. In some examples, if no input operator decision is detected while a given video segment is being displayed, monitoring interface 125A determines that the operator decision is that the event of interest is not described in that given video segment. As described above, in some examples, the input operator decision is a yes or no indication of whether a predetermined event of interest is depicted in the monitored video segment. However, in some examples, the input operator decision may also include a description of the type of event of interest, and a yes or no indication of whether the described event of interest is depicted in the monitored video segment.

[0056] At box 515, monitoring interface 125A of monitoring station 115A determines whether an input operator decision has been detected. If an operator decision is detected (box 515), then at box 520, monitoring interface 125A accesses the event label associated with the currently monitored video segment and updates the event label to reflect the input operator decision. For example, the event label may indicate whether a predefined event of interest is depicted in the corresponding monitored video segment. In some examples, the event label includes a description of the type of event of interest, and an indication of whether the described event of interest is depicted in the corresponding monitored video segment. In some examples, if no input operator decision is detected while a given video segment is being presented, then at box 520, monitoring interface 125A updates the event label of that video segment to indicate that no event of interest is described in the corresponding video segment. Figure 5As shown in the example, monitoring station 105A also provides the database 210 with the video clips of the video accessed at box 505 and the corresponding event tags accessed at box 520, so that the database 210 can create a record 220 of the video clips marked by the operator.

[0057] exist Figure 5 In the illustrated example, at box 525, the monitoring interface 125A of monitoring station 115A determines whether the input detected for the currently monitored video segment corresponds to an event for which an alarm should be triggered. If an alarm is to be triggered (box 525), then at box 530, monitoring interface 125A causes the alarm to be triggered. For example, at box 530, monitoring interface 125A may automatically trigger audible and / or visual alarms, contact the emergency response system to summon first responders, etc. At box 535, monitoring station 115A determines whether video surveillance monitoring should continue. If video surveillance monitoring should continue (box 535), processing returns to box 505 and subsequent boxes to enable monitoring station 115A to continue monitoring the current video feed and / or other video feeds that monitoring station 115A is receiving from imaging sensor 105A-D via network 110.

[0058] Figure 6 The document shows that it can be executed to implement what is included. Figure 2-4 Example neural network 205 and example database 210 in example video surveillance systems 200, 300, and 400, and example program 600. For convenience and without loss of generality, from... Figure 4 The execution of example program 600 is described from the perspective of example neural network 205 and example database 210 operating in example video surveillance system 400. Referring to the preceding figures and associated written description, Figure 6 Example program 600 begins execution at box 605, where database 210 stores records 220 of monitored video clips received via network 110 from monitoring stations 115A-B, along with corresponding event tags, as operator-tagged video clips, as described above. At box 610, neural network 205 uses the records 220 of operator-tagged video clips created and stored by database 210 to train a first instance or reference instance of its example inference engine 230 to infer events from the video clips, as described above. At box 615, neural network 205 deploys the trained instance of inference engine 230 to the monitoring stations to infer events from the monitored video feed, as described above.

[0059] exist Figure 7 Example program 700 is shown, which can be executed to implement the following: Figure 2-4 Example database 210 and / or execution in example video surveillance systems 200, 300 and 400 Figure 6 The processing at frame 605. For convenience and without loss of generality, starting from... Figure 4 The execution of example program 700 is described from the perspective of example database 210 operating in example video surveillance system 400. Referring to the preceding figures and associated written description, Figure 7 Example program 700 begins execution at block 705, where, as described above, database 210 receives monitored video segments and corresponding event tags from monitoring stations 115A-B via network 110. As described above, the event tags reflect decisions entered by a human operator regarding whether an event of interest is depicted or otherwise represented in the corresponding monitored video segment. At block 710, as described above, database 210 creates a record 220 of the operator-tagged video segment based on the received video segment received at block 705 and the corresponding event tags, and stores record 220 in example recording storage device 225 of database 210.

[0060] At box 715, database 210 determines whether a request for a training dataset has been received from neural network 205. As described above, in some examples, the request includes the number of records 220 requested to be included in the training dataset. In some examples, the request additionally or alternatively includes the type of event to be depicted or otherwise represented by the records 220 to be included in the training dataset. If a request for training data is received (box 715), database 210 retrieves the training set of records 220 from the record storage device 225 that satisfies the request and outputs the training set of records 220 to neural network 205 (e.g., via network 110) to facilitate the training of the neural network.

[0061] exist Figure 8 The document shows that it can be executed to implement the following: Figure 2-4 Example neural network 205 and / or execution in example video surveillance systems 200, 300 and 400 Figure 6 Example program 800 processes boxes 610 and 615. For convenience and without loss of generality, from... Figure 4 The example neural network 205 operating in the example video surveillance system 400 illustrates the execution of the example program 800 from the perspective of this example. Referring to the preceding figures and associated written description, Figure 8Example program 800 begins execution at block 805, where the example training data retrieval unit 250 of neural network 205 requests and retrieves from database 210 a set of training records 220 of operator-tagged video clips. As described above, in some examples, this request includes the number of records 220 requested to be included in the training data set. In some examples, the request additionally or alternatively includes the type of event to be depicted or otherwise represented by the records 220 to be included in the training data set.

[0062] At box 810, as described above, example training engine 240 and example comparator 245 use the obtained training set of records 220 to train example inference engine 230 of neural network 205 to infer events from reference video clips including the training set of records 220. At box 815, example deployer 305 of neural network 205 deploys instances of the trained inference engine 230 to one or more target monitoring locations as described above to infer events from monitored video feeds. For example, at box 815, deployer 305 may deploy example instances 310A-B of the trained inference engine 230 to perform or in conjunction with example monitoring stations 105A-B. Additionally or alternatively, in some examples, deployer 305 may deploy example instances 410A-B of the trained inference engine 230 to monitoring locations to perform video surveillance monitoring independently of monitoring stations 105A-B.

[0063] At box 820, neural network 205 acquires (e.g., via network 110) updated training data determined by one or more of the instances 310A-B of the trained inference engine 230 or monitoring stations 105A-B operating in conjunction with it. For example, and as described above, monitoring stations 105A-B can determine updated training data by comparing the inferred events output by instances 310A-B of the trained inference engine 230 for the corresponding monitored video segment with corresponding operator event labels generated by monitoring stations 115A-B based on operator decisions made by human operators 130A-B for the corresponding monitored video segment. At box 825, as described above, the neural network uses the updated training data to retrain a first instance or reference instance of its inference engine 230. At box 830, as described above, deployer 305 redeploys the retrained instances of the inference engine 230 to one or more target monitoring locations. At box 835, neural network 205 determines whether the retraining of its inference engine 230 should continue. If retraining is to continue, processing returns to box 820 and subsequent boxes so that neural network 205 can continue retraining its inference engine 230 based on updated training data received from monitoring stations 105A-B.

[0064] Figure 9 It is constructed to execute Figure 5 , Figure 6 , Figure 7 and / or Figure 8 Instructions to achieve Figure 1-4 The block diagram of the example processor platform 900 for example monitoring stations 115A-B is provided. For convenience and without loss of generality, the example processor platform 900 is described from the perspective of implementing example monitoring station 115A. Processor platform 900 can be, for example, a server, a personal computer, a mobile device (e.g., a cellular phone, a smartphone, or an iPad). TM Tablet computers, personal digital assistants (PDAs), internet devices, or any other type of computing device.

[0065] The processor platform 900 shown in the example includes a processor 912. The processor 912 shown in the example is hardware. For example, the processor 912 can be implemented by one or more integrated circuits, logic circuits, microprocessors, or controllers from any desired family or manufacturer. The hardware processor 912 can be a semiconductor-based (e.g., silicon-based) device.

[0066] The processor 912 of the illustrated example includes local memory 913 (e.g., cache). The processor 912 of the illustrated example communicates with main memory, including volatile memory 914 and non-volatile memory 916, via link 918. Link 918 may be implemented by a bus, one or more point-to-point connections, etc., or a combination thereof. Volatile memory 914 may be implemented by synchronous dynamic random access memory (SDRAM), dynamic random access memory (DRAM), RAMBUS dynamic random access memory (RDRAM), and / or any other type of random access memory device. Non-volatile memory 916 may be implemented by flash memory and / or any other desired type of memory device. Access to main memory 914, 916 is controlled by a memory controller.

[0067] The processor platform 900 shown in the example also includes interface circuitry 920. Interface circuitry 920 can be implemented using any type of interface standard, such as an Ethernet interface, a Universal Serial Bus (USB) interface, and / or a PCI express interface.

[0068] In the illustrated example, one or more input devices 922 are connected to interface circuitry 920. Input devices 922 allow a user to input data and commands into processor 912. Input devices can be implemented via, for example, audio sensors, microphones, cameras (still or video), keyboards, buttons, mice, touchscreens, touchpads, trackballs, track bars (e.g., contour points), voice recognition systems, and / or any other human-machine interface. Similarly, many systems (e.g., processor platform 900) allow a user to control the computer system and provide data to the computer using physical gestures (e.g., but not limited to hand or body movements, facial expressions, and facial recognition). In some examples, input data device 922 implements example monitoring interface 125A.

[0069] One or more output devices 924 are also connected to the interface circuitry 920 of the illustrated example. The output devices 924 may be implemented, for example, by display devices such as light-emitting diodes (LEDs), organic light-emitting diodes (OLEDs), liquid crystal displays, cathode ray tube displays (CRTs), touchscreens, haptic output devices, printers, and / or speakers. Therefore, the interface circuitry 920 of the illustrated example typically includes a graphics driver card, a graphics driver chip, or a graphics driver processor. In some examples, the output device 924 implements the example display 120A.

[0070] The interface circuitry 920 of the example shown also includes communication devices, such as transmitters, receivers, transceivers, modems, and / or network interface cards, to facilitate data exchange with external machines (e.g., computing devices of any computer equipment) via a network 926 such as example network 110 (e.g., Ethernet connection, digital subscriber line (DSL), telephone line, coaxial cable, cellular telephone system, etc.).

[0071] The processor platform 900 shown in the example also includes one or more mass storage devices 928 for storing software and / or data. Examples of such mass storage devices 928 include floppy disk drives, hard disk drives, compact disk drives, Blu-ray disc drives, RAID (Redundant Array of Independent Disks) systems, and digital multifunction disc (DVD) drives.

[0072] Corresponding to Figure 5 , 6 The encoded instructions 932 of instructions 7 and / or 8 can be stored in a mass storage device 928, a volatile memory 914, a non-volatile memory 916, a local memory 913, and / or a removable tangible computer-readable storage medium, such as a CD or DVD 936.

[0073] Figure 10 It is constructed to execute Figure 5 , Figure 6 , Figure 7 and / or Figure 8 The instructions to implement Figure 2-4 The example database 210 is a block diagram of an example processor platform 1000. The processor platform 1000 can be, for example, a server, a personal computer, a mobile device (e.g., a cellular phone, a smartphone, or an iPad). TM Tablets, PDAs, internet devices, or any other type of computing device.

[0074] The processor platform 1000 shown in the example includes a processor 1012. The processor 1012 shown in the example is hardware. For example, the processor 1012 may be implemented by one or more integrated circuits, logic circuits, microprocessors, or controllers from any desired family or manufacturer. The hardware processor 1012 may be a semiconductor-based (e.g., silicon-based) device.

[0075] The processor 1012 of the illustrated example includes local memory 1013 (e.g., cache). The processor 1012 of the illustrated example communicates with main memory, including volatile memory 1014 and non-volatile memory 1016, via link 1018. Link 1018 may be implemented by a bus, one or more point-to-point connections, etc., or a combination thereof. Volatile memory 1014 may be implemented by SDRAM, DRAM, RDRAM, and / or any other type of random access memory device. Non-volatile memory 1016 may be implemented by flash memory and / or any other desired type of memory device. Access to main memory 1014, 1016 is controlled by a memory controller.

[0076] The processor platform 1000 shown in the example also includes interface circuitry 1020. Interface circuitry 1020 can be implemented using any type of interface standard, such as an Ethernet interface, USB, and / or PCI express interface.

[0077] In the illustrated example, one or more input devices 1022 are connected to interface circuitry 1020. Input devices 1022 allow users to input data and commands into processor 1012. Input devices can be implemented using, for example, audio sensors, microphones, cameras (still or video), keyboards, buttons, mice, touchscreens, touchpads, trackballs, track bars (e.g., contour points), speech recognition systems, and / or any other human-computer interface. Furthermore, many systems (e.g., processor platform 1000) allow users to control the computer system and provide data to the computer using physical gestures, such as, but not limited to, hand or body movements, facial expressions, and facial recognition.

[0078] One or more output devices 1024 are also connected to the interface circuitry 1020 of the illustrated example. The output devices 1024 may be implemented, for example, by display devices (e.g., LED, OLED, liquid crystal display, CRT display, touchscreen, haptic output device, printer, and / or speaker). Therefore, the interface circuitry 1020 of the illustrated example typically includes a graphics driver card, a graphics driver chip, or a graphics driver processor.

[0079] The interface circuit 1020 of the example shown also includes communication devices, such as transmitters, receivers, transceivers, modems, and / or network interface cards, to facilitate data exchange with external machines (e.g., any kind of computing device) via a network 1026 such as example network 110 (e.g., Ethernet connection, DSL, telephone line, coaxial cable, cellular telephone system, etc.).

[0080] The processor platform 1000 shown in the example also includes one or more mass storage devices 1028 for storing software and / or data. Examples of such mass storage devices 1028 include floppy disk drives, hard disk drives, compact disk drives, Blu-ray disc drives, RAID systems, and DVD drives. In some examples, the mass storage device 1028 may implement an example recording storage device 225. Additionally or alternatively, in some examples, volatile memory 1014 may implement the example recording storage device 225.

[0081] Corresponding to Figure 5 , 6 The encoded instructions 1032 of instructions 7 and / or 8 can be stored in a mass storage device 1028, a volatile memory 1014, a non-volatile memory 1016, a local memory 1013, and / or a removable tangible computer-readable storage medium, such as a CD or DVD 1036.

[0082] Figure 11 It is constructed to execute Figure 5 , Figure 6 , Figure 7 and / or Figure 8 The instructions to implement Figure 2-4 The example neural network 205 is illustrated in the block diagram of an example processor platform 1100. The processor platform 1100 can be, for example, a server, a personal computer, a mobile device (e.g., a cellular phone, a smartphone, or an iPad). TM Tablets, PDAs, internet devices, or any other type of computing device.

[0083] The processor platform 1100 shown in the example includes a processor 1112. The processor 1112 shown in the example is hardware. For example, the processor 1112 can be implemented by one or more integrated circuits, logic circuits, microprocessors, or controllers from any desired family or manufacturer. The hardware processor 1112 can be a semiconductor-based (e.g., silicon-based) device. In this example, the processor 1112 implements an example inference engine 230, an example training engine 240, an example comparator 245, an example training data retrieval unit 250, and / or an example deployer 305.

[0084] The processor 1112 of the illustrated example includes local memory 1113 (e.g., cache). The processor 1112 of the illustrated example communicates with main memory, including volatile memory 1114 and non-volatile memory 1116, via link 1118. Link 1118 may be implemented by a bus, one or more point-to-point connections, etc., or a combination thereof. Volatile memory 1114 may be implemented by SDRAM, DRAM, RDRAM, and / or any other type of random access memory device. Non-volatile memory 1116 may be implemented by flash memory and / or any other desired type of memory device. Access to main memory 1114, 1116 is controlled by a memory controller.

[0085] The processor platform 1100 shown in the example also includes interface circuitry 1120. Interface circuitry 1120 can be implemented using any type of interface standard, such as an Ethernet interface, USB, and / or PCI express interface.

[0086] In the illustrated example, one or more input devices 1122 are connected to interface circuitry 1120. Input devices 1122 allow users to input data and commands into processor 1112. Input devices can be implemented using, for example, audio sensors, microphones, cameras (still or video), keyboards, buttons, mice, touchscreens, touchpads, trackballs, track bars (e.g., contour points), speech recognition systems, and / or any other human-computer interface. Furthermore, many systems (e.g., processor platform 1100) allow users to control the computer system and provide data to the computer using physical gestures, such as, but not limited to, hand or body movements, facial expressions, and facial recognition.

[0087] One or more output devices 1124 are also connected to the interface circuitry 1120 of the illustrated example. The output devices 1124 may be implemented, for example, by display devices (e.g., LED, OLED, liquid crystal display, CRT display, touchscreen, haptic output device, printer, and / or speaker). Therefore, the interface circuitry 1120 of the illustrated example typically includes a graphics driver card, a graphics driver chip, or a graphics driver processor.

[0088] The interface circuit 1120 of the example shown also includes communication devices, such as transmitters, receivers, transceivers, modems, and / or network interface cards, to facilitate data exchange with external machines (e.g., any kind of computing device) via a network 1126 such as example network 110 (e.g., Ethernet connection, DSL, telephone line, coaxial cable, cellular telephone system, etc.).

[0089] The processor platform 1100 shown in the example also includes one or more mass storage devices 1128 for storing software and / or data. Examples of such mass storage devices 1128 include floppy disk drives, hard disk drives, compact disk drives, Blu-ray disc drives, RAID systems, and DVD drives.

[0090] Corresponding to Figure 5 , 6 The encoded instructions 1132 of instructions 7 and / or 8 can be stored in mass storage device 1128, in volatile memory 1114, in non-volatile memory 1116, in local memory 1113 and / or on a removable tangible computer-readable storage medium, such as a CD or DVD 1136.

[0091] The foregoing disclosure provides examples of video surveillance utilizing neural networks. This document discloses further examples, including topics such as: video surveillance systems for implementing video surveillance using neural networks; at least one computer-readable storage medium comprising instructions (which, when executed by at least one processor, cause at least one processor to implement video surveillance using a neural network); units for implementing video surveillance using a neural network; and video surveillance methods for performing video surveillance using a neural network. The disclosed examples can be implemented individually and / or in one or more combinations.

[0092] Based on the foregoing, it will be appreciated that this document has disclosed example methods, apparatuses, systems, and artifacts (e.g., physical storage media) for implementing video surveillance using neural networks. The disclosed examples include neural networks with an inference engine trained to detect or infer events in a monitored video feed. The neural network inference engine is trained using a training set of reference video clips with reference event labels indicating whether the reference video clips depict a defined event. The trained inference engine is then deployed to one or more monitoring locations and operated (e.g., in parallel, asynchronous, collaborative, etc.) to infer whether one or more trained events (e.g., security breaches, presence of individuals, arrival of packages, etc.) are represented in their respective monitored video feeds. In some examples, instances of trained / retrained inference engines can be deployed to unmanned monitoring locations or to provide additional capacity at manned locations, allowing the capacity of the video surveillance system to easily scale as demand increases.

[0093] The foregoing disclosure provides examples of video surveillance using neural networks. Other examples of video surveillance using neural networks are disclosed below. The disclosed examples can be implemented individually and / or in one or more combinations.

[0094] Example 1 is a video surveillance system including a database for storing operator-tagged video clips, the operator-tagged video clips including reference video clips and corresponding reference event tags for describing the reference video clips. The system of Example 1 also includes a neural network comprising: a first instance of an inference engine; and a training engine for training the first instance of the inference engine based on a training set of the operator-tagged video clips obtained from the database, the first instance of the inference engine being used to infer events from the operator-tagged video clips included in the training set. The system of Example 1 also includes a second instance of the inference engine for inferring events from monitored video feeds, the second instance of the inference engine being based on the first instance of the inference engine.

[0095] Example 2 includes the subject of Example 1, wherein the reference event label indicates whether the corresponding reference video clip depicts a defined event.

[0096] Example 3 includes the subject of Example 1 and / or 2, wherein the first reference event label in the reference event label corresponding to the first reference video segment in the reference video segment indicates (i) the event type and (ii) whether the first reference video segment in the reference video segment depicts the event type.

[0097] Example 4 includes one or more of the topics in Examples 1-3, and also includes a monitoring station comprising: a display for presenting a first monitored video feed in the monitored video feed; and a monitoring interface for generating operator event tags based on operator decisions corresponding to monitored video segments of the first monitored video feed in the monitored video feed.

[0098] Example 5 includes the subject of Example 4, wherein the database communicates with the monitoring station to receive a first monitored video segment from the monitored video segments and a first operator event tag from the operator event tags corresponding to the first monitored video segment from the monitored video segments. The database is used to store the first monitored video segment from the monitored video segments and the corresponding first operator event tag from the operator event tags as a first reference video segment from the reference video segments and a first operator-marked video segment from the operator-marked video segments, which includes the corresponding first reference event tag from the reference event tags.

[0099] Example 6 includes the subject of Example 5, wherein the monitoring station is also used to implement a second instance of the inference engine.

[0100] Example 7 includes the subject of Example 6, wherein a second instance of the inference engine is used to output an inferred event for a second monitored video segment in a monitored video segment of a first monitored video feed, the monitoring interface is used to generate a second operator event label in the operator event label based on an operator decision detected for the second monitored video segment in the monitored video segment, and the monitoring station further includes a comparator for comparing the inferred event and the second operator event label in the operator event label to obtain updated training data.

[0101] Example 8 includes the subject of Example 7, wherein the neural network communicates with the monitoring station to receive the updated training data, and the training engine is used to retrain a first instance of the inference engine based on the updated training data.

[0102] Example 9 includes at least one non-transitory computer-readable storage medium, including computer-readable instructions that, when executed, cause one or more processors to at least: train a first instance of an inference engine based on a training set of operator-tagged video clips, the operator-tagged video clips including reference video clips and corresponding reference event labels for describing the reference video clips, the first instance of the inference engine being used to infer events from the operator-tagged video clips included in the training set; and deploy a second instance of the inference engine to infer events from monitored video feeds, the second instance of the inference engine being based on the first instance of the inference engine.

[0103] Example 10 includes the subject of Example 9, wherein the reference event label indicates whether the corresponding reference video clip depicts a defined event.

[0104] Example 11 includes the subject of Examples 9 and / or 10, wherein a first reference event label in the reference event label corresponding to a first reference video segment in the reference video segment indicates (i) the event type and (ii) whether the first reference video segment in the reference video segment depicts the event type.

[0105] Example 12 includes the subject of one or more of Examples 9-11, wherein the computer-readable instructions, when executed, cause the one or more processors to retrieve a first reference video segment from the reference video segment and a corresponding first reference event tag from the reference event tags from the monitoring station.

[0106] Example 13 includes the subject of Example 12, wherein the computer-readable instructions, when executed, cause the one or more processors to deploy a second instance of the inference engine to the monitoring station.

[0107] Example 14 includes the subject of Example 13, wherein when the second instance of the inference engine is initially deployed to the monitoring station, the second instance of the inference engine is a copy of the first instance of the inference engine.

[0108] Example 15 includes the subject of Example 13, wherein the monitoring station is used to obtain updated training data by comparing (i) inferred events output by a second instance of the inference engine for a segment fed by the monitored video, and (ii) operator event labels generated by the monitoring station for the segment fed by the monitored video, and the computer-readable instructions, when executed, also cause the one or more processors to retrain a first instance of the inference engine based on the updated training data.

[0109] Example 16 is an apparatus including units for obtaining operator-tagged video clips, the operator-tagged video clips including reference video clips and corresponding reference event tags for describing the reference video clips. The apparatus of Example 16 further includes units for training a first instance of an inference engine based on a training set of the operator-tagged video clips, the first instance of the inference engine being used to infer events from the operator-tagged video clips included in the training set. The apparatus of Example 16 further includes units for deploying a second instance of the inference engine to infer events from monitored video feeds, the second instance of the inference engine being based on the first instance of the inference engine.

[0110] Example 17 includes the subject of Example 16, wherein the reference event label indicates whether the corresponding reference video clip depicts a defined event.

[0111] Example 18 includes the subject of Examples 16 and / or 17, wherein a first reference event label in the reference event label corresponding to a first reference video segment in the reference video segment indicates (i) the event type and (ii) whether the first reference video segment in the reference video segment depicts the event type.

[0112] Example 19 includes one or more of the topics in Examples 16-18, wherein the unit for obtaining a record of the video clip marked by the operator is used to obtain a first reference video clip in the reference video clip and a corresponding first reference event tag in the reference event tag from the monitoring station.

[0113] Example 20 includes the subject of Example 19, wherein the unit for deploying a second instance of the inference engine is used to deploy the second instance of the inference engine to the monitoring station.

[0114] Example 21 includes the subject of Example 20, wherein when the second instance of the inference engine is initially deployed to the monitoring station, the second instance of the inference engine is a copy of the first instance of the inference engine.

[0115] Example 22 includes the subject of Example 20, wherein the monitoring station is used to obtain updated training data by comparing the following two items: (i) inferred events output by a second instance of the inference engine for a segment fed by the monitored video, and (ii) operator event labels generated by the monitoring station for the segment fed by the monitored video, and the unit used to train a first instance of the inference engine is also used to retrain the first instance of the inference engine based on the updated training data.

[0116] Example 23 is a video surveillance method comprising: training a first instance of an inference engine based on a training set of operator-tagged video clips by executing instructions with at least one processor, the operator-tagged video clips including reference video clips and corresponding reference event labels for describing the reference video clips, the first instance of the inference engine being used to infer events from the operator-tagged video clips included in the training set. The video surveillance method of Example 23 further comprises: deploying a second instance of the inference engine by executing instructions with the at least one processor to infer events from a monitored video feed, the second instance of the inference engine being based on the first instance of the inference engine.

[0117] Example 24 includes the subject of Example 23, wherein the reference event label indicates whether the corresponding reference video clip depicts a defined event.

[0118] Example 25 includes the subject of Examples 23 and / or 24, wherein a first reference event label in the reference event label corresponding to a first reference video segment in the reference video segment indicates (i) the event type and (ii) whether the first reference video segment in the reference video segment depicts the event type.

[0119] Example 26 includes one or more of the topics in Examples 23-25, wherein the record of accessing the operator-tagged video segment includes obtaining a first reference video segment from the reference video segment and a corresponding first reference event tag from the reference event tags obtained from the monitoring station.

[0120] Example 27 includes the subject of Example 26, wherein the deployment of the second instance of the inference engine includes deploying the second instance of the inference engine to the monitoring station.

[0121] Example 28 includes the subject of Example 27, wherein when the second instance of the inference engine is initially deployed to the monitoring station, the second instance of the inference engine is a copy of the first instance of the inference engine.

[0122] Example 29 includes the subject of Example 27, wherein the monitoring station is used to obtain updated training data by comparing the following two items: (i) inferred events output by a second instance of the inference engine for a segment fed by the monitored video, and (ii) operator event labels generated by the monitoring station for the segment fed by the monitored video, and training of the first instance of the inference engine also includes retraining the first instance of the inference engine based on the updated training data.

[0123] Although certain example methods, apparatuses, and articles of manufacture have been disclosed herein, the scope of this patent is not limited thereto. Rather, this patent covers all methods, apparatuses, and articles of manufacture that fall fully within the scope of the claims of this patent.

Claims

1. At least one non-transitory computer-readable medium, including computer-readable instructions, for causing one or more processors to at least: A machine learning model is trained based on a training dataset to generate a trained machine learning model, the machine learning model being a neural network including a first instance of an inference engine, a training engine, and a second instance of the inference engine, wherein the first instance of the inference engine is used to infer events from video clips labeled by operators included in the training dataset, wherein the training engine is used to train the first instance of the inference engine based on the training dataset, and wherein the second instance of the inference engine is used to infer events from monitored video clip feeds, the training dataset including a first monitored video clip having a corresponding first operator event label describing the events depicted by the first monitored video clip, the first operator event label being assigned to the first monitored video clip based on human input; A second instance of the inference engine is deployed to a monitoring location with a monitoring station, the monitoring station being used to execute the second instance of the inference engine to process video data from the imaging sensor; Updated training data is obtained by comparing (1) the second operator event label generated by the monitoring interface of the monitoring station based on the operator decision detected for the second monitored video segment with (2) the inferred event output by the second instance of the inference engine for the second monitored video segment; as well as The training engine is used to retrain the first instance of the inference engine based on the updated training data.

2. The at least one non-transitory computer-readable medium according to claim 1, wherein, The instructions are used to cause the one or more processors to retrain a first instance of the inference engine in response to an event.

3. The at least one non-transitory computer-readable medium according to claim 1, wherein, The instructions are used to cause the one or more processors to periodically retrain a first instance of the inference engine.

4. The at least one non-transitory computer-readable medium according to claim 1, wherein, The imaging sensor is implemented by at least one of the following: a camera, a smartphone, a photodiode, or a photodetector.

5. The at least one non-transitory computer-readable medium according to claim 1, wherein, The instructions are used to instruct the one or more processors to retrieve the training dataset from the cloud service.

6. The at least one non-transitory computer-readable medium according to claim 1, wherein, The training dataset includes a first training dataset and a second training dataset. The first training dataset is based on a first group of video clips labeled by a first group of one or more humans. The second training dataset is based on a second group of video clips labeled by a second group of one or more humans. The second group of video clips is different from the first group of video clips, and the second group of one or more humans is different from the first group of one or more humans.

7. A method comprising: A trained machine learning model is generated by executing instructions using at least one processor based on a training dataset. The machine learning model is a neural network comprising a first instance of an inference engine, a training engine, and a second instance of the inference engine. The first instance of the inference engine is used to infer events from operator-tagged video clips included in the training dataset. The training engine is used to train the first instance of the inference engine based on the training dataset. The second instance of the inference engine is used to infer events from monitored video clip feeds. The training dataset includes a first monitored video clip having a corresponding first operator event label describing the events depicted by the first monitored video clip. The first operator event label is assigned to the first monitored video clip based on human input. By using the at least one processor to execute instructions, a second instance of the inference engine is deployed to a monitoring location with a monitoring station, the monitoring station being used to execute the second instance of the inference engine to process video data from an imaging sensor; Updated training data is obtained by executing instructions using the at least one processor, wherein the updated training data is obtained by comparing (1) a second operator event label generated by the monitoring interface of the monitoring station based on an operator decision detected for the second monitored video segment with (2) an inferred event output by a second instance of the inference engine for the second monitored video segment; as well as By using the at least one processor to execute instructions, the training engine retrains a first instance of the inference engine based on the updated training data.

8. The method according to claim 7, wherein, The retraining of the first instance of the inference engine is in response to an event.

9. The method according to claim 7, wherein, Retraining of the first instance of the inference engine is performed periodically.

10. The method according to claim 7, wherein, The imaging sensor is implemented by at least one of the following: a camera, a smartphone, a photodiode, or a photodetector.

11. The method according to claim 7, wherein, The training dataset was obtained from a cloud service.

12. The method according to claim 7, wherein, The training dataset includes a first training dataset and a second training dataset. The first training dataset is based on a first group of video clips labeled by a first group of one or more humans. The second training dataset is based on a second group of video clips labeled by a second group of one or more humans. The second group of video clips is different from the first group of video clips, and the second group of one or more humans is different from the first group of one or more humans.

13. A system comprising: At least one memory; Machine-readable instructions; as well as One or more computing devices including a processor circuit module are configured to execute the machine-readable instructions to perform the method according to any one of claims 7 to 12.

14. An apparatus comprising: At least one memory; Machine-readable instructions; as well as A processor circuit module for executing the machine-readable instructions to perform the method according to any one of claims 7 to 12.

15. An apparatus comprising a module for performing the method according to any one of claims 7 to 12.

Images