Method for operating a security system

Abstract

A method for operating a security system of a facility includes receiving video from a video camera of the security system of the facility. An event of interest occurring in the video is identified via video analysis, and the event of interest is classified into one of a plurality of event severity levels. When the event of interest is classified as having a first event severity level of the plurality of event severity levels, an alert is automatically generated and sent to an operator of the security system. When the event of interest is classified as having a second event severity level of the plurality of event severity levels, an alert is automatically generated and sent to the first responder.

Description

Technical Field

[0001] The present disclosure relates generally to video security systems, and more particularly to identifying events in video security systems. Background Art

[0002] Many security systems are in use. Many security systems employ a large number of video cameras, thereby generating a large number of video streams. It may be difficult for a system operator to be able to view all video streams simultaneously in order to detect all possible events that are occurring. Furthermore, when investigating an event that has occurred, it may be difficult and time consuming to review all appropriate video streams in order to find video evidence of exactly what occurred during the event. What is desired is a method for operating a security system in which events of interest are detected and classified, and various alarms are sounded. Summary of the invention

[0003] The present disclosure relates generally to video security systems, and more specifically to identifying events in video security systems. An example may be found in a method for operating a security system of a facility. The illustrative method includes receiving a video from a video camera of a security system of the facility. An event of interest occurring in the video is identified via video analysis, and the event of interest is classified into one of a plurality of event severity levels. When the event of interest is classified as having a first event severity level in the plurality of event severity levels, an alert is sent to an operator of the security system. When the event of interest is classified as having a second event severity level in the plurality of event severity levels, an alert is sent to a first responder (e.g., police, fire, ambulance).

[0004] Another example may be found in a method for operating a security system of a facility. The method includes receiving a video from a video camera of a security system of the facility. Identifying an event of interest occurring in the video via video analysis. Identifying a suspect in the video that is associated with the event of interest. Identifying an image of the face of the suspect in the video. Submitting the image of the face of the suspect to a facial recognition engine to identify the identity of the suspect. Identifying an access card of the identified suspect, wherein the access card needs to be presented to an access card reader of the security system in order for the identified suspect to access an area of ​​the facility. Activating an access card trail for the access card of the identified suspect, the access card trail tracking the subsequent movement of the suspect in the facility, wherein the access card trail is configured to report each time the access card of the identified suspect is presented to the access card reader of the security system together with the location of the corresponding access card reader in the facility.

[0005] Another example can be found in a method for operating a security system of a facility. The method includes: receiving a video from a video camera of the security system of the facility; identifying an event of interest occurring in the video via video analysis; generating one or more captions describing the event of interest; extracting a first video clip of the event of interest; compiling a report including the first video clip of the event of interest and the one or more captions describing the event of interest; and exporting the report.

[0006] The above summary is provided to facilitate understanding of some innovative features unique to the present disclosure, but is not intended to be a complete description. A comprehensive understanding of the present disclosure can be obtained by viewing the entire specification, claims, drawings and abstract as a whole. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The present disclosure may be more fully understood by considering the following description of various examples in conjunction with the accompanying drawings, in which:

[0008] Figure 1 is a schematic block diagram illustrating an exemplary security system;

[0009] Figure 2A and Figure 2B are flow charts that collectively illustrate illustrative methods for operating a security system for a facility;

[0010] Figure 3A and Figure 3B are flow charts that collectively illustrate illustrative methods for operating a security system for a facility;

[0011] Figure 4 is a flow chart illustrating an illustrative method for operating a security system of a facility;

[0012] Figure 5 is a flow chart illustrating an exemplary method;

[0013] Figure 6 is a flow chart illustrating an exemplary method;

[0014] Figure 7 is a flow chart illustrating an exemplary method; and

[0015] Figure 8 is a schematic diagram of an exemplary incident report that may be generated.

[0016] Although the present disclosure is subject to various modifications and alternative forms, the details thereof have been shown by way of example in the accompanying drawings and will be described in detail. However, it should be understood that it is not intended to limit the present disclosure to the specific examples described. On the contrary, it is intended to cover all modifications, equivalents and alternatives that fall within the essence and scope of the present disclosure. DETAILED DESCRIPTION

[0017] The following description should be read with reference to the accompanying drawings, in which similar elements in different drawings are numbered in the same manner. The accompanying drawings are not necessarily drawn to scale, and they depict examples that are not intended to limit the scope of the present disclosure. Although examples of various elements are illustrated, those skilled in the art will recognize that many of the examples provided have suitable alternatives that can be utilized.

[0018] It is assumed herein that all numbers are modified by the term "about" unless the content clearly dictates otherwise. The recitation of numerical ranges by endpoints includes all numbers subsumed within that range (eg, 1 to 5 includes 1, 1.5, 2, 2.75, 3, 3.80, 4, and 5).

[0019] As used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the content clearly dictates otherwise. As used in this specification and the appended claims, the term "or" is generally employed in its sense including "and / or" unless the content clearly dictates otherwise.

[0020] It should be noted that references to "one embodiment", "some embodiments", "other embodiments", etc. in the specification indicate that the described embodiments may include a particular feature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic. Moreover, these phrases do not necessarily refer to the same embodiment. In addition, when a particular feature, structure, or characteristic is described in conjunction with one embodiment, it is contemplated that the feature, structure, or characteristic is described in conjunction with one embodiment, and it is contemplated that the feature, structure, or characteristic may be applied to other embodiments regardless of whether explicitly described, unless otherwise clearly stated.

[0021] Figure 1 1 is a schematic block diagram showing an exemplary security system 10 that may be deployed at a facility. The security system 10 includes several video cameras 12, respectively labeled 12a, 12b, and 12c. Although a total of three video cameras 12 are shown, this is merely exemplary, as the security system 10 may include any number of video cameras 12, and may include significantly more than three video cameras 12. The video camera 12 may include a fixed field of view (FOV) camera. The video camera 12 may include a variable FOV camera, such as, but not limited to, a pan-tilt-zoom (PTZ) camera configured to change its FOV. The video camera 12 may include both mounted video cameras and portable video cameras. In some cases, the video camera 12 may include a camera mounted to a drone.

[0022] The video cameras 12 are configured to provide their video streams to a monitoring station 14. In some cases, the monitoring station 14 is installed at the same facility as the video cameras 12. In some cases, the monitoring station 14 may be installed at a location remote from where the video cameras 12 are installed or otherwise located. The video cameras 12 may provide video streams to the monitoring station 14 over a wired or wireless network, for example, using any of a variety of different communication protocols. In some cases, the video cameras 12 may be configured to communicate with each other over a peer-to-peer network that allows a first video camera 12 to tell a second video camera 12 that a scene within the FOV of the first video camera 12 appears to be moving toward or into the FOV of the second video camera 12, thereby alerting the second video camera 12. In some cases, the video cameras 12 may be configured to execute at least some video analysis algorithms to, for example, detect events of interest in the FOV.

[0023] In some cases, monitoring station 14 includes a display 16 that can be used to display one or more real-time video streams and / or one or more recorded video streams and / or video clips. Monitoring station 14 can be operated by a user viewing the video streams and / or video clips displayed on display 16. Monitoring station 14 can be a computer, such as a desktop computer or a laptop computer. In some cases, at least some of the functionality of monitoring station 14 can be present within one or more cloud-based servers.

[0024] In some cases, a user may be able to send an alert based on what the user sees in the displayed video. In some cases, monitoring station 14 includes a controller 18 that is configured to control the operation of monitoring station 14. In some cases, controller 18 may be configured to execute a plurality of different video analysis algorithms to detect one or more events of interest in a video stream. In some cases, a video camera may execute a plurality of different video analysis algorithms to detect one or more events of interest in a corresponding video stream.

[0025] In some cases, the controller 18 may be configured to create and issue one or more alarms based on what the controller 18 (and / or the video camera 12) sees within the video stream (e.g., events of interest identified by the video analytics algorithm). In some cases, there may be several possible events of different severity levels, and several possible alarms of different severity levels. As an example, some alarms may include sending an alarm to the display 16 for viewing by an operator of the monitoring station. Some alarms may include sending an alarm to a first responder 22. The first responder 22 may be, for example, a police department, a fire department, and / or an ambulance. The first responder 22 may be the security department of the facility in which the security system 10 is installed. In some cases, an alarm may be sent to the first responder 22 as well as the display 16.

[0026] In some cases, the controller 18 may be configured to track a particular individual's movement through the facility by tracking when and where the individual presents his / her access card to an access card reader 20 (labeled 20a, 20b, and 20c, respectively). In some cases, the controller 18 may instruct an access card reader 20 to deny access to a particular individual using a particular access card reader 20 to help control or isolate the particular individual.

[0027] Figure 2A and Figure 2B 2 is a flowchart collectively illustrating an exemplary method 24 for operating a security system of a facility, such as the security system 10. The exemplary method 24 includes receiving a video from a video camera of the security system of the facility, such as the video camera 12, as indicated by box 26. Identifying an event of interest occurring in the video, as indicated by box 28. In some cases, a video analysis algorithm running on the monitoring station 14 and / or one or more video cameras 12 is used to identify the event of interest. The event of interest may include a person exhibiting an interesting behavior. Examples include one or more of the following: a person carrying a weapon, a person having a facial expression showing fear, a person kicking another person, a person punching another person, a person throwing an object, a person dropping and leaving an object, a person shouting and / or screaming, and a person running. In some cases, identifying an event of interest occurring in the video may include: using artificial intelligence to identify the event of interest, wherein the artificial intelligence includes a situational awareness event model. In some cases, the event of interest is classified into one of a plurality of event severity levels, as indicated by box 30.

[0028] When the event of interest is classified as having a first event severity level among the plurality of event severity levels, an alert is sent to an operator of the security system, as indicated by block 32. For example, the first event severity level may be a lower severity level alert. When the event of interest is classified as having a second event severity level among the plurality of event severity levels, an alert is sent to a first responder, as indicated by block 34. For example, the second event severity level may be a higher severity level alert. In some cases, method 24 may include sending an alert to both an operator of the security system and a first responder when the event of interest is classified as having a second event severity level among the plurality of event severity levels, as indicated by block 36.

[0029] In some cases, method 24 may include generating one or more subtitles describing the event of interest, as indicated by box 38. In some cases, the one or more subtitles may include natural language subtitles describing the event of interest occurring in the video. The video may include a video track and a sound track, and the one or more subtitles may include subtitles describing the event of interest in the video track and / or subtitles describing the event of interest in the sound track. In some cases, generating one or more subtitles describing the event of interest may include: generating one or more subtitles using artificial intelligence, wherein the artificial intelligence includes a context-aware video subtitle model that generates natural language subtitles for the event of interest. In some cases, the context-aware video subtitle generation model may be run on the monitoring station 14 and / or one or more video cameras 12.

[0030] A first video clip capturing the event of interest may be extracted, as indicated at block 40. A report including the first video clip of the event of interest and one or more captions describing the event of interest may be compiled, as indicated at block 42. In some cases, the report may be generated by monitoring station 14 and / or one or more video cameras 12. The report may be exported, as indicated at block 44.

[0031] continue Figure 2B , method 24 may include compiling a report including a first video clip of the event of interest, a second video clip of the event of interest, and one or more captions describing the event of interest, as indicated by block 48. In some cases, the first video clip may be captured by a first video camera and the second video clip may be captured by a second video camera. Sometimes, the first video camera and the second video camera communicate via a peer-to-peer network.

[0032] In some cases, the method 24 may include identifying a suspect in the video associated with the event of interest, as indicated by block 50. An image of the face of the suspect in the video may be identified in the video (e.g., in a video frame), as indicated by block 52. The image of the face of the suspect may be submitted to a facial recognition engine to identify the identity of the suspect, as indicated by block 54. An access card of the identified suspect may then be identified, wherein the access card needs to be presented to an access card reader 20 of the security system 10 in order for the identified suspect to access areas of the facility, as indicated by block 56. An access card trail may be activated in the monitoring station 14 for the access card of the identified suspect in order to track the subsequent movement of the suspect in the facility, wherein the access card trail is configured to report each time the access card of the identified suspect is presented to an access card reader 20 of the security system 10, along with the location of the corresponding access card reader in the facility, as indicated by block 58. In some cases, method 24 may include at least partially deactivating an access card of the identified suspect to prevent the suspect from accessing one or more areas of the facility, as indicated by block 60. In some cases, method 24 may include activating an audio warning of video camera 12 when the event of interest is classified as having a first event severity level among a plurality of event severity levels and / or when the event of interest is classified as having a second event severity level among a plurality of event severity levels, as indicated by block 62.

[0033] Figure 3A and Figure 3B 6 are flow charts collectively illustrating an exemplary method 64 for operating a security system of a facility, such as security system 10. Method 64 includes receiving video from a video camera of a security system of the facility, such as video camera 12, as indicated by block 66. Identifying an event of interest occurring in the video via video analysis, as indicated by block 68. Identifying a suspect associated with the event of interest in the video, as indicated by block 70. Identifying an image of the face of the suspect in the video, as indicated by block 72. Submitting the image of the face of the suspect to a facial recognition engine to identify the identity of the suspect, as indicated by block 74. Identifying an access card of the identified suspect, wherein the access card needs to be presented to an access card reader of the security system in order for the identified suspect to access an area of ​​the facility, as indicated by block 76. Activate an access card trail for the identified suspect's access card to track the suspect's subsequent movement within the facility, wherein the access card trail is configured to report each time the identified suspect's access card is presented to an access card reader of the security system along with the location of the corresponding access card reader within the facility, as indicated by box 78.

[0034] In some cases, the method 64 may include at least partially deactivating an access card of the identified suspect to prevent the suspect from accessing one or more areas of the facility, as indicated at block 80. In some cases, the method 64 may include classifying the event of interest into one of a plurality of event severity levels, as indicated at block 82. When the event of interest is classified as having a first event severity level (e.g., a lower severity level) of the plurality of event severity levels, an alert may be sent to an operator of the security system, as indicated at block 84.

[0035] continue Figure 3B , the method 64 may include sending an alert to a first responder when the event of interest is classified as having a second event severity level (e.g., a higher severity level) among the plurality of event severity levels, as indicated by block 86. In some cases, the method 64 may include generating one or more captions describing the event of interest, as indicated by block 88. In some cases, the one or more captions are generated by a context-aware video caption model that generates natural language captions for the event of interest. A first video clip of the event of interest may be extracted, as indicated by block 90. ​​A report including the first video clip of the event of interest and the one or more captions describing the event of interest may be compiled, as indicated by block 92. The report may be exported, as indicated by block 94.

[0036] Figure 4 is a flow chart showing an illustrative method 96 for operating a security system of a facility, such as security system 10. Method 96 includes receiving a video from a video camera of a security system of the facility, such as video camera 12, as indicated by block 98. Identifying an event of interest occurring in the video (e.g., using video analytics), as indicated by block 100. Generating one or more subtitles describing the event of interest, as indicated by block 102. In some cases, the one or more subtitles may include natural language subtitles describing the event of interest occurring in the video. In some cases, the one or more subtitles are generated by a context-aware video subtitle model that generates natural language subtitles for the event of interest. In some cases, the video may include a video track and a sound track. The one or more subtitles may include subtitles describing the event of interest in the video track and / or subtitles describing the event of interest in the sound track. Extracting a first video clip of the event of interest, as indicated by block 104. Compiling a report including the first video clip of the event of interest and the one or more subtitles describing the event of interest, as indicated by block 106. In some cases, extracting an audio clip of the event of interest and including it in the report. Exporting the report, as indicated by block 108.

[0037] Figure 51 is a flow chart illustrating an exemplary method 110 of processing video data 112. Video data 112 may include both video tracks and audio tracks, and each may be processed separately. Figure 5 , video data 112 is segmented into image data 114 and sound data 116. Sound data 116 may be processed using sound algorithms 118 to identify events of interest and to classify audio from possible suspects and / or possible victims. Image data 114 may be processed using video algorithms 120 to identify events of interest and to determine who is a suspect and who is a victim. Video algorithms 120 output faces of possible suspects and faces of possible victims to database 122. Database 122 outputs image data with timestamps to algorithm 124. Sound data is also provided as input to algorithm 124. Algorithm 124 processes images and sounds and outputs to NLG (natural language generator) engine 126. In some cases, everything between video data 112 and NLG engine 126 may be considered to represent video processing block 128. NLG engine 126 outputs report 130. Report 130 may include one or more video clips of events of interest and one or more natural language subtitles describing what is happening in the video clips. Because reports 130 can be automatically generated by the system, the operator can have an easily accessible repository of forensic evidence for various events of interest. In addition, when an event of interest is detected within the FOV of one of the many cameras, the system can alert the operator, thereby focusing the operator's attention on more important incidents. In some cases, for more serious events of interest, the system can automatically alert first responders to improve response time to the event of interest.

[0038] Figure 61 is a flow chart showing an exemplary method 132 for detecting an event of interest. Camera 134 may detect a possible situation (e.g., an event of interest) by running video analysis on a video stream captured by camera 134. Once a possible situation is detected, camera 134 may notify an adjacent peer camera 136 to start video recording. Camera 134 may also utilize AI (artificial intelligence) 138 to detect abuse or dangerous situations in its FOV. In some cases, AI 138 may send an alert to a first responder 140. Camera 134 may zoom in to obtain more information, such as identifying the abuse and / or the identity of the suspect and / or victim, as shown at 142. When so provided, the suspect's access card 144 may be marked as a tracking card so that the movement of the suspect with the access card 144 may be tracked and / or restricted. In some cases, the adjacent peer cameras 136 may form a merged video clip 146 that includes evidence recorded by each camera (including camera 134 and adjacent peer cameras 136). The merged video clip 146 may become part of output 148. In some cases, the output from the NLG engine 126 is recorded, as indicated by block 150 , and then included as part of the output 148 .

[0039] Figure 7 1 is a flow chart showing an illustrative method 152 for processing data once a violation is detected. A violation is detected and reported to a video management system (VMS) system, as indicated by box 154. AI is used to determine a suspect who has committed a violation, as indicated by box 156. The identity of the suspect may be provided to a first responder, as indicated by box 158. The identity of the suspect may be sent to an access control system, as indicated by box 160. Once the AI ​​identifies the suspect, the camera is instructed to zoom in to collect more data, as indicated by box 162. The video captured by the camera is merged, as indicated by box 164. The additional data captured by the camera after zooming in is also provided to the NLG engine, as indicated by box 166. The video from box 164 and the incident report generated by the NLG engine at box 166 may be combined into an investigation report, as indicated by box 168. In some cases, the investigation report may be provided to the first responder 170.

[0040] Figure 8 is an exemplary report 172 that may be output (such as in Figure 7168 of the investigation report cited at box 168 of ). At 174, camera 1 has identified a tall man in a blue shirt holding a knife via one or more video analysis algorithms. The context-aware video captioning engine generates natural language captions for the event of interest. The suspect is yelling at a woman in a white top and blue jeans. The suspect is identified as Andrew Steven, and the victim is identified as Jenny. One minute later, camera 1 detects Jenny's scream (via the video track and / or audio track), as indicated by 176. Subsequently, the suspect and the victim move from camera 1 toward camera 2 and are eventually detected by camera 2, as indicated by 180. The video clips from camera 1 are merged and exported from camera 1, as indicated by 178. Camera 2 detects that the suspect attempts to stab the victim, and the victim throws a water bottle at the suspect, as indicated by 182. The context-aware video captioning engine generates additional natural language captions for the event of interest. The video clips from camera 2 are exported from camera 2, as indicated by 184. The behavior has stopped, as indicated at 186. Perhaps the suspect dropped the knife, or the victim ran away because he threw a water bottle at the suspect. All recorded video clips and natural language captions are merged and exported, as indicated at 188. The report is complete, as indicated at box 190.

[0041] Although several exemplary embodiments of the present disclosure have been described in this manner, it will be readily understood by those skilled in the art that other embodiments may be made and used within the scope of the claims appended hereto. However, it should be understood that the present disclosure is in many respects merely illustrative. Details (especially details related to shape, size, arrangement of parts, and exclusion and order of steps) may be changed without exceeding the scope of the present disclosure. Of course, the scope of the present disclosure is defined in the language of the appended claims.

Claims

1. A method for operating a safety system of a facility, the method comprising: receiving video from a video camera of said security system of said facility; identifying events of interest occurring in the video via video analysis; classifying the event of interest into one of a plurality of event severity levels; sending an alert to an operator of the security system when the event of interest is classified as having a first event severity level of the plurality of event severity levels; as well as When the event of interest is classified as having a second event severity level among the plurality of event severity levels, an alert is sent to a first responder.

2. The method of claim 1, wherein identifying the event of interest occurring in the video comprises: The events of interest are identified using artificial intelligence, wherein the artificial intelligence includes a context-aware event model.

3. The method according to claim 1, comprising: generating one or more subtitles describing the event of interest; extracting a first video clip of the event of interest; compiling a report including the first video clip of the event of interest and the one or more captions describing the event of interest; as well as Export the report. 4 . The method of claim 3 , wherein the one or more subtitles include natural language subtitles describing the event of interest occurring in the video.

5. The method of claim 4, wherein the video comprises a video track and an audio track, wherein the one or more subtitles comprise: subtitles describing the event of interest in the video track; as well as Subtitles describing the event of interest in the soundtrack.

6. The method of claim 3, wherein generating one or more subtitles describing the event of interest comprises: The one or more captions are generated using artificial intelligence, wherein the artificial intelligence includes a context-aware video caption model that generates natural language captions for the event of interest.

7. The method according to claim 3, comprising: extracting a second video clip of the event of interest; as well as The report is compiled including the first video segment of the event of interest, the second video segment of the event of interest, and the one or more captions describing the event of interest.

8. The method of claim 7, wherein the first video clip is captured by a first video camera and the second video clip is captured by a second video camera, wherein the first video camera and the second video camera communicate via a peer-to-peer network.

9. The method according to claim 1, comprising: identifying a suspect in the video associated with the event of interest; identifying an image of the suspect's face in the video; submitting the image of the face of the suspect to a facial recognition engine to identify the identity of the suspect; identifying an access card of the identified suspect, wherein the access card needs to be presented to an access card reader of the security system in order for the identified suspect to access an area of ​​the facility; activating an access card trail for the access card of the identified suspect to track subsequent movement of the suspect within the facility, wherein the access card trail is configured to report each time the access card of the identified suspect is presented to an access card reader of the security system along with the location of the corresponding access card reader within the facility; and The access card of the identified suspect is at least partially deactivated to prevent the suspect from accessing one or more areas of the facility.

10. A method according to claim 1, comprising activating an audio warning via the video camera when the event of interest is classified as having the first event severity level among the multiple event severity levels and / or when the event of interest is classified as having the second event severity level among the multiple event severity levels.

Images