Video surveillance system with audio analysis adapted to a specific environment to assist in identifying unusual events in the specific environment

Abstract

The present invention provides a method and system for identifying abnormal sounds in a specific environment. A normal audio stream obtained in the absence of abnormal sounds can be used as a baseline for subsequently processing an incoming audio stream with a processor to determine whether the incoming audio stream from a microphone in the specific environment includes an abnormal audio event for the specific environment. When it is determined that the incoming audio stream includes an abnormal audio event for the specific environment, an electronic database can be accessed to determine the location of the abnormal audio event in the specific environment. A camera with a field of view including the location of the abnormal audio event in the specific environment can be identified, and a video stream from the identified camera can be retrieved and displayed.

Description

Technical Field

[0001] The present disclosure relates generally to video surveillance systems, and more particularly to video surveillance systems that use audio to help identify unusual events in a monitored environment. Background Art

[0002] Audio analysis is becoming more and more common in video surveillance venues due to its advantages of providing complementary information to facility monitoring. In many cases, in order to detect abnormal audio events, separate audio analysis modules are developed and trained to detect corresponding audio events. For example, one audio analysis module may be developed and trained to detect gunshots, while another audio analysis module may be developed and trained to detect screams. These trained audio analysis modules may be purchased from various vendors for use.

[0003] Each of these audio analysis modules is typically applied to an incoming audio stream independently of the other audio analysis modules to detect corresponding audio events. In use, there may be dozens, hundreds, or even thousands of audio analysis modules running to detect an expected range of audio event types of interest in the monitored environment. In addition, each environment may require a different set of audio analysis modules. For example, a child screaming in a school may not be a cause for great concern, but a child screaming in an airport may be a serious anomaly. In addition, background noise in a particular monitored environment may interfere with the reliability of these existing audio analysis modules because the audio analysis modules are typically developed and trained in the absence of similar background noise. Summary of the invention

[0004] The present disclosure relates generally to video surveillance systems, and more specifically to video surveillance systems that use audio to help identify abnormal events in a monitored environment. In one example, a method for identifying abnormal sounds in a specific environment includes: receiving a normal audio stream from a microphone located in the specific environment when no abnormal audio event is present; and using at least a portion of the normal audio stream as a baseline for subsequently processing the incoming audio stream to determine whether the incoming audio stream from the microphone in the specific environment includes an abnormal audio event specific to the environment. When it is determined that the incoming audio stream includes an abnormal audio event for the specific environment, the method can access an electronic database to determine the location of the abnormal audio event in the specific environment, identify a camera with a field of view that includes the location of the abnormal audio event in the specific environment, and retrieve a video stream from the identified camera and display it on a display.

[0005] Alternatively or in addition to any of the examples above, in another example, a normal audio stream may be received during the training mode.

[0006] Alternatively or in addition to any of the above examples, in another example, during training mode, the method may also include: dividing a normal audio stream into a plurality of normal audio clip segments; preparing a plurality of abnormal audio clip segments by superimposing known abnormal audio events onto one or more normal audio clip segments of the plurality of normal audio clip segments; and training an audio classification model using the plurality of normal audio clip segments and one or more of the plurality of abnormal audio clip segments to identify abnormal audio events in a specific environment.

[0007] Alternatively or in addition to any of the examples above, in another example, an audio classification model may be used to determine whether an incoming audio stream from a particular environment includes unusual audio events for the particular environment during an operating mode.

[0008] Alternatively or in addition to any of the examples above, in another example, the audio classification model can be a self-learning model.

[0009] Alternatively or in addition to any of the examples above, in another example, the self-learning model may use reinforcement learning and / or transfer learning.

[0010] Alternatively or in addition to any of the above examples, in another example, the method may also include: presenting one or more determined abnormal audio events to an operator; receiving a classification from the operator that, for a particular environment, the determined abnormal audio event is indeed an abnormal audio event or should be considered a normal audio event; and updating the audio classification model based on the classification received from the operator.

[0011] Alternatively or in addition to any of the examples above, in another example, determining the location of an unusual audio event in a particular environment can be based at least in part on physical locations of microphones in the particular environment stored in an electronic database.

[0012] Alternatively or in addition to any of the examples above, in another example, the microphone can be a directional microphone having a directional orientation, and wherein determining the location of an abnormal audio event in a particular environment can be based at least in part on the physical location of the microphone in the particular environment and the directional orientation of the microphone.

[0013] Alternatively or in addition to any of the examples above, in another example, a camera may include a housing that houses a microphone and provides an incoming audio stream and a video stream.

[0014] Alternatively or in addition to any of the examples above, in another example, the camera may be housed separately from the microphone, and the microphone may be processed separately from the camera.

[0015] Alternatively or in addition to any of the above examples, in another example, the method may further include sending an alert to an operator when it is determined that the incoming audio stream includes an abnormal audio event for a particular environment.

[0016] In another example, a method for identifying abnormal sounds in a specific environment may include entering a training mode. In the training mode, the method may include: capturing real-time audio from multiple microphones in a specific environment; dividing the real-time audio into multiple audio files; saving at least some of the multiple audio files as normal audio files containing normal audio signatures for a specific environment; superimposing the abnormal audio signature on at least some of the multiple normal audio files, and saving the resulting files as abnormal audio files containing abnormal audio signatures; and using the normal audio files and the abnormal audio files to train an audio classification model. The method may also include entering an operating mode. In the operating mode, the method may include: capturing real-time audio from each of the multiple microphones; dividing the real-time audio into multiple operational audio files; processing the operational audio files using an audio classification model via a processor to identify one or more abnormal audio signatures in a specific environment; determining the location of one of the abnormal audio signatures in the abnormal audio signatures in the specific environment; and retrieving a video stream from a camera having a field of view including the location and displaying it on a display.

[0017] Alternatively or in addition to any of the examples above, in another example, the method may further include receiving a user input confirming or rejecting the identification of the anomalous audio signature.

[0018] Alternatively or in addition to any of the examples above, in another example, the audio classification model can be updated based on user input.

[0019] Alternatively or in addition to any of the examples above, in another example, the audio classification model can be a self-learning model.

[0020] Alternatively or in addition to any of the examples above, in another example, the self-learning model may use reinforcement learning and / or transfer learning.

[0021] In another example, a system for identifying abnormal sounds in a specific environment may include a memory for storing an audio classification model, a display, and a controller operably coupled to the memory and the display. The controller may be configured to: receive a normal audio stream from a microphone located in the specific environment when there is no abnormal audio event; use at least a portion of the normal audio stream as a baseline for training the audio classification model to identify whether an incoming audio stream from the microphone includes an abnormal audio event for the specific environment; use the trained audio classification model to determine whether the incoming audio stream from the microphone includes an abnormal audio event for the specific environment; when determining that the incoming audio stream includes an abnormal audio event for the specific environment, determine the location of the abnormal audio event in the specific environment; identify a camera with a field of view that includes the location of the abnormal audio event in the specific environment; and retrieve a video stream from the identified camera and display it on the display.

[0022] Alternatively or in addition to any of the examples above, in another example, the audio classification model can be a self-learning model.

[0023] Alternatively or in addition to any of the above examples, in another example, the controller can be further configured to: present one or more determined abnormal audio events to an operator via a display; receive a classification from the operator that, for a particular environment, the determined abnormal audio event is indeed an abnormal audio event or should be considered a normal audio event; and update the audio classification model based on the classification received from the operator.

[0024] In another example, an audio camera for capturing audio and video of a particular environment may include a housing; a camera housed by the housing for providing a video stream; one or more directional microphones, each of which is configured to receive sound from a primary audio direction, wherein the one or more directional microphones are carried by the housing so that the primary audio direction of each of the one or more directional microphones is oriented in a different direction from the housing so that the approximate direction of sound events emanating from the particular environment can be determined; and a controller housed by the housing, the controller being operably coupled to the camera and each of the one or more directional microphones, and the controller being configured to provide audio and video output to a remote device.

[0025] Alternatively or in addition to any of the above examples, in another example, the camera can be a pan-tilt-zoom (PTZ) camera having a field of view. The controller can be configured to: use one or more directional microphones to determine an approximate direction of a sound event emanating from a particular environment, and control the field of view of the PTZ camera to face the determined approximate direction of the sound event so as to capture a video stream of the source of the sound event.

[0026] Alternatively or in addition to any of the examples above, in another example, the primary directions of one or more directional microphones may be oriented to have uniform angular spacing.

[0027] The above summary is provided to facilitate understanding of some features of the present disclosure, but is not intended to be a complete description. A comprehensive understanding of the present disclosure can be obtained by taking the entire specification, claims, drawings and abstract as a whole. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The present disclosure may be more fully understood upon consideration of the following detailed description of various embodiments in conjunction with the accompanying drawings, in which:

[0029] Figure 1 is a schematic diagram of an exemplary building or other structure including a video surveillance system, etc.;

[0030] Figure 2 is used for Figure 1 A block diagram of an exemplary sound analysis system for a video surveillance system;

[0031] Figure 3 is used in Figure 1 A flowchart of an illustrative method for performing video and audio analysis in a video surveillance system;

[0032] Figure 4 is a flow chart of an illustrative method for generating an audio model for classifying audio noise in an audio stream;

[0033] Figure 5 is a flowchart of an exemplary method for identifying abnormal audio events using an audio model, the audio model using Figure 4 generated by the method of

[0034] Figure 6 is an exemplary camera with an audio sensor;

[0035] Figure 7 is used in Figure 1 Another exemplary method of performing video and audio analysis in a video surveillance system of; and

[0036] Figure 8 is used in Figure 1 Another illustrative method for performing video and audio analysis in a video surveillance system.

[0037] Although the present disclosure is subject to various modifications and alternative forms, the details thereof have been shown by way of example in the accompanying drawings and will be described in detail. However, it should be understood that it is not intended to limit the aspects of the present disclosure to the specific embodiments described. On the contrary, it is intended to cover all modifications, equivalents and alternatives that fall within the spirit and scope of the present disclosure. DETAILED DESCRIPTION

[0038] The following detailed description should be read with reference to the accompanying drawings, in which similar elements in different drawings are numbered in the same manner. The detailed description and the accompanying drawings depict exemplary embodiments and are not intended to limit the scope of the present disclosure, and the drawings are not necessarily drawn to scale. The exemplary embodiments shown are intended to be exemplary only. Unless expressly stated to the contrary, some or all of the features of any exemplary embodiment may be incorporated into other exemplary embodiments.

[0039] The various systems and / or methods described herein may be implemented or performed using a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array signal (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0040] In some cases, the method or system may utilize a dedicated processor or controller. In other cases, the method or system may utilize a common or shared controller. Whether the system or method is described relative to a dedicated controller / processor or a common controller / processor, each method or system may utilize either or both of the dedicated controller / processor or the common controller / processor. For example, a single controller / processor may be used for a single method or system or any combination of methods or systems. In some cases, the system or method may be implemented in a distributed system, wherein parts of the system or method are distributed between various components of the distributed system. For example, some parts of the method may be executed locally, while other parts may be executed by a remote device such as a remote server. These are just examples.

[0041] Public areas (such as, but not limited to, commercial buildings, manufacturing sites, airports, transportation hubs, schools, malls, shopping centers, theaters, stadiums, etc.) may use surveillance cameras to provide security for the area. In some cases, it may be desirable to include audio analysis to supplement video surveillance for facility security or monitoring. For example, audio analysis can be used to identify abnormal or worrying noises, such as, but not limited to, fire alarms, robbery sounds, screaming, shouting, gunshots, or noises or crackling noises from abnormal equipment. Currently, in order to detect abnormal audio events, separate audio analysis modules are developed and trained to detect corresponding audio events. For example, one audio analysis module can be developed and trained to detect gunshots, while another audio analysis module can be developed and trained to detect screams. These trained audio analyses can be purchased from various suppliers for use.

[0042] Each of these audio analysis modules is typically applied to an incoming audio stream independently of the other audio analysis modules to detect corresponding audio events. In use, there may be dozens, hundreds, or even thousands of audio analysis modules running to detect an expected range of audio event types of interest in the monitored environment. In addition, each environment may require a different set of audio analysis modules. For example, a child screaming in a school may not be a cause for great concern, but a child screaming in an airport may be a serious anomaly. In addition, background noise in a particular monitored environment may interfere with the reliability of these existing audio analysis modules because the audio analysis modules are typically developed and trained in the absence of similar background noise.

[0043] Furthermore, the sound localization of an anomaly may be limited to the camera that detects the anomaly. Generally speaking, microphones attached to cameras today are omnidirectional. When so configured, a camera may be pointed in one direction while audio is captured in an orientation not visible in the camera, which inhibits accurate sound localization. The present disclosure overcomes these and other deficiencies of current video surveillance systems.

[0044] Steering Figure 1, which is a schematic diagram of an exemplary building or structure 10, which includes a building management system (BMS) 12 for controlling one or more client devices serving the building or structure 10. The exemplary BMS 12 includes a security system, an HVAC system, a lighting control system, and a fire control system. The security system includes a video surveillance system including multiple cameras and audio sensors (e.g., microphones) set throughout the environment. As described herein according to various exemplary embodiments, the BMS 12 can be used to control one or more client devices to control and / or monitor certain environmental conditions (e.g., temperature, ventilation, humidity, lighting, security, etc.). Such a BMS 12 can be implemented in places such as office buildings, factories, manufacturing sites, distribution sites, retail buildings, hospitals, fitness clubs, airports, transportation hubs, schools, shopping centers, movie theaters, restaurants, and even residences. Although the BMS 12 is described with respect to a building, it should be understood that the BMS 12 or portions thereof can also be used in outdoor environments.

[0045] Figure 1 The illustrated BMS 12 includes one or more heating, ventilation and air conditioning (HVAC) systems 20, one or more security systems 30, one or more lighting systems 40, one or more fire protection systems 50, and one or more access control systems 60. These are just a few examples of systems that may be included or controlled by the BMS 12. In some cases, the BMS 12 may include more or fewer systems, depending on the needs of the building. For example, some buildings may also include a refrigeration system or chiller. In another example, the BMS 12 may include only a video surveillance system.

[0046] In some cases, each system may include a client device configured to provide one or more control signals for monitoring and / or controlling one or more building control components and / or devices of the BMS 12. For example, in some cases, the HVAC system 20 may include an HVAC control device 22 for communicating with and controlling one or more HVAC devices 24a, 24b, and 24c (collectively 24) to serve the HVAC needs of the building or structure 10. Although the HVAC system 20 is shown as including three devices, it should be understood that the structure may include less than three or more than three devices 24, as desired. Some exemplary devices may include, but are not limited to, furnaces, heat pumps, electric heat pumps, geothermal heat pumps, electric heating units, air conditioning units, rooftop units, humidifiers, dehumidifiers, air exchangers, air cleaners, dampers, valves, blowers, fans, motors, air cleaners, ultraviolet (UV) lamps, and the like. The HVAC system 20 may also include a system of ductwork and vents (not explicitly shown). The HVAC system 20 may also include one or more sensors or devices 26 configured to measure parameters of the environment to be controlled. The HVAC system 20 may include more than one of each type of sensor or device to control the system as needed. It is contemplated that large buildings (such as, but not limited to, office buildings) may include multiple different sensors in each room or in certain types of rooms. The one or more sensors or devices 26 may include, but are not limited to, temperature sensors, humidity sensors, carbon dioxide sensors, pressure sensors, occupancy sensors, proximity sensors, etc. Each of the sensors / devices 26 may be operably connected to the control device 22 via a corresponding communication port (not explicitly shown). It is contemplated that the communication port may be wired and / or wireless. When the communication port is wireless, the communication port may include a wireless transceiver, and the control device 22 may include a compatible wireless transceiver. It is contemplated that the wireless transceiver may communicate using standard and / or proprietary communication protocols. Suitable standard wireless protocols may include, for example, cellular communications, ZigBee, Bluetooth, WiFi, IrDA, dedicated short-range communications (DSRC), EnOcean, or any other suitable wireless protocol as needed.

[0047] In some cases, the security system 30 may include a security control device 32 for communicating with and controlling one or more security units 34 to monitor the building or structure 10. The security system 30 may also include a plurality of sensors / devices 36a, 36b, 36c, 36d (collectively referred to as 36). The sensors / devices 36 may be configured to detect threats within and / or around the building 10. In some cases, some of the sensors / devices 36 may be configured to detect different threats. For example, some of the sensors / devices 36 may be limit switches located on the doors and windows of the building 10, which are activated by an intruder entering the building 10 through the doors and windows. For example, other suitable security sensors / devices 36 may include fire, smoke, water, carbon monoxide and / or natural gas detectors. Other suitable security system sensors / devices 36 may include motion sensors that detect the movement of intruders in the building 10, noise sensors or microphones that detect the sound of breaking glass or other sounds in the environment, security card access systems or electronic locks, etc. It is contemplated that the motion sensor may be a passive infrared (PIR) motion sensor, a microwave motion sensor, a millimeter wave indoor radar sensor, an ultrasonic motion sensor, a tomographic motion sensor, a camera with motion detection software, a vibration motion sensor, etc. In some cases, one or more of the sensors / devices 36 may include a camera with or without a sound sensor or microphone incorporated into its housing. In some cases, the sensors / devices 36 may include a horn or alarm, a damper actuator controller (e.g., which closes the damper during a fire event), a light controller for automatically turning lights on / off to simulate occupancy, and / or any other suitable device / sensor. These are just examples.

[0048] In some cases, the lighting system 40 may include a lighting control device 42 for communicating with and controlling one or more light groups 44 having lighting units L1-L10 to serve the building or structure 10. In some embodiments, one or more of the lighting units L1-L10 may be configured to provide visual illumination (e.g., in the visible spectrum), and one or more of the lighting units L1-L10 may be configured to provide ultraviolet (UV) light to provide irradiation, sometimes used to kill pathogens on building surfaces. One or more of the lighting units L1-L10 may include a multi-sensor bundle, which may include, but is not limited to, humidity sensors, temperature sensors, microphones, motion sensors, etc. The lighting system 40 may include emergency lights, sockets, lighting, exterior lights, curtains, and general load switches, some of which may be subject to "dimming" control, which changes the amount of power delivered to various building control devices.

[0049] In some cases, the fire protection system 50 may include fire control equipment 52 for communicating with and controlling one or more fire barriers 54 having fire protection units F1-F6 to monitor and service the building or structure 10. The fire protection system 50 may include smoke / heat sensors, sprinkler systems, warning lights, etc.

[0050] In some cases, the access control system 60 may include an access control device 62 for communicating with and controlling one or more access control units 64 to allow entry, exit, and / or around the building or structure 10. The access control system 60 may include a door, door lock, window, window lock, turnstile, parking gate, elevator, or other physical barrier, where access can be electronically controlled. In some embodiments, the access control system 60 may include one or more sensors 66 (e.g., RFID, etc.), which are configured to allow entry to a building or certain portions of the building 10.

[0051] In a simplified example, the BMS 12 may be used to control a single HVAC system 20, a single security system 30, a single lighting system 40, a single fire protection system 50, and / or a single access control system 60. In other embodiments, the BMS 12 may be used to communicate with and monitor / control multiple discrete building control devices 22, 32, 42, 52, and 62 of multiple systems 20, 30, 40, 50, 60. The devices, units, and controllers of the systems 20, 30, 40, 50, 60 may be located in dedicated spaces (e.g., offices, studios, etc.) of the building 10 or in different areas and rooms outside thereof, such as common space areas (lobbies, lounges, etc.). In some cases, the systems 20, 30, 40, 50, 60 may be powered by line voltage and may be powered by the same or different circuits. It is contemplated that the BMS 12 may be used to control other suitable building control components that may be used to serve the building or structure 10.

[0052] According to various embodiments, the BMS 12 may include a host device 70 that may be configured to communicate with the discrete systems 20, 30, 40, 50, 60 of the BMS 12. In some cases, the host device 70 may be configured with an application that assigns the devices of the discrete systems to specific device (entity) categories (e.g., public space devices, dedicated space devices, outdoor lighting, integrated controllers, etc.). In some cases, there may be multiple hosts. For example, in some examples, the host device 70 may be one or more of the control devices 22, 32, 42, 52, 62. In some cases, the host device 70 may be a hub located outside the building 10 at an external or remote server (also referred to as a "cloud").

[0053] In some cases, the building control devices 22, 32, 42, 52, 62 may be configured to transmit command signals to their respective building control components in order to activate or deactivate the building control components in a desired manner. In some cases, the building control devices 22, 32, 42, 52, 62 may be configured to receive the classification of the building control components and, taking into account the classification of the building control components, may transmit corresponding command signals to their respective building control components.

[0054] In some cases, the building control devices 22, 32, 62 may be configured to receive signals from one or more sensors 26, 36, 66 located throughout the building or structure 10. In some cases, the building control devices 42 and 52 may be configured to receive signals from one or more sensors operably and / or communicatively coupled to the lighting units L1-L10 and fire units F1-F6 located throughout the building or structure 10, respectively. In some cases, one or more sensors may be integrated with and form a part of one or more of their corresponding building control devices 22, 32, 42, 52, 62. In other cases, one or more sensors may be provided as separate components from the corresponding building control devices. In other cases, some sensors may be separate components of their corresponding building control devices, while other sensors may be integrated with their corresponding building control devices. These are just some examples. The building control devices 22, 32, 42, 52, 62 and the host device 70 may be configured to use signals received from one or more sensors to operate or coordinate the operation of the various BMS systems 20, 30, 40, 50, 60 located throughout the building or structure 10. As will be described in greater detail herein, the building control devices 22, 32, 42, 52, 62 and the host device 70 may be configured to use signals received from one or more sensors to detect and locate abnormal noises.

[0055] One or more sensors 26, 36, 66, L1-L10, and F1-F6 may be any of a temperature sensor, a humidity sensor, an occupancy sensor, a pressure sensor, a flow sensor, a light sensor, a sound sensor (e.g., a microphone), a camera, a current sensor, a smoke sensor, and / or any other suitable sensor. In one example, at least one of sensors 26, 36, 66 or other sensors may be an occupancy sensor. Building control devices 22, 32, 42, 62, and / or host device 70 may receive a signal from an occupancy sensor indicating occupancy in a room or area of ​​building or structure 10. In response, building control devices 22, 32, 42, and / or 62 may send a command to activate one or more building control components located in or serving the room or area where occupancy is sensed.

[0056] Likewise, in some cases, at least one of the sensors 26 may be a temperature sensor configured to send a signal indicative of a current temperature in a room or zone of the building or structure 10. The building control device 22 may receive the signal indicative of the current temperature from the temperature sensor 26. In response, the building control device 22 may send a command to the HVAC equipment 24 to activate and / or deactivate the HVAC equipment 24 that is located in or servicing the room or zone to adjust the temperature according to a desired temperature set point.

[0057] In yet another example, one or more of the sensors may be current sensors. The current sensors may be coupled to one or more building control components and / or circuitry that provides power to one or more building control components. The current sensors may be configured to send a signal to a corresponding building control device that indicates an increase or decrease in current associated with operation of the building control component. This signal may be used to provide confirmation that a command transmitted by the building control device has been successfully received and acted upon by the building control component. These are just a few examples of configurations for the BMS 12 and communications that may be performed between sensors and control devices.

[0058] In some cases, data received from the BMS 12 may be analyzed and used to dynamically (e.g., automatically) trigger or provide service requests, work orders, changes in operating parameters (e.g., set points, schedules, etc.) to various devices 24, 34, 64, L1-L10, F1-F6, and / or sensors 26, 36, 66 in the BMS 12. In some cases, data received from the BMS 12 may be analyzed and used to dynamically (e.g., automatically) trigger or provide information about the health status of occupants of a building or area.

[0059] In still other cases, the data received from the BMS 12 may be analyzed and used to dynamically (e.g., automatically) trigger or provide information about noise levels or noise-generating events in a building or area. It is contemplated that data may be received from control devices 22, 32, 42, 62, devices 24, 34, 64, L1-L10, F1-F6, and / or sensors 26, 36, 66 as needed. In some cases, the data received from the BMS 12 may be combined with video data from an image capture device of a security system. It is contemplated that the video data may be obtained from certain sensors 26, 36, 66 that are image capture devices associated with separate systems 20, 30, 60 of the BMS 12, or may be provided as separate image capture devices as needed, such as video (or still image) capture cameras 80a, 80b (collectively 80). An "image" may include a static single-frame image or an image stream (e.g., video) captured at multiple frames per second. While the exemplary building 10 is shown as including two cameras 80, it is contemplated that the building may include fewer than two or more than two cameras as desired. It is also contemplated that the cameras (either discrete cameras 80 or cameras associated with discrete systems 20, 30, 60) may be considered "smart" edge cameras (which may be considered Internet of Things (IoT) devices) capable of independently processing image streams, or "dumb" cameras used as sensors to collect video information analyzed by an independent video analytics engine. Some exemplary cameras may include visible light security cameras, but may also include cameras mounted on drones, thermal imagers (e.g., IR) cameras, and / or any other suitable cameras.

[0060] It is contemplated that data from the BMS 12 and / or sensors 26, 36, 66, 80 may be systematically analyzed and compared to baseline data from the BMS 12 to monitor activity from individuals within different rooms / spaces within a building or complex of buildings by identifying their unique acoustic signatures. For example, real-time audio data may be compared to an audio model representing a normal sound profile and an abnormal sound profile for a particular environment. Detection of abnormal sounds in an environment may trigger an alert, such as, but not limited to, a security alert, a maintenance alert, etc. In some cases, the location of the source of the abnormal sound may be automatically identified, and the video feed of a camera capturing the identified location may be automatically displayed.

[0061] Figure 21 is a schematic block diagram of an exemplary system 100 for identifying abnormal sounds in a specific environment. The system 100 may form a part of any of the above-mentioned BMS systems 20, 30, 40, 50, 60 or be used in combination with any of the above-mentioned BMS systems 20, 30, 40, 50, 60. For example, the system 100 may communicate with any of the BMS systems 20, 30, 40, 50, 60 so that normal and / or abnormal sounds are associated or correspond to the operating cycles of the BMS systems 20, 30, 40, 50, 60 and / or normal behavior within a specific environment. In other examples, the system 100 may be a stand-alone system. It is also conceivable that the system 100 may be used in areas outside of traditional buildings, such as, but not limited to, public transportation or other areas where people may gather. In some cases, the system 100 may control one or more of the following items as needed: an HVAC system, a security system, a lighting system, a fire protection system, a building access control system, and / or any other suitable building control system.

[0062] In some cases, the system 100 includes a controller 102 and one or more edge devices 104. The edge device 104 may include, but is not limited to, a microphone (or other sound sensor) 106, a static camera or video camera 108, a building access control system reader or device 110, an HVAC sensor 112, a motion sensor 114, and / or any device or sensor described herein. In some cases, the microphone (or other sound sensor) 106 may be provided as a stand-alone network sensor. In some cases, the microphone 106 may be a directional microphone that hears sounds from a specific direction, or may be an omnidirectional microphone that hears sounds from all directions. In some cases, one or more microphones (or other sound sensors) 106 may be part of or incorporated into one or more static cameras or video cameras 108. The controller 102 may be configured to receive data from the edge device 104, analyze the data, and make decisions based on the data, as will be described in more detail herein. For example, controller 102 may include control circuitry and logic components configured to operate, control, command, etc., and / or issue alerts or notifications to various components (not expressly shown) of system 100 .

[0063] The controller 102 may communicate with any number of edge devices 104 as desired, such as, but not limited to, one, two, three, four, ten, one hundred, or more edge devices. In some cases, there may be more than one controller 102, each communicating with multiple edge devices. It is contemplated that the number of edge devices 104 may depend on the size and / or functionality of the system 100. The edge devices 104 may be selected and configured to monitor different aspects or locations of a building and / or area of ​​the system 100. For example, some of the edge devices 104 may be located inside a building. In some cases, some of the edge devices 104 may be located outside a building. Some of the edge devices 104 may be located in open areas, such as parks or public transportation stops. These are just some examples.

[0064] The controller 102 may be configured to communicate with the edge device 104 via a first network 116 including a local area network (LAN) or a wide area network (WAN), or a connection to an external computer may be made (e.g., via the Internet using an Internet service provider). Such communication may occur via a first communication port 122 at the controller 102 and a communication interface (not explicitly shown) at the edge device 104. The first communication port 122 of the controller 102 and / or the communication interface of the edge device 104 may be a wireless communication port including a wireless transceiver for wirelessly sending and / or receiving signals via the wireless network 116. However, this is not required. In some cases, the first network 116 may be a wired network or a combination of a wired network and a wireless network.

[0065] The controller 102 may include a second communication port 124, which may be a wireless communication port including a wireless transceiver for sending and / or receiving signals through a second wireless network 118. However, this is not required. In some cases, the second network 118 may be a wired network or a combination of a wired network and a wireless network. In some embodiments, the second communication port 124 may communicate with a wired or wireless router or gateway for connecting to the second network 118, but this is not required. When so configured, the router or gateway may be integral with the controller 102 (e.g., within the controller), or may be provided as a separate device. The second network 118 may be a wide area network or a global network (WAN), including, for example, the Internet. The controller 102 may communicate with an external web service hosted by one or more external web servers 120 (e.g., cloud) through the second network 118.

[0066] The controller 102 may include a processor 126 (e.g., a microprocessor, a microcontroller, etc.) and a memory 130. In some cases, the controller 102 may include a user interface 132, which includes a display and a device for receiving user input (e.g., a touch screen, a button, a keyboard, etc.). In some cases, the user interface 132 may be integral with the controller 102. Alternatively or in addition, the controller 102 may be operably coupled to a remotely located user interface 136, which includes a display and a device for receiving user input. For example, the remotely located user interface 136 may be a display in a security monitoring station, a portable device (such as but not limited to a smart phone, a tablet computer, a laptop computer, etc.), or other such devices. It is contemplated that the remote user interface 136 may communicate with the controller 102 via the first network 116 and / or the second network 118 as needed.

[0067] The memory 130 may communicate with the processor 126. The memory 130 may be used to store any desired information, such as, but not limited to, control algorithms, configuration protocols, set points, scheduling times, diagnostic limits (such as, for example, differential pressure limits, ΔT limits), security system arming modes, audio classification models, etc. In some embodiments, the memory 130 may include a specific control program or module configured to analyze data obtained from the edge device 104 for a specific condition or situation. For example, the memory 130 may include, but is not limited to, a sound classification module 134 and a model generation module 138, which contain instructions and / or data that can be executed by the processor 126. The sound classification module 134 may be configured to detect sounds and / or activities that are abnormal for a specific environment, as will be described in more detail herein. The model generation module 138 may be configured to generate a model of a desired sound profile for a specific area and a model of an undesirable sound profile for a specific area, as will be described in more detail herein. The memory 130 may include one or more of the sound classification module 134 and the model generation module 138. In some cases, the memory 130 may include additional sound classification modules or model generation modules. Memory 130 may be any suitable type of storage device including, but not limited to, RAM, ROM, EPROM, flash memory, a hard drive, etc. In some cases, processor 126 may store information within memory 130 and may subsequently retrieve the stored information from memory 130 .

[0068] In some embodiments, the controller 102 may include an input / output block (I / O block) 128 having a plurality of wiring terminals for receiving one or more signals from the edge device 104 and / or system components and / or for providing one or more control signals to the edge device 104 and / or system components. For example, the I / O block 128 may communicate with one or more components of the system 100 (including but not limited to the edge device 104). The controller 102 may have any number of wiring terminals for accepting connections from one or more components of the modular control system 100. However, how many wiring terminals are utilized and which terminals are wired depends on the specific configuration of the system 100. Different systems 100 with different components and / or component types may have different wiring configurations. In some cases, the I / O block 128 may be configured to receive wireless signals from the edge device 104 and / or one or more components or sensors (not explicitly shown). Alternatively or in addition, the I / O block 128 may communicate with another controller. It is also contemplated that the I / O block 128 may communicate with another controller that controls a separate building control system, such as, but not limited to, a security system base module, a HVAC controller, or the like.

[0069] In some cases, a power conversion block (not explicitly shown) may be connected to one or more wires of the I / O block 128 and may be configured to drain or steal energy from one or more wires of the I / O block 128. The power drained by one or more wires of the I / O block may be stored in an energy storage device (not explicitly shown), which may be used to at least partially power the controller 102. In some cases, the energy storage device may be a capacitor or a rechargeable battery. In addition, the controller 102 may include a backup energy source, such as a battery, which may be used to supplement the power supplied to the controller 102 when the amount of available power stored by the energy storage device is less than the optimal value or insufficient to power certain applications. Certain applications or functions performed by the basic module may require more energy than other applications or functions. If the energy stored in the energy storage device is insufficient, in some cases, the processor 126 may prohibit certain applications and / or functions.

[0070] The controller 102 may also include one or more sensors, such as, but not limited to, a temperature sensor, a humidity sensor, an occupancy sensor, a proximity sensor, etc. In some cases, the controller 102 may include an internal temperature sensor, but this is not required.

[0071] When so configured, the user interface 132 can be any suitable user interface 132 that allows the controller 102 to display and / or request information and accept one or more user interactions with the controller 102. For example, the user interface 132 can allow the user to enter data locally (such as control set points, start time, end time, schedule time, diagnostic limits, responses to warnings), associate sensors to alarm modes, etc. In one example, the user interface 132 can be a physical user interface accessible at the controller 102, and can include a display and / or different keypads. The display can be any suitable display. In some cases, the display can include or can be a liquid crystal display (LCD), and in some cases can include or can be an electronic ink display, a fixed segment display, or a dot matrix LCD display. In other cases, the user interface can be a touch screen LCD panel used as both a display and a keypad. The touch screen LCD panel can be suitable for requesting the values ​​of multiple operating parameters and / or receiving such values, but this is not required. In other cases, the user interface 132 can be a dynamic graphical user interface.

[0072] In some cases, the user interface 132 need not be physically accessed by the user at the controller 102. Instead, the user interface may be a virtual user interface 132 accessible via the first network 116 and / or the second network 118 using a mobile wireless device (such as a smart phone, tablet, e-reader, laptop, personal computer, key fob, etc.). In some cases, the virtual user interface 132 may be provided by one or more apps executed by a user's remote device for the purpose of remotely interacting with the controller 102. Through the virtual user interface 132 provided by the app on the user's remote device, the user may change control set points, start times, end times, schedule times, diagnostic limits, responses to alerts, update their user profile, view energy usage data, arm or disarm security systems, configure alarm systems, and the like.

[0073] In some cases, changes made to the controller 102 through the user interface 132 provided by the app on the user's remote device may first be transmitted to the external web server 120. The external web server 120 may receive and accept user input typed through the virtual user interface 132 provided by the app on the user's remote device and associate the user input with the user account on the external web service. If the user input includes any changes to the existing control algorithm, including any temperature set point changes, humidity set point changes, schedule changes, start and end time changes, window anti-frost setting changes, operating mode changes, and / or changes to the user's profile, the external web server 120 may update the control algorithm, where applicable, and transmit at least a portion of the updated control algorithm to the controller 102 via the second network 118, where at least a portion of the updated control algorithm is received via the second communication port 124 and may be stored in the memory 130 for execution by the processor 126. In some cases, the user may observe the effects of his input at the controller 102.

[0074] The virtual user interface 132 may include one or more web pages transmitted by an external web server (e.g., web server 120) via a second network 118 (e.g., WAN or the Internet), rather than a dedicated app. The one or more web pages forming the virtual user interface 132 may be hosted by an external web service and associated with a user account having one or more user profiles. The external web server 120 may receive and accept user input typed via the virtual user interface 132, and associate the user input with a user account on the external web service. If the user input includes changes to an existing control algorithm, including any control set point changes, scheduling changes, start and end time changes, window anti-frost setting changes, operating mode changes, and / or changes to a user's profile, the external web server 120 may update the control algorithm, where applicable, and transmit at least a portion of the updated control algorithm to the controller 102 via the second network 118, where at least a portion of the updated control algorithm is received via the second communication port 124 and may be stored in the memory 130 for execution by the processor 126. In some cases, the user may observe the effect of his input at the controller 102.

[0075] In some cases, the user may use a user interface 132 provided at the controller 102 and / or a virtual user interface as described herein. These two types of user interfaces are not mutually exclusive. In some cases, the virtual user interface 132 may provide more advanced capabilities to the user. It is also contemplated that the same virtual user interface 132 may be used for multiple BMS components.

[0076] As described above, the controller 102 may alternatively or additionally communicate with a remote user interface 136 or a display device via the first network 116 and / or the second network 118. The remote user interface 136 may be located, for example, in a control room, a main office, a monitoring station, etc. Alternatively or in addition, the remote user interface may be a portable device carried by a user (e.g., a smart phone, a tablet, a laptop, a watch, etc.). The remote user interface 136 may be a physical device as described above or a virtual user interface (e.g., accessible via the Internet). In some cases, the remote user interface may include a display and / or different keypads. The display may be any suitable display. It is contemplated that in addition to communicating with the controller 102, the remote user interface 136 may also communicate with other building control devices or systems including the edge device 104.

[0077] The system 100 may also include a video management system (VMS) and / or a network video recorder (NVR) 140. The VMS / NVR 140 may communicate with the controller 102 and / or the edge device 104 via the first network 116 and / or the second network 118. The VMS / NVR 140 may be configured to record images and / or video from still cameras or video cameras 108. In some cases, the VMS / NVR 140 may receive information from the controller 102 and control the field of view of one or more cameras 108 in response to the received information, although this is not required.

[0078] It is contemplated that identifying and / or locating abnormal noises in a particular environment can provide information to a supervisory user and / or portion of the BMS that can be used to more appropriately respond to irregularities in a particular environment. It is contemplated that the system 100 can detect abnormal sounds in a particular environment and locate the anomaly by identifying the location and / or orientation of the anomaly. In some cases, the system 100 can further identify the abnormal sound. Figure 3 A schematic flow chart 200 is shown that provides an overview of an exemplary method and system for performing adaptive video and audio analysis. In general, the system can detect abnormal audio signatures, locate the anomaly, and send this information to the VMS / NVR 140 and / or other edge devices 104 to change and / or select the field of view of one or more cameras 108 to locate the area. Additionally, the detection of abnormal audio can be a triggering event that sends a message to required personnel for appropriate action.

[0079] Initially, the system 100 receives a camera adaptive audio signature and model 202. The audio signature and model 202 may be stored in the memory 130 of the controller 102, in the sound classification module 134 (or generally, the audio analysis module). Figure 4, which shows a schematic flow chart of an illustrative method 300 for generating camera adaptive signatures and models 202. These can be generated on site so that the background acoustics of a particular environment can be taken into account. As described above, cameras 108 and sound sensors 106 are deployed in different environments such as roads, facilities, commercial institutions, etc. Each of these environments has different conventional or normal noises associated with them. For example, cameras on roads collect traffic noise, and cameras in facilities and commercial institutions will receive machine noise, people's conversations, etc. In some cases, the system 100 is trained using deep neural artificial intelligence (AI) training to identify abnormal sounds or noises for a particular environment. It is contemplated that the collection and analysis of the audio stream used to generate the model can be performed by the same controller (e.g., controller 102) that is used to operate the analysis or can be a separate controller or computer as needed. The training cycle can be performed continuously or at predetermined intervals over a period of time, such as but not limited to hours, days, weeks, etc. When so configured, this can allow the system 100 to train abnormal sound detection under various conditions. For example, people talking in an office building during the day may be considered normal audio, whereas people talking in an office building at midnight may be considered abnormal. This is just one example.

[0080] First, in one example, the controller 102 is placed into a training mode, and when there are no abnormal audio events, audio is received from one or more microphones 106 located in a specific environment during normal use of the specific environment 302. It should be understood that a specific environment may include one or more microphones 106. Audio can be received and processed from each microphone 106 present in a specific area. It is also conceivable that a building or space may include more than one environment that is trained and analyzed separately, because the uses of such spaces may be different, and the acoustics of such spaces may be different. In order to increase the accuracy of the model, at least some of the training can be completed while the specific environment is used for its intended purpose. For example, in an airport, at least some of the training can be completed when the airport is open and serving passengers. One or more microphones can be independent network sound sensors, each of which has a unique IP address, and one or more microphones can be incorporated into a security camera or a combination thereof. The IP address and / or camera ID of the network sound sensor can be used to identify which microphone to obtain audio from (during training and during normal operation), and to identify the physical location of the sound sensor within the specific environment. The physical location and / or directional orientation of each microphone 106 and / or camera 108 may be stored in an electronic database within the memory 130 of the controller 102, in an electronic database on the external server 120, and / or in an electronic database within the VMS / NVR 140. By supplying the electronic database with the identification of a particular microphone 106 and / or camera 108, the electronic database may return the corresponding physical location and / or directional orientation of the particular microphone 106 and / or camera 108.

[0081] It is contemplated that the audio stream 302 may include audio from a single microphone 106, or may include multiple audio streams collected simultaneously from multiple microphones 106. When audio input is received simultaneously from more than one sound sensor, the audio streams may be analyzed separately as microphones in different physical locations may record different audio profiles during normal use of the environment. The normal audio stream may be used as a baseline when the incoming audio streams are subsequently processed by the controller 102 or processor 126 to determine whether the incoming audio streams received from one or more microphones in a particular environment include abnormal audio events for the particular environment.

[0082] In some cases, when the controller 102 receives the audio stream, the controller 102 can divide or split the normal audio stream into N second audio files 304 to form a plurality of normal audio clips or files. For example, the input audio stream can be divided into one second, two seconds, three seconds, four seconds or longer audio clips as needed. Each second audio file in these N second audio files can be saved as a normal sound mode 306. It is conceivable that if an abnormal sound occurs during a training cycle or training mode, the supervising user can remove the corresponding audio file from the N second audio files saved. The same N second audio clips can then be used to prepare a plurality of abnormal audio clips. For example, a known abnormal sound or noise previously captured can be superimposed 308 on the N second normal audio clips to create an abnormal audio clip with a normal audio clip as background noise. Known abnormal sounds or noises may include but are not limited to screaming, loud voices, gunshots, glass breaking, machine noises, etc. The prepared or changed audio clips are then saved as abnormal audio clips 310.

[0083] One or more normal audio clips of a plurality of normal audio clips and one or more abnormal audio clips of a plurality of abnormal audio clips may be used to train an audio classification model to identify abnormal audio events in a particular environment. For example, audio features (such as, but not limited to, Mel-frequency cepstral coefficients (MFCC), spectrographs, zero crossings, signal energy, energy entropy, spectral centroid, spread spectrum, spectral entropy, spectral roll-off, etc.) may be extracted 312 from normal audio and abnormal audio. These audio features may then be used to create one or more audio classification models for classifying future audio files into normal or abnormal modes 314. The audio classification model may be used to determine whether an incoming audio stream from a particular environment during an operating mode includes abnormal audio events for a particular environment. For example, real-time audio received during an operating mode of the system 100 may then be compared with a stored audio classification model to determine whether the real-time audio is classified as normal or abnormal. The audio classification model may include identification features, such as, but not limited to, the identity of the microphone from which the normal audio stream was obtained, the physical location of the microphone, a time of day, a day of the week, etc. This may allow the system 100 to compare the incoming audio stream with training audio obtained under similar operating conditions.

[0084] Back to Figure 3, once the audio classification model 202 is trained and stored in the memory 130 of the controller 102, the operating mode can be initiated. In the operating mode, real-time audio and video data 206 is received from one or more cameras 108 and / or one or more microphones 106. The input data 206 can be divided into an audio data stream 208 and a video data stream 210. In some cases, the camera 108 can accommodate the microphone 106 and provide both the incoming audio stream 208 and the incoming video stream 210. In other cases, the camera 108 can be housed separately from the microphone 106, wherein the microphone is processed separately from the camera. The real-time audio data stream 208 can be analyzed using an audio analysis 204 (e.g., a sound classification module 134) to determine whether the audio data stream 208 includes abnormal sounds or noise.

[0085] Now refer to Figure 5 , which shows a schematic flow chart of an illustrative method 250 for performing audio analysis 204 (e.g., analyzing incoming or real-time audio data for identifying abnormal sounds in a particular environment). As described above, the analysis of the audio data stream 208 can be performed by the controller 102 or the processor 126, and in some cases, for example, executing instructions stored in a dedicated module, such as but not limited to the sound classification module 134. First, the incoming audio stream is divided into N second audio files 252 to form a plurality of audio clip segments. For example, the audio stream 208 can be divided into one second, two second, three second, four second segments or longer audio clip segments as needed. In some cases, the incoming audio stream can be divided into segments having the same length as the audio clip segments during the training phase, although this is not required. Next audio features (such as but not limited to Mel frequency cepstral coefficients (MFCC), spectrograph, zero crossing, signal energy, energy entropy, spectral centroid, spread spectrum, spectral entropy, spectral roll-off, etc.) can be extracted 254 from the incoming audio segment.

[0086] The audio classification model generated during the training phase may be accessed from the model generation module 138 and applied to the audio features of the plurality of incoming audio clips 256. The controller 102 may identify similarities between the incoming separated audio stream features and the audio classification model. The controller 102 may also use the comparison to determine whether the audio stream includes normal sound or noise 258. For example, if the incoming separated audio stream features are similar to the audio features of the normal sound audio model, the controller 102 may determine that the sound is normal or expected and take no further action 260. If the incoming separated audio stream features are similar to the audio features of the abnormal sound audio model, the controller 102 may determine that the sound is abnormal or undesirable and take no further action 260. Figure 5In the example shown, a counter is started. After the counter reaches a predetermined number of unusual sound events, the controller 102 / processor 126 may determine that the incoming audio stream includes unusual audio events for the particular environment.

[0087] When an abnormal audio event is detected, the controller 102 / processor 126 can access an electronic database to determine the location of the microphone 106 that recorded the abnormal sound event, which is associated with the location of the abnormal sound event. In some cases, in addition to the physical location of the microphone 106, the controller 102 / processor 126 can access an electronic database to determine the directional orientation of the microphone 106 (when the microphone 106 is a directional microphone). Therefore, the location of the abnormal sound event can be determined at least in part based on the physical location of the microphone 106 and the directional orientation of the microphone 106. As will be described in more detail herein, the controller 102 / processor 126 (and / or VMS / NVR 140) can also use an electronic database to identify a camera 108 that has a field of view that includes the location of the abnormal audio event in a particular environment. The video stream from the identified camera can be retrieved and displayed on a display (such as, but not limited to, the remote user interface 136). In some cases, where the camera 108 is a pan / tilt / zoom (PTZ) camera, the controller 102 / processor 126 may send a PTZ command to a particular camera to direct the field of view of the particular camera toward the location of the detected unusual audio event.

[0088] It is also contemplated that after a predetermined number of abnormal sound events, an alert 262 may be generated and transmitted to an operator. It is contemplated that an alert may be sent to a remote or mobile device or VMS / NVR 140 of a supervisor or other user. The notification may be a natural language message that provides details about the abnormal sound, the original location, and / or the recommended action. In some cases, the alert may trigger additional actions taken by the BMS 12. For example, a workplace disturbance or security threat may result in the automatic locking of one or more doors of a building. In another example, an equipment failure alert may result in the shutdown of a piece of equipment. Here are just some examples.

[0089] In some cases, an incoming audio clip can be presented to an operator or user, and the operator or user can manually mark or classify the clip as normal or abnormal 264. These marked clips can be stored in the VMS / NVR 140 for further optimizing the model through self-learning techniques. For example, the audio classification model can be a self-learning model. Some self-learning techniques can include field learning mechanisms, such as reinforcement learning or transfer learning. The audio classification model can be updated based on the classification received from the operator. In some cases, only abnormal audio events (or their subsets) can be presented to the operator. In other cases, only normal audio events (or their subsets) can be presented to the operator. In some cases, abnormal and normal audio events (or their subsets) can be presented to the operator. In one example, an incoming audio clip that has been determined or classified as an abnormal audio event can be presented to the operator. Then, the operator can enter a classification, which is for a specific environment, the determined abnormal audio event is indeed an abnormal audio event or should be considered as a normal audio event. The audio classification model can then be updated based on the classification received from the operator. In some cases, the audio classification model can classify audio into more categories than just normal and abnormal. For example, the audio classification model can be trained to classify abnormal audio events into categories that indicate severity or threat level, such as low, medium, and high. This can help operators prioritize detected abnormal audio events.

[0090] Back to Figure 3 In addition to performing audio analysis 204, the controller 102 / processor 126 can perform audio localization 212 in parallel with the audio analysis 204. For example, as described above, the controller 102 can use the IP address of the network sound sensor or the ID number (or IP address) of the camera to determine the physical location of the sensor that has received the analyzed audio. It is conceivable that the physical location of the sensor and its identifier can be stored in an electronic database in the memory 130 of the controller 102. When the microphone 106 is a directional microphone, the directional orientation of the microphone 106 can also be stored in the electronic database. It is also conceivable that the controller 102 can be configured to determine how far away the audio originates from the sound sensor. This can be performed by analyzing the volume of the sound and determining whether other nearby sound sensors have similar audio signatures. Based on triangulation, the approximate location of the source of the sound can be determined. The controller 102 can send the localization information 214 to the VMS / NVR140 or other edge device 104 for storage (for record keeping) for improving the sound classification model, and / or to facilitate further operations when an abnormal sound event is detected.

[0091] Additionally or alternatively, when the audio sensor 106 is a networked sensor, peer-to-peer communication can be used to perform audio localization 212 for notifying the audio anomaly from the corresponding audio sensor or microphone 106 to the nearest or configured camera 108 by sending a signal. The audio stream identifier or IP address of the networked audio sensor can identify the microphone 106 from which the abnormal audio signature (and therefore the approximate location) is detected. Peer-to-peer communication between the edge device 104 or camera 108 and the networked audio sensor / microphone 106 can convey anomaly alerts. As described herein, orientation and audible field of view can be considered to locate audio anomalies.

[0092] As reference Figure 5 As described, when an abnormal sound event is detected, an alarm 262 can be generated. The alarm can be transmitted directly to the responsible person or can be sent to the VMS / NVR 140 or other edge device 104. The alarm 262 may include an indication of the abnormal sound event and / or priority. For example, the alarm 262 may include a priority classification indicating the degree of urgency (e.g., high, medium, low). In some cases, the VMS / NVR 140 or other edge device 104 can use the alarm 262 in combination with the audio positioning data 212 to determine which camera is closest to the source of the abnormal sound event or which camera has the best view of the source location. For example, in some cases, the alarm can be sent directly to the identified camera 108. Then, the identified camera 108 can transmit its video feed or adjust its field of view. In the event that the camera is a pan-tilt-zoom (PTZ) camera, the position of the camera can be adjusted to focus the camera's field of view on the positioning position 216. In some cases, focusing the camera on the positioning position can include selecting a camera with the best field of view without moving the camera's position. The live video feed from the camera pointed at the positioning location can then be transmitted to the responsible person along with other relevant information. The person can then use the video feed to verify the abnormal sound event (if possible) and take further action. In some cases, the person can generate a maintenance request. In other cases, the person can call an emergency responder. In some other cases, the person can travel to the positioning location to study the abnormal sound event. These are just some examples. It should be understood that the actions taken by the person can be unique to a specific environment and the type of abnormal sound event. In some cases, there may not be a camera capable of capturing the sound location. In such cases, the operator can be notified that there is no available video feed.

[0093] In some cases, audio localization may be performed via an audio camera having an audio sensor or microprocessor embedded into the periphery of the camera. Figure 6An exploded perspective view of an exemplary audio camera 400 is shown. Generally, a plurality of directional microphones may be embedded on or in a housing near the front of the camera. The camera may be placed to cover a 360 degree orientation (or other desired angular span, such as 90 degrees, 180 degrees, etc.).

[0094] Figure 6 The audio camera 400 does not show all structural elements and / or functional elements of the camera, but for the sake of clarity, only some of these elements are shown. The exemplary audio camera 400 may include a housing 402 for encapsulating the components of the audio camera 400. The housing 402 may include a plurality of components configured to be fixed to each other. In some cases, the audio camera 400 may be a dome camera including a transparent protective dome 404. However, this is not required. In some cases, the audio camera 400 may be a bullet camera. As required, the audio camera 400 may have a fixed field of view or may be a pan-tilt-zoom (PTZ) camera. It is conceivable that the audio camera 400 may be used indoors and / or outdoors as well as during the day and / or at night as required. In some cases, the housing 402 may be weatherproof for use outside, or one or more night vision light emitting diodes (LEDs) may be provided near the dome 404 for use at night.

[0095] Within the housing 402, the exemplary audio camera 400 may include or house a camera or lens 406. The lens 406 may be configured to direct incident light toward an image sensor (not explicitly shown). The image sensor may process the light captured by the lens 406 into a digital signal. The digital signal (e.g., a video recording) may be stored in a memory (not explicitly shown) of the audio camera 400 or transferred to a VMS or NVR (such as, but not limited to, the VMS / NVR 140). In some cases, the image sensor may be provided as part of a control printed circuit board 410 or otherwise communicatively coupled to the control printed circuit board, although this is not required. The control printed circuit board 410 may include a processor or controller (not explicitly shown). Although some components are described as being part of the control printed circuit board 410, these components may be provided separately from the control printed circuit board 410. In some cases, the controller may be configured to poll various sensors for data, analyze the sensor data, and determine the location of the origin of the sound or noise. The controller may also communicate with a memory or be operably coupled to the memory (not explicitly shown). The memory may be used to store any desired information, such as, but not limited to, machine instructions for how to process data from the sensor and / or digital signals from the image sensor. The memory may be any suitable type of storage device, including, but not limited to, RAM, ROM, EPROM, flash memory, hard drive, etc. In some cases, the controller and / or image sensor may store information in the memory, and may subsequently retrieve the stored information from the memory.

[0096] In some embodiments, the audio camera 400 may be equipped with a communication module. The communication module may allow the audio camera 400 to communicate with the BMS 12 or other components of the system 100 for identifying abnormal sounds in a particular environment, such as but not limited to a network video recorder (NVR) and / or a remote monitoring station. The communication module may provide wired and / or wireless communication. In one example, the communication module may use any desired wireless communication protocol, such as but not limited to cellular communication, ZigBee, REDLINK, etc., as desired. TM , Bluetooth, WiFi, IrDA, Dedicated Short Range Communication (DSRC), EnOcean, and / or any other suitable public or proprietary wireless protocol. In another example, the communication module can communicate via a network cable. In some cases, the network cable can be a power over Ethernet (POE) cable. The exemplary camera 400 can receive power via a POE cable, a separate power cable, a battery, or any other suitable power source as desired.

[0097] The exemplary audio camera 400 may also include a rear frame 408. The rear frame may form part of the housing 402, or may be mounted to the housing 402 to mount the audio camera 400 to a wall or ceiling. In some cases, the rear frame 408 may be coupled to the exterior of the housing 402, while in other cases, the rear frame 408 may be within or internal to the housing 402. In some cases, the rear frame 408 may accommodate cable connections. For example, the rear frame 408 may accommodate, for example, connections between a network cable and a control printed circuit board 410, connections between a power cable and the control printed circuit board 410, and / or connections between an audio cable and the control printed circuit board 410. It is contemplated that the audio camera 400 may include other cables and / or connections as desired. In some cases, the internal circuitry of the connection port within the rear frame 408 may be used to test the connection between the camera 400 and the network. The port may include LEDS that illuminate in a certain color to indicate connectivity.

[0098] The audio camera 400 may also include one or more directional microphones 412a, 412b, 412c, 412d (collectively referred to as 412), which are embedded in the thickness of the housing 402 or disposed within the interior of the housing 402, adjacent to its front end 414, as required. The directional microphones 412 can receive sound from the main audio direction or a predetermined angle. For example, the directional microphones 412 can each receive sound at an angle range of about 5°, about 10°, about 15°, etc. It is contemplated that the angle can be less than 5° or greater than 15°, as required. For example, an omnidirectional microphone can be a type of directional microphone that receives sound over 360°. As required, the directional microphones 412 can be oriented to have uniform angular spacing or can be eccentrically spaced around the perimeter of the housing 402. The directional microphones 412 can be carried by the housing 402 so that each microphone 412 is oriented from the housing 402 in a different direction, so that the approximate direction of a sound event emanating from a particular environment can be determined. In some cases, the number of directional microphones 412 used may be determined by the angle of the directional microphones and the desired coverage. For example, to achieve 360° coverage with directional microphones 412 having a coverage angle of 5°, seventy-two microphones may be used.

[0099] Each directional microphone 412 may be identified using an identifier in the audio stream or using an IP address. A directional microphone 412 may receive sound only from a predetermined direction corresponding to the directional orientation of the microphone 412. Thus, the location of the sound may be determined based on the directional microphone 412 receiving the sound. In some cases, the sound may be captured by multiple directional microphones 412. In such cases, the directional microphone with the strongest signal may be used to determine which direction the sound is coming from.

[0100] It is contemplated that the directional microphone 412 may be operably coupled 416 to the control printed circuit board 410 so that the control printed circuit board 410 may process audio and / or transmit audio. For example, the controller and / or the control printed circuit board 410 may be configured to use multiple directional microphones 412 to determine the approximate direction of a sound event emanating from a particular environment. The controller and / or the control printed circuit board 410 may be configured to control the field of view of the camera 400 (e.g., a PTZ camera) to face the determined approximate direction of the sound event so as to capture a video stream of the source of the sound event. Alternatively or in addition thereto, the controller or control printed circuit board 410 may be configured to provide audio and video output to a remote device.

[0101] Figure 7 A schematic flow chart 500 of an overview of another exemplary method and system for performing adaptive video and audio analysis is shown. In general, the system can detect abnormal audio signatures, locate the abnormality, and send the information to the VMS / NVR 140 and / or other edge devices 104 to change and / or select one or more cameras 108 to the field of view of the located area. In addition, the detection of abnormal audio can be a trigger event that sends a message to the required personnel for appropriate action. First, when there is no abnormal event, the system 100 can receive a normal audio stream from one or more microphones 106 located in a specific environment, as shown in box 502. In some cases, a normal audio stream can be collected during a training cycle or mode. The normal audio stream can then be used as a baseline for subsequent processing of the incoming audio stream with the processor 126, as shown in box 504. When the system 100 is in an operating mode, the incoming audio stream can be processed.

[0102] The normal audio stream can be used to determine whether the incoming audio stream from the microphone 106 in the specific environment includes an abnormal audio event for the specific environment, as shown in block 506. This can be done using any of the techniques described herein. If the normal audio stream does not include an abnormal audio event for the specific environment, the processor 126 can continue to process the incoming audio stream. If it is determined that the incoming audio stream includes an abnormal audio event for the specific environment, the electronic database is accessed to determine the location of the abnormal audio event in the specific environment, as shown in block 508. As described above, the physical location of the microphone 106 and / or the directional orientation of the microphone 106 can be used to be stored in the electronic database and can be used at least in part to determine the location of the abnormal audio event.

[0103] Once the location of the unusual audio event is determined, a camera 108 having a field of view that includes the location of the unusual audio event in the particular environment may be identified, as shown in block 510. The video stream from the identified camera may then be retrieved and displayed on a display, as shown in block 512.

[0104] Figure 8 A schematic flow chart 600 is shown that provides an overview of another exemplary method and system for performing adaptive video and audio analysis. In general, the system can detect abnormal audio signatures, locate the abnormality, and send this information to the VMS / NVR 140 and / or other edge devices 104 to change and / or select the field of view of one or more cameras 108 to locate the area. In addition, the detection of abnormal audio can be a trigger event that sends a message to the required personnel for appropriate action. First, the system 100 can enter a training mode, as shown in box 602. While in training mode, the system 100 can capture real-time audio from multiple microphones in a specific environment, as shown in box 604. The real-time audio can be divided into multiple audio files, as shown in box 606. At least some of the multiple audio files can be saved as normal audio files containing normal audio signatures for a specific environment, as shown in box 608. The abnormal audio signature can be superimposed on at least some of the normal audio files in the multiple normal audio files, and the resulting files can be saved as abnormal audio files containing abnormal audio signatures, as shown in box 610. The normal audio files and the abnormal audio files may be used to train an audio classification model, as shown in block 612 .

[0105] Then, the system 100 can enter the operation mode, as shown in box 614. In the operation mode, real-time audio can be captured from each of the plurality of microphones, as shown in box 616. The real-time audio can be divided into a plurality of operational audio files, as shown in box 618. The operational audio files can then be processed using an audio classification model via a processor to identify one or more abnormal audio signatures in a particular environment, as shown in box 620. This can be done using any of the techniques described herein. The location of an abnormal audio signature in the abnormal audio signature in a particular environment can be determined, as shown in box 622. As described above, the physical location of the microphone 106 and / or the directional orientation of the microphone 106 can be used for storage in an electronic database and can be used at least in part to determine the location of an abnormal audio event. Then a video stream from a camera having a field of view including the location can be retrieved and displayed on a display, as shown in box 624.

[0106] Those skilled in the art will recognize that the present disclosure may be embodied in various forms other than the specific embodiments described and contemplated herein. Therefore, changes in form and details may be made without departing from the scope and spirit of the present disclosure as described in the appended claims.

Claims

1. A method for identifying abnormal sounds in a specific environment, the method comprising: When there is no abnormal audio event, receiving a normal audio stream during the training mode from a microphone located in the specific environment; While in the training mode: Dividing the normal audio stream into a plurality of normal audio clip segments; preparing a plurality of abnormal audio clips by superimposing known abnormal audio events onto one or more normal audio clips of the plurality of normal audio clips; as well as Using one or more of the plurality of normal audio clips and the plurality of abnormal audio clips to train an audio classification model to identify abnormal audio events in the specific environment; in an operating mode, using at least a portion of the normal audio stream as a baseline for subsequent processing of an incoming audio stream with a processor, including using the audio classification model to determine whether the incoming audio stream from the microphone in the particular environment includes an abnormal audio event for the particular environment; When it is determined that the incoming audio stream includes an unusual audio event for the specific environment, accessing an electronic database to determine a location of the unusual audio event in the specific environment, wherein the electronic database stores a physical location of the microphone and / or a directional orientation of the microphone; identifying a camera having a field of view that includes a location of the unusual audio event in the particular environment; and The video stream from the identified camera is retrieved and displayed on the display.

2. The method according to claim 1, wherein the audio classification model is a self-learning model, and the self-learning model uses reinforcement learning and / or transfer learning.

3. The method according to claim 1, further comprising: presenting one or more determined abnormal audio events to an operator; receiving a classification from the operator, the classification being whether the determined abnormal audio event is indeed an abnormal audio event or should be considered a normal audio event for the particular environment; and The audio classification model is updated based on the classification received from the operator. 4 . The method of claim 1 , wherein determining the location of the unusual audio event in the specific environment is based at least in part on physical locations of the microphones in the specific environment stored in the electronic database.

5. The method of claim 4, wherein the microphone is a directional microphone having a directional orientation, and wherein determining the location of the unusual audio event in the specific environment is based at least in part on the physical location of the microphone in the specific environment and the directional orientation of the microphone. 6 . The method of claim 1 , further comprising sending an alert to an operator when it is determined that the incoming audio stream includes an abnormal audio event for the specific environment.

7. The method of claim 1, wherein in the training mode: At least some of the plurality of audio files are saved as normal audio files containing normal audio streams for the specific environment.

8. The method according to claim 7, further comprising: In the described operating mode: capturing real-time audio from each of the plurality of microphones; dividing the real-time audio into a plurality of operable audio files; processing, via a processor, the operational audio file using the audio classification model to identify one or more anomalous audio signatures in the particular environment; determining a location of one of the abnormal audio signatures in the specific environment; as well as A video stream from a camera having a field of view including the location is retrieved and displayed on a display.

Images