Parameter-based load balancing in distributed monitoring systems

Abstract

This application discloses parameter-based load balancing in a distributed surveillance system. A distributed video management system for video surveillance allows monitoring of camera allocation parameters and dynamically reallocating video cameras to available camera nodes in response to detecting changes in the allocation parameters. Thus, the system can provide load balancing of the processing of video data from the video cameras with reference to the allocation parameters. Changes in the allocation parameters can be attributed to a number of potential contexts, including changes in camera node availability, changes in the nature of the captured video data, changes in computational load, or other changes that result in changes in the allocation parameters. The allocation parameters can be continuously monitored to allocate video cameras to camera nodes in the system, enhancing system performance based on potential changing conditions.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is related to U.S. patent application No. _____, filed date [Doll No. STL 074919.00], entitled “SELECTIVE USE OF CAMERAS IN A SURVEILLANCE SYSTEM,” U.S. patent application No. _____, filed date [Doll No. STL 074921.00], entitled “LOW LATENCY BROWSER BASED CLIENT INTERFACE FOR A DISTRIBUTED SURVEILLANCE SYSTEM,” U.S. patent application No. ___, filed date [Doll No. STL 074922.00], entitled “DISTRIBUTED SURVEILLANCE SYSTEM WITH ABSTRACTED FUNCTIONAL LAYERS,” and U.S. patent application No. ____, filed date [Doll No. STL 074923.00], entitled “DISTRIBUTED SURVEILLANCE SYSTEMWITH DISTRIBUTED VIDEO ANALYSIS,” all of which are filed contemporaneously herewith and are specifically incorporated by reference for all that they disclose or teach. Background Art

[0003] Video surveillance systems are a valuable security resource for many facilities. In particular, advances in camera technology have made it economically feasible to install video cameras that provide robust video coverage throughout a facility, assisting security personnel in maintaining on-site safety. Such video surveillance systems may also include recording features that allow for the storage of video data. Stored video data can also help entities provide more robust security, allowing for valuable analysis or assisting with investigations. Live video data feeds can also be monitored in real time at the facility as part of facility security.

[0004] While advances in video surveillance technology have increased the power and popularity of such systems, numerous drawbacks still limit their value. For example, while camera technology has significantly improved, the volume of data generated by such systems continues to grow. This raises the question of how to efficiently store large amounts of video data in a way that can be easily retrieved or otherwise processed. Consequently, the effective management of video surveillance data is becoming increasingly difficult.

[0005] Proposed approaches for managing video surveillance systems include using network video recorders to capture and store video data, or using enterprise servers for video data management. As explained in more detail below, each of these approaches presents unique challenges. Therefore, a need remains for improved video surveillance systems with robust video data management and access. Summary of the Invention

[0006] The present disclosure generally relates to a distributed video surveillance system including distributed processing resources capable of processing and / or storing video data from multiple video cameras at multiple camera nodes. A particular aspect of the present disclosure includes adaptively assigning video cameras to camera nodes for processing video data in accordance with an assignment parameter. In this regard, assigning video cameras to camera nodes in the system can provide load balancing of the assignment parameter. Furthermore, the dynamic or adaptive assignment of cameras can provide robustness to the system based on potential changes in the assignment parameter.

[0007] Thus, a first aspect of the present disclosure includes a method for processing video data in a video surveillance system. The method includes capturing video data at a plurality of video cameras and transmitting the video data from the plurality of video cameras to at least one camera node via a communication network according to a first camera allocation configuration. Executing a video processing module at the camera node to process the video data received at the camera node. The method also includes monitoring camera allocation parameters at the camera node. The camera allocation parameters are based at least in part on the video data received at one or more camera nodes. The method also includes modifying the first camera allocation configuration to a second camera allocation configuration different from the first camera allocation configuration based at least in part on the camera allocation parameters.

[0008] Another aspect of the present disclosure includes a video surveillance system comprising a plurality of video cameras operable to capture video data, and at least one camera node operable to receive video data from one or more of the plurality of video cameras according to a first camera allocation configuration. The system further comprises a video processing module executed at the one or more camera nodes to process the video data received at those camera nodes. A master node is in operable communication with the at least one camera node to monitor camera allocation parameters based at least in part on the video data received at the at least one camera node. The master node is operable to modify the first camera allocation configuration to a second camera allocation configuration different from the first camera allocation configuration in response to a change in the camera allocation parameters.

[0009] This Summary is provided to introduce some concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0010] Other implementations are also described and depicted herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 Two examples of prior art video surveillance systems are depicted.

[0012] Figure 2 An example of a distributed video surveillance system according to the present disclosure is depicted.

[0013] Figure 3 A schematic diagram depicting an example master node of a distributed video surveillance system.

[0014] Figure 4 A schematic diagram depicting an example camera node of a distributed video surveillance system.

[0015] Figure 5 Depicts an example of abstract camera, processing, and storage layers for a distributed video surveillance system.

[0016] Figure 6 Depicted are examples of a client in operable communication with a distributed video surveillance system to receive real-time data for presentation in a native browser interface of the client.

[0017] Figure 7 An example of distributed video analytics for a distributed video surveillance system is depicted.

[0018] Figure 8 An example of a first camera allocation configuration for multiple video cameras and camera nodes of a distributed video management system is depicted.

[0019] Figure 9 An example of a second camera allocation configuration for a plurality of video cameras and camera nodes of a distributed video management system in response to detecting that a camera node is unavailable is depicted.

[0020] Figure 10 Depicted is an example of a second camera allocation configuration for a plurality of video cameras and camera nodes of a distributed video management system in response to a change in an allocation parameter at one of the camera nodes.

[0021] Figure 11An example of a second camera allocation configuration for a plurality of video cameras and camera nodes of a distributed video management system is depicted in which a video camera is disconnected from any camera node based on a priority of the video camera.

[0022] Figure 12 Depicted are example operations for load balancing in a distributed video management system.

[0023] Figure 13 A processing device is depicted that can facilitate aspects of the present disclosure. DETAILED DESCRIPTION

[0024] Although the examples in the following disclosure are susceptible to various modifications and alternative forms, specific examples are shown in the drawings and described in detail herein. However, it should be understood that it is not intended to limit the scope of this disclosure to the particular forms disclosed, but that this disclosure will cover all modifications, equivalents, and alternatives that fall within the scope defined by the claims.

[0025] Figure 1 Two prior art approaches to system architecture and management of video surveillance systems are described. These two approaches include Figure 1 The top portion of the device-based system 1 and Figure 1 2. In the device-based system 1, a video camera 10 is in operable communication with a network 15. A device 12 is also in communication with the network 15. The device 12 receives video data from the video camera 10 and displays the video data on a monitor 14 connected to the device 12.

[0026] Given the simplicity of the hardware required to implement system 1, appliance-based systems 1 generally offer a relatively low-cost solution. However, due to the limited processing power of most devices 12, the number of cameras supported by appliance-based systems may be limited, as all video cameras 10 only provide video data to the device 12 for processing and display on the display 14. Furthermore, the system is not scalable, as once the processing power of a device 12 is reached (e.g., due to the number of cameras in system 1), expansion for additional cameras is not possible. Instead, to supplement system 1, entirely new devices 12 must be implemented as separate, standalone systems without integration with existing devices 12. Furthermore, due to the relatively limited processing power of devices 12, appliance-based systems 1 offer limited capabilities for video data analysis or storage capacity. Furthermore, such systems 1 generally facilitate viewing and / or storage of a limited number of real-time video data feeds from video cameras 10 at any given time and typically allow such video to be presented only on a single monitor 14 or a limited number of monitors connected to the device 12. That is, in order to review real-time or archived video data, a user must be physically present at the location of the device 12 and monitor 14 .

[0027] The enterprise server-based system 20 typically includes a plurality of video cameras 10 in operable communication with a network 15. A server instance 16 also communicates with the network 15 and receives all video data from all cameras 10 for processing and storage. The server 16 typically includes a storage array and acts as a digital video recorder (DVR) to store the video data received from the cameras 10. Clients 18 can be connected to the network 15. The clients 18 can allow a physical location remote from the server 16 to view the video data from the server 16 (e.g., in contrast to the device-based system 1 in which the monitor 14 is directly connected to the device 12). However, the server 16 typically includes platform-dependent proprietary software for processing the video data from the cameras 10 for storage in the server 16 storage array.

[0028] Furthermore, the server 16 and the clients 18 include platform-dependent proprietary software to facilitate communications between the server 16 and the clients 18. Consequently, a user or enterprise must purchase and install a platform-dependent client software package on any client 18 that it desires to use to access video data and / or control the system 20. This limits the ability of users to access video data from the system 20 because any user must have access to a pre-configured client 18 equipped with the appropriate platform-dependent proprietary software, which requires additional expense to license such software.

[0029] Compared to the appliance-based system 1, the enterprise server-based system 20 is typically a relatively expensive implementation that can be installed for large enterprises. For example, when a single server 16 handles all processing and storage of all video data from the system, such a system 20 typically requires a very powerful server 16 to facilitate the management of the video data from the cameras 10. In addition, the platform of the server 16 and the client 18 relies on proprietary software that requires licensing fees, which can be based on the number and / or features (e.g., data analysis features) of cameras 10 available to the user. Furthermore, the proprietary software that enables the functionality of the client 18 must be installed and configured as a separate software package. In turn, installing and maintaining software at the client 18 can increase the complexity of the system 1. Furthermore, if a user wishes to use different client 18 devices, any such device must first be provisioned with the software resources required for operation. As a result, the ability to access and manage the system 1 is limited.

[0030] While such an enterprise server-based system 20 can be scaled, the capital cost of expanding the system 20 is high. Specifically, while the server 16 does have a limit on the number of cameras 10 it can support relative to the device 12 in terms of computational complexity, this limit is typically higher than the number of cameras 10 that the device 12 can support. In any case, once the maximum number of cameras 10 is reached, any additional cameras 10 effectively require the use of additional servers 16 or the purchase of a new system 20 by increasing the capacity of the server 16 and paying for licensing fees for the additional servers 16 or capacity. Furthermore, the proprietary software that needs to be installed on the client 18 is typically platform-dependent and required for any client 18 wishing to interact with the system 20. This increases the complexity and cost of any client 18 and limits the functionality of the system 20. Furthermore, the enterprise server-based system 20 includes a static camera-to-server mapping, such that in the event of a server unavailability or failure, all cameras 10 mapped to the server 16 become unavailable for real-time video streaming or storage of video data, thereby rendering the system 20 ineffective in the event of such a failure.

[0031] Therefore, the present disclosure relates to a distributed video management system (VMS) 100 comprising a distributed architecture. An example of such a VMS 100 is depicted in FIG. Figure 2The distributed architecture of VMS 100 facilitates numerous benefits over the aforementioned appliance-based system 1 or server-based system 20. Generally, VMS 100 includes three functional layers that can be abstracted relative to one another to provide the ability to dynamically reconfigure the mapping between video cameras 110, camera nodes 120 for processing video data, and storage capacity 150 / 152 within VMS 100. While this is discussed in greater detail below, the abstraction of VMS 100's functional layers facilitates a highly dynamic and configurable system that is easily scalable, robust to component failures, adaptable to given events, and cost-effective to install and operate. Because the functional layers are abstract, static component-to-component mappings are not required. That is, any one or more video cameras 110 can be associated with any one of a plurality of camera nodes 120, which can receive video data from the associated video cameras 110 and process the video data from the associated video cameras 110. The camera node 120 then processes the video data (e.g., for storage in the storage volumes 150 / 152 or for real-time streaming to the client device 130 for viewing the video data in real time). The camera node 110 is operable to perform video analytics on the video data of the associated camera 110 or on stored video data (e.g., of the associated video camera 100 or of a non-associated video camera 110). Furthermore, when storage resources of the system 100 are also drawn from the camera node 120, the video data can be stored in a flexible manner that allows it to be retrieved by any camera node 120 of the system.

[0032] In this regard, upon failure of any given node in the system, the cameras assigned to the failed camera node can be reassigned (e.g., automatically) to another camera node, such that processing of video data is virtually uninterrupted. Furthermore, the association of cameras to nodes can be dynamically modified in response to actual processing conditions at the nodes (e.g., a camera can be associated from a node performing complex video analysis to another node). Similarly, because camera nodes 120 can be relatively inexpensive hardware components, additional camera nodes 120 can be easily added (e.g., in a plug-and-play manner) to system 100 to provide highly granular scalability (e.g., as opposed to having to deploy entirely new server instances in the case of server-based systems 20 that only provide low granularity scaling).

[0033] The flexibility of the VMS system 100 extends to the client 130 in the system. The client 130 can refer to a client device or software delivered to a device for execution at the device. In any aspect, the client 130 can be used to view the video data of the VMS 100 (e.g., in real time or from the storage device 150 / 152 of the system 100). Specifically, the present disclosure contemplates the use of a standard web browser application that is generally available and executable on various computing devices. As described in more detail below, the VMS 100 can utilize the processing power at each camera node 120 to process the video data into an appropriate transmission mechanism, which can be based at least in part on the context of the request for video data. As an example, a request from a client 130 for viewing real-time video data from a camera 110 in real time can cause the camera node 120 to process the video data of the camera 110 into a real-time, low-latency format to be delivered to the client 130. Specifically, such a low-latency protocol may include a transport mechanism that allows data to be received and presented at the client using a standard web browser, using only the native functionality of a standard web browser, or via executable instructions provided by a web page sent to the client 130 for presentation in a standard web browser (e.g., without requiring the installation of external software at the client in the form of a third-party application, browser plug-in, browser extension, etc.). Consequently, any computing device that executes a standard web browser can be used as a client 130 to access the VMS 100, without requiring any proprietary or platform-dependent software, and without requiring any pre-configuration of the client 130. This can allow access by any computing system operating any operating system, as long as the computing device is capable of executing a standard web browser. Thus, a desktop computer, laptop computer, tablet computer, smartphone, or other device can act as a client 130.

[0034] The abstract architecture of VMS 100 can also allow for flexible data processing. For example, camera nodes 120 of VMS 100 can apply analytical models to video data processed at camera nodes 120 to perform video analytics on the video data. The analytical models can generate analytical metadata about the video data. Non-limiting examples of analytical methods include object detection, object tracking, face recognition, pattern recognition / detection, or any other suitable video analytics technique. Given the abstraction between video cameras 110 and camera nodes 120 of VMS 100, the configuration of the processing of video data can be flexible and adaptable, which can allow even relatively complex analytical models to be applied to some or all of the video data, thereby being dynamically pre-configured in response to peak analytical loads.

[0035] Continue to refer Figure 2, schematically depicts a VMS 100 for managing edge monitoring devices in a monitoring system according to the present disclosure. The VMS 100 includes a plurality of cameras 110, each of which is in operable communication with a network 115. For example, Figure 2 As shown, cameras 110a to 110g are shown. However, it should be understood that additional or fewer cameras may be provided in the VMS 100 according to the present disclosure without limitation.

[0036] Camera 110 may be an Internet Protocol (IP) camera capable of providing packetized video data from camera 110 for transmission over network 115. Network 115 may be a local area network (LAN). In other examples, network 115 may be any suitable communications network, including a public switched telephone network (PSTN), an intranet, a wide area network (WAN) (such as the Internet), a digital subscriber line (DSL), a fiber optic network, or other suitable network, without limitation. Each video camera 110 may be independently associated with (e.g., assigned to) a given one of multiple camera nodes 120.

[0037] Therefore, the VMS 100 further includes a plurality of camera nodes 120. For example, Figure 2 , three camera nodes 120 are shown, including a first camera node 120a, a second camera node 120b, and a third camera node 120c. However, it should be understood that additional or fewer camera nodes 120 may be provided without departing from the scope of the present disclosure. Furthermore, camera nodes 120 may be added to or removed from the system 100 at any time, in which case the camera-to-node assignment or mapping may be automatically reconfigured. Each camera node 120 may also be in operable communication with the network 115 to facilitate receiving video data from one or more of the cameras 110 associated with each respective node 120.

[0038] The VMS 100 also includes at least one master node 140. The master node 140 is operable to manage the operation and / or configuration of the camera nodes 120 to receive and / or process video data from the cameras 110, coordinate storage resources of the VMS 100, generate and maintain a database related to captured video data of the VMS 100, and / or facilitate communications with the clients 130 for accessing video data of the system 100.

[0039] While a single master node 140 is shown and described, the master node 140 may include camera nodes 120 responsible for certain system management functions. Not all management functions of the master node 140 need to be performed by a single camera node 120. In this regard, while a single master node 140 is described for simplicity, it will be appreciated that the master node functions described herein with respect to a single master node 140 may actually be distributed among different camera nodes 120. Thus, a given camera node 120 may serve as the master node 140 for coordinating camera assignments for the camera nodes 120, while another camera node 120 may serve as the master node 140 for maintaining a database of video data related to the system. Thus, as will be described in greater detail below, the various management functions of the master node 140 may be distributed among various camera nodes 120. Thus, while a single given master node 140 is shown, it will be appreciated that any of the camera nodes 120 may serve as the master node 140 for different corresponding functions of the system 100.

[0040] Furthermore, the various management functions of the master node 140 can be subject to a master selection technique to distribute such functions to different ones of the camera nodes 120 to perform the master node functions. For example, the role of the master node 140 can be assigned to a given camera node 120 using a master selection technique, such that all management functions of the master node 140 are assigned to the given camera node 120. Alternatively, individual ones of the management functions can be individually distributed to one or more camera nodes 120 using master selection. This provides a robust system in which even the unavailability of a master node 140 or camera node 120 performing some management functions can be easily corrected by applying master selection to select a new master node 140 in the system or to reallocate management functionality to a new camera node 120.

[0041] The hardware of the camera nodes 120 and the master node 140 can be identical. In other examples, a dedicated master node 140 can be provided that can have different processing capabilities than other camera nodes 120 (e.g., more or less powerful hardware in terms of processor and / or memory capacity). Furthermore, not all camera nodes 120 have the same processing capabilities. For example, some camera nodes 120 can include increased computing specifications relative to other camera nodes 120, including, for example, increased memory capacity, increased processor capacity / speed, and / or increased graphics processing capabilities.

[0042] As can be appreciated, the VMS 100 can store video data from the video cameras 110 in storage resources of the VMS 100. In one embodiment, the storage capacity can be provided in one or more different exemplary configurations. Specifically, in one example, each camera node 120 and / or master node 140 can have an attached storage device 152 at each respective node. In this regard, each respective node can store metadata for video data processed by the node and any metadata generated at the node on the corresponding attached storage device 152 at each respective node for use with the video data processed at the node 120. In an alternative arrangement, the locally attached storage device 152 at each camera node 120 and master node 140 can include physical drives abstracted into a logical storage unit 150. In this regard, it is possible that video data processed at a first node can be at least partially transferred to another node for data storage. In this regard, the logical storage unit 150 can be presented as an abstract storage device or storage resource accessible by any node 120 in the system 100. The actual physical form of the logical storage unit 150 may take any suitable form or combination of forms. For example, the physical drives associated with each node may include a storage array, such as a RAID array, which forms a single virtual volume that can be addressed by any camera node 120 or master node 140. Additionally or alternatively, the logical storage unit 150 may be in operative communication with a network 115 with which the camera nodes 120 and master node 140 also communicate. In this regard, the logical storage unit 150 may include a network attached storage (NAS) device capable of receiving data from any camera node 120. The logical storage unit 150 may include storage local to the camera node 120, or may include remote storage, such as a cloud-based storage resource or the like. In this regard, although both the logical storage unit 150 and the locally attached storage device 152 are shown in Figure 2 , but the locally attached storage device 152 may include at least a portion of the logical storage unit 150. In addition, the VMS 100 does not necessarily include two types of storage devices, which is for illustration purposes only. Figure 2 Shown in.

[0043] Further references Figure 3, a schematic diagram illustrating an example of a master node 140 is shown. The master node 140 may include multiple modules for managing the functionality of the VMS 100. As described above, although a single master node 140 including a master node module is shown, it should be understood that any camera node 120 can serve as the master node 140 for any individual functionality of the master node module. In other words, the role of the master node 140 for any one or more of the master node functionalities can be distributed among the camera nodes 120. In any aspect, the modules corresponding to the master node 140 may include a web server 142, a camera distributor 144, a storage manager 146, and / or a database manager 148. In addition, the master node 140 may include a network interface 126 that facilitates communication between the master node 140 and the video cameras 110, camera nodes 120, storage devices 150, clients 130, or other components of the VMS 100.

[0044] The web server 142 of the master node 140 can coordinate communications with the client 130. For example, the web server 142 can transmit a user interface (e.g., HTML code that defines how the browser renders the user interface) to the client 130, which allows the client 130 to render the user interface in a standard browser application. The user interface may include design elements and / or code for retrieving and displaying video data from the VMS 100 in a manner described in more detail below.

[0045] Regarding the camera allocator 144, the master node 140 can facilitate camera allocation or assignment, such that the camera allocator 144 creates and executes camera-to-node mappings to determine which camera node 120 is responsible for processing video data from the video cameras 110. That is, compared to the device-based system 1 or the enterprise server-based system 50, a subset of the video cameras 110 of the VMS 100 can be assigned to different camera nodes 120. For example, the camera allocator 144 can be operable to communicate with the cameras 110 to provide instructions to the video cameras 110 regarding the camera nodes 120 to which the video cameras 110 will transmit their video data. Alternatively, the camera allocator 144 can instruct the camera nodes 120 to establish communication with and receive video data from specific ones of the video cameras 110. The camera allocator 144 can create such camera-to-node associations and record the associations in a database or other data structure. In this regard, the system 100 can be a distributed system, as any one of the camera nodes 120 can receive and process video data from any one or more of the video cameras 110.

[0046] Furthermore, the camera allocator 144 is operable to dynamically reconfigure camera-to-node mappings during the load balancing process. In this regard, the camera allocator 144 can monitor allocation parameters at each camera node 120 to determine whether to modify the camera-to-node mapping. In this regard, the VMS 100 can be monitored for changes, and the camera allocator 144 can responsively modify camera allocations from a first camera allocation configuration to a second camera allocation configuration to improve or maintain system performance. The allocation parameters can be any one or more of a plurality of parameters that are monitored and used to determine camera allocations. Thus, the allocation parameters can change in response to a variety of events that may occur within the VMS 100, as described in greater detail below.

[0047] For example, in the event of a failure, power loss, or another event that renders a camera node 120 unavailable, the camera allocator 144 may detect or otherwise be notified of the camera node's unavailability. Subsequently, the camera allocator 144 may reassign the video camera previously associated with the unavailable node to another node 120. The camera allocator 144 may communicate with the reassigned camera 110 to update instructions for communicating with the new camera node 120. Alternatively, the newly assigned camera node may assume the role of establishing contact with and processing video data from the video camera 110 that previously communicated with the unavailable camera node 120 to update instructions and establish a new camera-to-node assignment based on the new assignment provided by the camera allocator 144. In this regard, the system 100 provides increased redundancy and flexibility with respect to processing video data from the cameras 100. Furthermore, even in the absence of a camera node 120 failure, the video data feed from the camera 110 may be load-balanced to the camera nodes 120, allowing for the application of different analytical models, etc.

[0048] A given camera node 120 may be paired with a subset of cameras 110, the subset including one or more of the cameras 110. As an example, in Figure 2 In the example embodiment, cameras 110a-110c can be paired with camera manager 120a, so that camera manager 120a receives video data from cameras 110a-110c. Cameras 110d-110f can be paired with camera manager 120b, so that camera manager 120b receives video data from cameras 110d-110f. Camera 110g can be paired with camera manager 120c, so that camera manager 120c receives video data from camera 110g. However, this configuration can change in response to load balancing operations, failure of a given camera node, network conditions, or any other parameters.

[0049] For example, reference Figure 8 , showing a first camera allocation configuration. Two camera nodes, camera node 120a and camera node 120b, can process data from video cameras 110a - 110e via network 115. Figure 9 is a schematic representation for illustration. Thus, although camera 110 is shown as communicating directly with node 120, camera 110 may communicate with node 120 via a network connection. Similarly, although master node 140 is shown as communicating directly with camera node 120, such communication may also be via network 115 ( Figure 8 In any respect, Figure 8 In the illustrated first camera allocation configuration, video cameras 110a, 110b, and 110c transmit video data to a first camera node 120a for processing and / or storage. Additionally, video cameras 110d and 110e transmit video data to a second camera node 120b for processing and / or storage. The first camera allocation can be established by camera allocator 144 of master node 140 in a manner that distributes the mappings of video cameras 110 among available camera nodes 120 to balance allocation parameters among the camera nodes 120.

[0050] Upon detecting a change in the allocation parameter, the camera allocator 144 may modify the first camera allocation in response to detecting the change in the monitored allocation parameter. For example, such a change may be in response to adding or removing a camera node 120 from the VMS 100, when the computational load at the camera node 120 changes, when the video data from the video camera 110 changes, or when any other change results in a change in the allocation parameter. For example, with further reference to Figure 9 , depicts a scenario in which camera node 120b becomes unavailable (e.g., due to a loss of communication at camera node 120b, a loss of power at camera node 120b, or any other failure or condition that causes camera node 120b to lose the ability to process and / or store video data). In response, master node 140 may detect such a change and change the first camera allocation configuration from Figure 8 The camera allocation configuration shown is modified to the second camera allocation configuration, such as Figure 9 shown.

[0051] exist Figure 9 In the second camera allocation configuration shown, all cameras 110a-110e are mapped to communicate with camera node 120a. However, it should be understood that other camera nodes 120 ( Figure 910d and 110e are assigned to any available node 120 in the VMS 100. Therefore, only two camera nodes 120a and 120b are shown for simplicity of explanation. In this regard, the modification of the camera allocation configuration can be based at least in part on the allocation parameters. That is, the camera allocation parameters can be used to load balance the system based on the video data of the camera 110 on all available camera nodes 120 (e.g., based on the allocation parameters). Therefore, although all video cameras 110 are reallocated to Figure 9 The first camera node 120a is used in the example, but cameras 110d and 110e may be assigned to alternative camera nodes in other ways to balance the computational load and storage load or other distribution parameters across all available nodes 120.

[0052] In addition, although the camera node 120 is Figure 9 Another scenario, which is shown as unavailable in FIG, but in which load balancing may occur, is when one or more camera nodes 120 are added to the system, making one or more additional camera nodes available. In this scenario, a new camera allocation configuration may be generated to balance the video data processing of all cameras 110 in the VMS 100 with respect to allocation parameters based on the video data generated by the cameras 110. In this regard, it will be appreciated that changes to the allocation parameters monitored by the camera allocator 144 of the master node 140 may occur in response to any number of conditions, and such changes may result in modifications to the existing camera allocation configuration.

[0053] Thus, the allocation parameters may be related to the video data of the camera node 110 being assigned. For example, the allocation parameters may be related to time-based parameters, the spatial coverage of the camera, the computational load of processing the camera's video data, the camera's assignment category, and the camera's assignment priority. The allocation parameters may be at least partially affected by the nature of the video data of a given camera. For example, a given camera may present video data that is more computationally demanding than another camera. For example, a first camera may be facing the main entrance of a building. A second camera may be located in an interior hallway that does not receive much traffic. Video analytics may be applied to both sets of video data from the first camera and the second camera to perform face recognition. The video data from the first camera may be more computationally demanding for the camera node than the video data from the second camera simply because of the nature / location of the first camera, which is located at the main entrance and includes more faces than the second camera. In this regard, the camera allocation parameters may be based at least in part on the video data of the particular camera to be assigned to the camera node.

[0054] in this regard, Figure 10Another scenario is depicted in which a change in a camera allocation parameter is detected, and the camera allocation configuration is modified in response to the change. Figure 10 You can Figure 8 The first camera allocation configuration is changed to Figure 10 The second camera allocation configuration shown in . Figure 10 , video camera 110e may begin capturing video data that causes the computational load on camera module 120b to increase beyond a threshold. In turn, camera allocator 144 of master node 140 may detect this change and modify the first camera allocation configuration to a second camera allocation configuration such that camera 110d is associated with camera node 120a. That is, camera node 120b may be dedicated to processing video data from camera 110e in response to a change in the video that increases the computational load for processing this video data. An example would be video data that includes a significant increase in detected objects (e.g., additional faces to be processed using face recognition) or motion to be processed. In Figure 10 In the example shown, camera node 120a may have sufficient capacity to process video data from camera 110d.

[0055] Figure 11 An example is further shown in which the total computing capacity of the VMS 100 based on the available camera nodes 120 is exceeded. Figure 11In the depicted scenario, camera 110d may be disconnected from any camera node 120, preventing the VMS 100 from processing its video data. That is, if the total VMS 100 capacity is exceeded, cameras may be selectively "abandoned." Cameras may have assigned priority values, which may be based in part on the allocation parameters described above. For example, if two cameras with overlapping spatial coverage are provided (e.g., one camera monitors an area from a first direction, while another camera monitors the same area from a different direction), one of the cameras with overlapping spatial coverage may have a relatively low priority. Consequently, when one of the cameras is disconnected, continuity of monitoring of the area covered by the camera can be maintained while reducing the computational load on the system. When available computational load is restored (e.g., due to changes in the computational load of other cameras or by adding another node to the system), the disconnected camera may be reassigned to a camera node using load balancing methods. In other cases, other allocation parameters may be used to determine priority, including establishing camera categories. For example, a camera may be assigned to an "interior camera" category or a "surrounding camera" category based on whether its location / field of view is inside or outside the facility. In this case, one type of camera may be prioritized over another type of camera based on a specific scenario occurring, which may be related to the VMS 100 (e.g., computing capacity / load of the VMS 100) or an external event (e.g., an alarm at the facility, a shift change at the facility, etc.).

[0056] The master node 140 may also include a storage manager 146. Video data captured by the cameras 110 is processed by the camera nodes 120 and, once processed, may be stored in persistent storage. The video data generated by the VMS 100 may include a relatively large amount of data for storage. Therefore, the VMS 100 may generally implement a storage policy for video data captured and / or stored by the VMS 100. As will be described in greater detail below, the abstract storage resources of the VMS 100 facilitate persistent storage of the video data by the camera nodes 120 in a manner that allows any camera node 120 to access the stored video data, regardless of the camera node 120 that processed the video data. Thus, any camera node 120 may be able to retrieve and reprocess the video data according to the storage policy.

[0057] For example, a storage policy may indicate that video data of a predefined currency (e.g., video data captured within the last 24 hours of operation of the VMS 100) may be stored in its entirety at the video data's original resolution. However, long-term storage of such video data at full resolution and full frame rate may be impractical or infeasible. Therefore, the storage policy may include an initial period of full data retention, wherein all video data is stored at full resolution, and subsequent processing of the video data after the initial period to reduce the size of the video data on disk.

[0058] To this end, the storage policy may specify other parameters that control how video data is stored or whether such data is retained. The storage manager 146 may execute the storage policy based on the parameters of the storage policy with respect to the stored video data. For example, based on the parameters defined in the storage policy, video data may be deleted or stored at a reduced size (e.g., by reducing the video resolution, frame rate, or other video parameters to reduce the overall size of the video data on disk). Reducing the size of video data stored on disk may be referred to as "pruning." One such parameter governing the pruning of video data may relate to the amount of time that has passed since the video data was captured. For example, the size of data older than a given period (e.g., greater than 24 hours) may be deleted or reduced. Furthermore, multiple pruning stages may be performed such that the size of the data is further reduced or deleted as the video becomes less recent.

[0059] Furthermore, since any camera node 120 is operable to retrieve any video data from storage for reprocessing, video data can be reprocessed (e.g., truncated) by a camera node different from the camera node that originally processed and stored the video data from the video camera. Thus, reprocessing or truncating can be performed by any camera node 120. Reprocessing of video data by a camera node can be performed during idle periods of the camera node 120 or when it is determined that the camera node 120 has spare computing capacity. This can occur at different times for different camera nodes, but can occur during times of low processing load, such as after business hours or during times when a facility is closed or activity is reduced.

[0060] Furthermore, parameters for pruning can be related to analytical metadata for the video data. As described in greater detail elsewhere in this application, the camera node 120 may include an analytical model to apply video analytics to the video data processed by the camera module. Such video analytics may include generating analytical metadata about the video. For example, the analytical model may include object detection, object tracking, facial recognition, pattern detection, motion analysis, or other data extracted from the video data when analyzed using the analytical model. The analytical metadata may provide parameters for data pruning. For example, any video data without motion may be deleted after an initial retention period. In another example, only video data that includes specific analytical metadata may be retained (e.g., only video data in which a given object was detected may be stored). Furthermore, only data from a specific camera 110 may be retained beyond the initial retention period. Thus, highly valuable video data feeds (e.g., video data associated with critical locations such as building entrances or high-security areas of a facility) can be maintained without reducing their size. In any regard, the storage manager 146 may manage the application of such storage policies to the video data stored by the VMS 100.

[0061] The master node 140 may also include a database manager 148. As described above, the video camera 110 may be associated with any camera node 120 for processing and storing video data from the video camera 120. Furthermore, the video data may be stored in an abstracted manner in a logical storage unit 150, which may or may not be physically co-located with the camera node 120. Therefore, the VMS 100 may advantageously maintain a record of the video data captured by the VMS 100, providing important system metadata about the video data. Such system metadata may include, among other potential information, which video camera 110 captured the video data, the time / date the video data was captured, which camera node 120 processed the video data, which video analytics were applied to the video data, resolution information about the video data, frame rate information about the video data, the size of the video data, and / or the location where the video data is stored. Such information may be stored in a database generated by the database manager 148. The database may include correlations between the video data and the system metadata associated with the video data. In this regard, the provenance of the video data may be recorded by the database manager 148 and captured in the resulting database. The database can be used to manage video data and / or track the flow of video data through the VMS 100. For example, as described above, the storage manager 146 can utilize the database to apply storage policies to the data. In addition, a request for data from a client 130 can include a reference to the database to determine the location of video data to be retrieved for a given parameter (such as any one or more of the metadata components described above). The database can be generated by the database manager 148, but the database can be distributed among all camera nodes 120 to provide redundancy to the system in the event that the master node 140 executing the database manager 148 fails or is unavailable. Database updates corresponding to any given camera node 120 can be driven by specific events or can occur at predetermined time intervals.

[0062] The database can also associate video data with analytical metadata about the video data. For example, as described in more detail below, analytical metadata can be generated by applying video analytics to the video data. Such analytical metadata can be embedded in the video data itself, or provided as a separate metadata file associated with a given video data file. In either aspect, the database can associate such analytical metadata with the video data. This can facilitate pruning activities or searching video data. With regard to the former, as described above, pruning according to a storage policy can include processing video data based on analytical metadata (e.g., based on the presence or absence of moving or detected objects). In addition, a search conducted by a user can request all video data in which a particular object or similar condition was detected.

[0063] Further references Figure 4, shows an illustrative example of a camera node 120. As can be understood from the foregoing, the camera node 120 may include an instance of the database 132 provided by the master node 140 executing the database manager 148. In this regard, the camera node 120 may reference the database for retrieving and / or providing video from the logical storage volume of the VMS 100 and / or for reprocessing the video data (e.g., according to a storage policy).

[0064] The camera node 120 may include a video analytics module 128. The video analytics module 128 may be operable to apply an analytics model to video data processed by the camera node 120 upon receipt from the camera 110. The video analytics module 128 may apply a machine learning model to the video data processed at the camera node 120 to generate analytics metadata. For example, as described above, the video analytics module 128 may apply a machine learning model to detect objects, track objects, perform facial recognition, or other analytics on the video data, which in turn may result in the generation of analytics metadata about the video data.

[0065] The camera node 120 may also include a module adapted to process the video data into an appropriate transport mechanism based on the nature of the data or the intended use of the data. In this regard, the camera node 120 includes a codec 122 (i.e., an encoder / decoder) that can decode the received data and re-encode the data into a different coded video format. The coded video format may include packetized data such that each data packet is encoded according to the selected coded video format. The camera node 120 may also include a container formatter 124 that can package the coded video packets into an appropriate container format. The camera module 120 also includes a network interface 126 that is operable to determine a communication protocol for transmitting the coded video packets in the digital container format.

[0066] Formatting the video data into an appropriate transport mechanism can allow for optimized delivery and / or storage of the video data. For example, the video data can be delivered from the camera 110 to the camera node 120 using the Real Time Streaming Protocol (RTSP). However, RTSP may not be the optimal protocol for storing and / or delivering the video data to the client 130 (e.g., RTSP is typically not supported by standard web browsers and, therefore, typically requires specific software or plug-ins (such as specific video players) to render the video in the browser display). The camera node 120 can reformat the video data into an appropriate transport mechanism based on the context in which the video data is requested.

[0067] After selecting an appropriate communication protocol, the network interface 126 can use the communication protocol to transmit the encoded video packets to a standard web browser on the client device. In one example, a client 130 can request to view video data from a given video camera 110 in real time. Therefore, the codec 122, the container formatter 124, and the network interface 126 can each select an appropriate encoded video format, container format, and communication protocol to facilitate the transport mechanism to provide the video data to the client 130 in real time. In contrast, the client 130 can alternatively request video data from the logical storage unit of the VMS 100. As can be appreciated, the liquidity of such data is not as important as in the case of real-time data. One or more different encoded video formats, container formats, and communication protocols can be selected. For example, in such cases where data liquidity is less important, a more resilient or bandwidth-efficient encoded video format, container format, and communication protocol can be selected, which has a higher latency in providing video to the client 130.

[0068] For purposes of illustration and not limitation, the transport mechanism may include any combination of encoded video formats, container formats, and communication protocols. Example transport mechanisms include JSMpeg, HTTP Live Streaming (HLS), MPEG-1, and WebRTC. JSMpeg utilizes MPEG-1 encoding (e.g., MPEG-TS splitter, WebAssembly MPEG-1 video decoder, and MPEG-2 audio decoder). In this regard, the JSMpeg transport mechanism uses a transport stream (TS) container format and the WebSocket communication protocol. The JSMpeg transport mechanism can then be decoded at the client 130 using a JSMpeg program that can be included in a web page (e.g., HTML code sent to a browser, etc.) and does not require the use of plug-ins or other applications other than a native web browser. For example, the JSMpeg transport mechanism can use a WebGL&Canvas2D rendering program and WebAudio sound output. The JSMpeg transport mechanism can provide extremely low latency for video data, but utilizes slightly higher bandwidth consumption relative to the other transport mechanisms described herein.

[0069] Another transport mechanism may be WebRTC, which may utilize H.264 encoding, VP8, or another encoding. WebRTC may utilize container formats including MPEG-4 or WebM. The communication protocol of WebRTC may include RTC peer connections to provide signaling. Video may be delivered using WebSockets. In the WebRTC transport mechanism, standard browsers may include native decoders for decoding encoded video data. WebRTC provides extremely low latency for video data, but increases the complexity of the system by using a signaling server in the form of an RTC peer connection. However, WebRTC has relatively low bandwidth utilization.

[0070] Another transport mechanism that can be utilized includes HLS or MPEG-DASH. The encoded video format of HLS / MPEG-DASH can be MPEG-2, MPEG-4, or H.264. The container format can be MPEG-4, and the communication protocol can be HTTP. In this regard, the decoder can natively decode the encoded video data. The HLS / MPEG-DASH transport mechanism has higher latency than the other transport mechanisms described, but has robust browser support and lower network bandwidth usage.

[0071] As described above, the VMS 100 may include an abstraction system that allows the capture of video data, the processing of video data, and the storage of video data to be abstracted between the various components of the VMS 100. For example, further reference is made to Figure 4 , schematically depicts three "layers" of VMS 100 functionality. Specifically, an acquisition layer 310, a processing layer 320, and a storage layer 330 are shown. Camera 110 may include acquisition layer 310. Camera node 120 and master node 140 may include processing layer 320. In addition, a logical storage volume may include storage device 150 of storage layer 330. These layers are referred to as abstraction layers because the specific combination of hardware components that acquire, process, and store video data for VMS system 100 can be variable and dynamically associated. In other words, network communication between the hardware components of VMS 100 can allow for the abstraction of each of the acquisition, processing, and storage functions. Thus, for example, any of cameras 110 can provide video data to any of camera nodes 120, which can store the video data in the logical storage volume of storage device 150 without restriction.

[0072] As described above, the VMS 100 also includes a client 130 that is operable to communicate with the network 115. The client 130 is operable to communicate with the VMS 100 to request and receive video data from the system 100. In this regard, the VMS 100 can not only store video data from the video camera 110, but also provide a real-time stream of the video data for viewing by one or more users. For example, video surveillance cameras are often monitored in real time by security personnel. By "real-time" or "near real-time," it is intended that the data provided has sufficient currency to be used for security operations. In this regard, real-time or near real-time does not require instantaneous delivery of video data, but can include delays that do not affect the effectiveness of the surveillance video data, such as delays of less than 5 seconds, less than 3 seconds, or less than approximately 1 second.

[0073] One goal of the present disclosure is to help clients 130 use standard web browser applications to present real-time video data to users in a convenient manner. It is particularly noteworthy that it is particularly beneficial to allow clients 130 to execute common and low-cost applications for accessing video data (for example, compared to requiring pre-installed and pre-configured platforms that rely on proprietary software to interact with management systems). In this regard, the specific application type expected to be used at client 130 is a standard web browser. Examples of such browsers include Google Chrome, Mozilla Firefox, Microsoft Edge, Microsoft Internet Explorer, Opera browser and / or Apple Safari. Such standard web browsers are capable of natively processing certain data received via a network to generate a user interface on a client device. For example, such standard web browsers typically include native application programming interfaces (APIs) or other default functions to allow the web browser to present a user interface, facilitate user interaction with websites, etc., and establish communication between the client and the server.

[0074] The client 130 may include a standard internet browser capable of communicating with one or more of the web server 142 and / or the camera manager 120 to access the video data of the VMS 100. In contrast to previously proposed systems that rely on proprietary client software to be executed to communicate with the server for retrieving video data, the client 130 of the VMS 100 can use any standard web browser application to access the video data. A standard internet browser application means that the browser application may not require any plug-ins, add-ons, or other programs to be installed or executed by the browser application, except for functionality provided natively in the browser. It should be noted that while certain functionality regarding the user interface for searching, retrieving, and displaying videos may be delivered to the web browser as code, etc., by the web server 142, any such functionality may be provided without user interaction or preconfiguration of the web browser. Therefore, any such functionality is still considered native functionality of the web browser. In this regard, the client 130 can receive all necessary data from the web pages served by the VMS 100 to facilitate access to the video data of the VMS 100, without having to download programs, install plug-ins, or otherwise modify or configure the browser application from a native configuration. That is, all necessary information and / or instructions required to receive and display a user interface and / or video data from the VMS 100 may be provided locally with a standard browser or delivered from the VMS system 100 to allow execution of the client 130. Any suitable computing device capable of executing a standard web browser application that operatively communicates with the network 115 may be used as the client 130 to access the video data of the VMS 100. For example, any laptop computer, desktop computer, tablet computer, smartphone device, smart television, or another device capable of executing a standard internet browser application may function as the client 130.

[0075] Further references Figure 6, depicts an example of a VMS 100 providing video data to a client 130. In this case, a reverse proxy 200 can be utilized to facilitate communication with the client 130. Specifically, the reverse proxy 200 can be facilitated by the web server 142 of the master node 140, as described above. That is, the web server 142 can act as the reverse proxy 200. In this regard, the client 130 can connect to the reverse proxy 200. A user interface 400 comprising HTML or other web page content can be provided from the reverse proxy 200. For example, the user interface 400 provided by the reverse proxy 400 can include a list 404 or a searchable index of available video data from the cameras 110 of the VMS 100. This can include a list of available real-time video data feeds for real-time delivery to the client 130, or can allow access to stored video data. In the latter regard, a search function can be enabled to perform searches (e.g., using any video metadata, such as acquisition date / time, camera identification, facility location, and / or analytical metadata, such as objects identified from the video data). In this regard, the web server 142 can act as a signaling server to provide information about available video data. After selecting a given portion of the video data, a request for the specific video data can be issued from the client 130 to the reverse proxy 200. The reverse proxy 200 can then communicate with a given one of the camera nodes 120 to retrieve the requested video data. The user interface 400 may also include a video display 402. The video data can be requested from the appropriate camera node 120 by the web server 142, formatted in an appropriate transport mechanism, and delivered to the client 130 by the web server 142 acting as the reverse proxy 200, for decoding and displaying the video data in the video display 402. Thus, the use of the reverse proxy 200 allows all data delivered to the client 130 to be provided from a single server, which may have appropriate security certificates that meet many of the browser's security requirements.

[0076] In one example, the transmission mechanism for processing data by camera node 120 can be based at least in part on the characteristics of the request from client 130. In this regard, reverse proxy 200 can determine the characteristics of the request. Examples of such characteristics include the nature of the video data (e.g., real-time or archived video data), the identity of camera 110 capturing the video data, the network location of client 130 relative to reverse proxy 200 or the camera node 120 that will provide the video data, or other characteristics. Based on these characteristics, an appropriate encoding video format, container format, and communication protocol are selected for processing the video data by camera node 120. Camera node 120 can provide the video data to reverse proxy 200 for transmission to client 130. As described above, in at least some cases, the video data provided to client 130 can be real-time or near-real-time video data, which can be presented by client 130 in a standard web browser format without requiring the installation of plug-ins or other applications on client 130.

[0077] The user may wish to change the video data displayed in user interface 400. The user can then select a new video data source. In one embodiment, the transport mechanism can be configured such that new video data can be requested from the appropriate camera node 120 by network server 142 and delivered to user interface 400 without requiring a page reload. That is, the data in video display 402 can typically be changed without requiring a page reload. This can provide greater utility for users attempting to monitor multiple video data sources using a standard web browser.

[0078] The video data provided to the client 130 for presentation in the video display 402 may include metadata, such as analytics metadata. As described above, such analytics metadata may relate to any appropriate video analytics applied to the video data and may include, for example, highlighting detected objects, object recognition, individual recognition, object tracking, and the like. Thus, the video data may be annotated to include some analytics metadata. The analytics metadata may be embodied within the video data or provided via a separate data channel. In the example where the analytics metadata is provided via a separate channel, the client 130 may receive the analytics metadata and annotate the video data in the video display 402 when presented in the user interface 400. Furthermore, it will be appreciated that different types of data comprising the user interface 400 may be delivered to the client 130 using different transport mechanisms. For example, the above examples of transport mechanisms may be used to deliver video data for display in the video display 402. However, the user interface itself may communicate over a standard TCP / IP connection using HTML and the secure TLS security protocol. Furthermore, metadata (e.g., analytics metadata) may be provided as embedded data within the video data or provided as a separate data stream for presentation in the user interface 130, as described above. Where a separate data stream is used to deliver metadata, the metadata may be delivered via a different transport mechanism than the video data itself.

[0079] Return Reference Figure 5 , abstracting the functionality of the VMS 100 into various functional layers can also provide advantages related to the analysis of video data by the camera nodes 120. Specifically, the application of analytical models (e.g., machine learning modules) can be relatively computationally heavy for the camera nodes 120. Although the camera nodes 120 may be equipped with a graphics processing unit (GPU) or other specially adapted hardware to assist in performing the computational load, there may be certain instances where the processing power of a given camera node 120 may not be able to apply the analytical model to all video data from a given video camera 110. For example, in some cases, the video data from a given camera 110 can be advantageously divided into different portions of the data, which can be provided to different camera nodes 120 for separate processing of the different portions of the data. By "slicing" the data in this manner, analysis of different portions of the video data can be performed simultaneously at different ones of the camera nodes 120, which can increase the speed and / or throughput of performing analysis on the video data.

[0080] Therefore, if Figure 7As shown, camera 110 of VMS 100 can be in operable communication with network 115. At least first node 120a and second node 120b can also communicate with network 115 to receive video data from camera 110. First node 120a can include a first analysis model 210a, and second node 120b can include a second analysis model 210b. First analysis model 210a can be the same as or different from second analysis model 210b.

[0081] The video data from the video camera 110 can be divided into at least a first video portion 212 and a second video portion 214. Although referred to as video data portions, it should be understood that as little as a single frame of video data can include the respective portions 212 and 214 of video data. The first portion 212 of video data can be provided to the first camera node 120a, and the second portion 214 of video data can be provided to the second camera node 120b.

[0082] The second portion 214 of video data may be provided to the second camera node 120b in response to a trigger detected by any of the master node, camera node 120a, camera node 120b, or camera 110. The trigger may be based on any number of conditions or parameters. For example, a periodic trigger may be established such that the second portion 214 of video data is provided to the second camera node 120b in a periodic manner based on time, the amount of camera data, or other periodic trigger. In this regard, the first analysis model 210a may require relatively lower computational complexity relative to the second analysis model 210b. Therefore, providing all video data to the second camera node 120b for processing using the second analysis model 210b may not be computationally efficient. However, every Nth portion (e.g., comprising a fixed duration, a video size on disk, or a given number of frames) may be provided from camera 110 to the second camera node 210b, where N is a positive integer. In this regard, every hundredth of a second of video data may include the second portion 214 of video data, every thousandth of a frame of video data may include the second portion 214 of video data, and so on.

[0083] In another scenario, the second portion 214 of video data can be provided to the second camera node 120b based on system video metadata or analytical video metadata of the first portion 212 of video data. For example, after a given object is detected from the first portion 212 of video data, subsequent frames of video data including the second portion 214 of video data can be provided to the second camera node 120b. As an example of this operation, the first camera node 120a can detect a person from the first portion 212 of video data using the first analysis model 210a. Subsequently, the second portion 214 of video data can be directed to the second camera node 120b for processing by the second analysis model 210b, which can be particularly suitable for facial recognition. In this regard, video data from camera 110 can be directed to specific nodes for processing to allow for the application of different analysis models, etc.

[0084] refer to Figure 12 , illustrates example operations 1200 according to one aspect of the present disclosure. Operation 1200 may include a capture operation 1202, in which video data is captured at a plurality of video cameras. As described above, the video cameras may be in operative communication with a network. Subsequently, operation 1200 may also include a transmit operation 1204 to transmit the video data to the plurality of camera nodes. As described above, any one or more of the plurality of cameras may transmit 1204 their respective video data to any one or more of the camera nodes. Specifically, the cameras and the camera nodes may establish communication based on a first camera allocation configuration provided by a camera distributor of the system master node (e.g., to load balance video data processed across the camera nodes based on the allocation parameters).

[0085] Operation 1200 may include processing operation 1206 to process the video data received by each corresponding camera node. Specifically, as described above, in at least one example, processing operation 1206 may include encoding the video data into encoded video packets, packaging the encoded video packets into a transmission container, and selecting a communication protocol for transmitting the video packets. Furthermore, processing operation 1206 may include storing the video data in a storage device of the VMS.

[0086] The monitoring operation 1208 may monitor camera allocation parameters at one or more camera nodes. The monitoring operation 1208 may include, for example, receiving status information from one or more camera nodes, monitoring system resource usage at the camera nodes, and / or monitoring network traffic. In any aspect, the operation 1200 may also include a detection operation 1210, in which the camera allocator may detect a change in camera allocation parameters across one or more of the camera nodes. As described above, a change in camera allocation parameters at one or more camera nodes may be caused by any of a number of events, including, for example, a camera node becoming unavailable, a new camera node becoming available, an increase in computational load at a camera node, a decrease in computational load at a camera node, a change in the nature of video data received from a video camera at the camera node, a change in video analytics applied to the video data, and the like.

[0087] In response to the detection operation 1210, a modification operation 1212 may be performed to modify the first camera allocation configuration to a second camera allocation configuration that is different from the first camera allocation configuration. Specifically, the modification operation 1212 may be performed to balance the allocation parameters at the camera node in response to the change detected in the detection operation 1210. Furthermore, the process may be iterative, such that the monitoring operation 1208, the detection operation 1210, and the modification operation 1212 may be performed periodically during operation of the VMS system. For example, such monitoring may be performed at a frequency sufficient to detect any changes (e.g., unavailability of a camera node) without significantly interrupting the processing of video data from the cameras assigned to the camera node. For example, such monitoring may be performed at a rate of at least 1 Hz, although higher frequencies may be employed without limitation.

[0088] Figure 13 13. The processing device 1300 is a simplified example diagram of a processing device 1300 suitable for implementing various aspects of the disclosed technology. For example, the processing device 1300 can generally describe the architecture of the camera node 130, the master node 140, and / or the client 130. The processing device 1300 includes one or more processor units 1302, a memory 1304, a display 1306, and other interfaces 1308 (e.g., buttons). The memory 1304 typically includes both volatile memory (e.g., RAM) and non-volatile memory (e.g., flash memory). An operating system 1310, such as Microsoft An operating system, Apple macOS operating system, or Linux operating system, resides in memory 1304 and is executed by processor unit 1302, but it will be appreciated that other operating systems may be employed.

[0089] One or more applications 1312 are loaded into the memory 1304 and executed by the processor unit 1302 on the operating system 1310. The applications 1312 may receive input from various input local devices such as a microphone 1334, input accessories 1335 (e.g., a keypad, a mouse, a stylus, a touchpad, a joystick, an instrument mounted input, etc.). Additionally, the applications 1312 may be connected to a wired or wireless network (e.g., a mobile phone network, a wireless ... ) to communicate with such devices to receive input from one or more remote devices, such as remotely located smart devices. The processing device 1300 may also include various other components, such as a positioning system (e.g., a global positioning satellite transceiver), one or more accelerometers, one or more cameras, an audio interface (e.g., a microphone 1334, an audio amplifier and speaker, and / or an audio jack), and a storage device 1328. Other configurations may also be used.

[0090] Processing device 1300 also includes a power supply 1316, which is powered by one or more batteries or other power sources and provides power to the other components of processing device 1300. Power supply 1316 may also be connected to an external power source (not shown) that recharges or recharges the internal batteries or other power sources.

[0091] Example implementations may include hardware and / or software embodied by instructions stored in memory 1304 and / or storage 1328 and processed by processor unit 1302. Memory 1304 may be memory of a host device or an accessory coupled to a host.

[0092] The processing system 1300 may include various tangible processor-readable storage media and intangible processor-readable communication signals. Tangible processor-readable storage may be embodied via any available media accessible by the processing system 1300 and includes both volatile and non-volatile storage media, removable and non-removable storage media. Tangible processor-readable storage media do not include intangible communication signals and include volatile and non-volatile, removable and non-removable storage media implemented in any method or technology for storing information such as processor-readable instructions, data structures, program modules, or other data. Tangible processor-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CDROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other tangible medium that can be used to store the desired information and is accessible by the processing system 1300. In contrast to tangible processor-readable storage media, intangible processor-readable communication signals may embody processor-readable instructions, data structures, program modules, or other data residing in a modulated data signal such as a carrier wave or other signal transmission mechanism. The term "modulated data signal" means an intangible communication signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, intangible communication signals include signals propagated over wired media, such as a wired network or direct-wired connection, and wireless media, such as acoustic, RF, infrared, and other wireless media.

[0093] Some embodiments may include an article of manufacture. The article of manufacture may include a tangible storage medium for storing logic. Examples of storage media may include one or more types of processor-readable storage media capable of storing electronic data, including volatile or non-volatile memory, removable or non-removable memory, erasable or non-erasable memory, writable or rewritable memory, etc. Examples of logic may include various software elements, such as software components, programs, applications, computer programs, application programs, system programs, machine programs, operating system software, middleware, firmware, software modules, routines, subroutines, operation segments, methods, processes, software interfaces, application program interfaces (APIs), instruction sets, computing codes, computer codes, code segments, computer code segments, words, values, symbols, or any combination thereof. In one embodiment, for example, the article of manufacture may store executable computer program instructions that, when executed by a computer, cause the computer to perform the methods and / or operations according to the described embodiments. Executable computer program instructions may include any suitable type of code, such as source code, compiled code, interpreted code, executable code, static code, dynamic code, etc. Executable computer program instructions can be implemented according to a predefined computer language, manner or syntax to instruct a computer to perform a certain operation segment. Instructions can be implemented using any suitable high-level, low-level, object-oriented, visual, compiled and / or interpreted programming language.

[0094] One general aspect of the present disclosure includes a method for processing video data in a video surveillance system. The method includes capturing video data at a plurality of video cameras and transmitting the video data from the plurality of video cameras to at least one camera node via a communication network according to a first camera allocation configuration. The method also includes executing a video processing module at the camera node to process the video data received at the camera node and monitoring camera allocation parameters at the camera node. The camera allocation parameters are based at least in part on the video data received at the at least one camera node. The method further includes modifying the first camera allocation configuration to a second camera allocation configuration different from the first camera allocation configuration based at least in part on the camera allocation parameters.

[0095] Implementations may include one or more of the following features. For example, the method may further include operating the plurality of camera nodes such that, in a first camera allocation configuration, each of the plurality of camera nodes is operable to receive video data from a corresponding subset of the plurality of cameras. The method may further include detecting a change in a plurality of available camera nodes in the video surveillance system. A second camera allocation configuration may include allocating the plurality of cameras to the available camera nodes after detecting the change in the plurality of available camera nodes.

[0096] In one example, the modification operation is in response to a computational capacity of the camera node exceeding a threshold. This can be in response to an increased computational load associated with video analytics performed on video data by the camera node via the video processing module. In one example, the second camera allocation includes reassigning a video camera from the plurality of cameras to another camera node having video analytics capabilities for the camera node to which the video camera was reassigned.

[0097] In one example, the camera assignment parameters include a field of view of each of the plurality of cameras. The second camera assignment may include disassociating the camera node from a video camera that shares at least a partially overlapping field of view with another video camera associated with the camera node.

[0098] Another general aspect of the present disclosure includes a video surveillance system. The system includes a plurality of video cameras operable to capture video data; and at least one camera node operable to receive video data from one or more of the plurality of video cameras according to a first camera allocation configuration. The system also includes a video processing module executed at the at least one camera node to process the video data received at the at least one camera node. The system also includes a master node in operable communication with the at least one camera node to monitor camera allocation parameters based at least in part on the video data received at the at least one camera node, and in response to a change in the camera allocation parameters, modify the first camera allocation configuration to a second camera allocation configuration different from the first camera allocation configuration.

[0099] Implementations may include one or more of the following features. In one example, the system may include a plurality of camera nodes. In a first camera allocation configuration, each of the plurality of camera nodes is operable to receive video data from a respective subset of the plurality of video cameras. The master node is operable to detect a change in a plurality of available camera nodes in the video surveillance system, and a second camera allocation configuration includes allocating the plurality of video cameras to the available camera nodes upon detecting the change in the plurality of available camera nodes.

[0100] In one example, the master node may modify the first camera allocation configuration to a second camera allocation configuration in response to a computational capability of the camera node exceeding a threshold value in response to an increased computational load associated with video analytics performed by the camera node on video data via a video processing module. In one example, the second camera allocation may include reassigning a video camera from the plurality of cameras to another camera node having video analytics capabilities for the camera node to which the video camera was reassigned.

[0101] In one example, the camera assignment parameters may include a field of view for each of the plurality of cameras. The second camera assignment may include disassociating the camera node from a video camera that shares at least a partially overlapping field of view with another video camera associated with the camera node.

[0102] Another general aspect of the present disclosure includes one or more tangible processor-readable storage media embodied with instructions for executing, on one or more processors and circuits of a device, a process for processing video data in a video surveillance system. The process may include capturing video data at a plurality of video cameras and transmitting the video data from the plurality of video cameras to at least one camera node via a communication network according to a first camera allocation configuration. The process also includes executing a video processing module at the camera node to process the video data received at the camera node and monitoring camera allocation parameters at the camera node. The camera allocation parameters are based at least in part on the video data received at the at least one camera node. The process also includes modifying the first camera allocation configuration to a second camera allocation configuration different from the first camera allocation configuration based at least in part on the camera allocation parameters.

[0103] Implementations may include one or more of the following features. In one example, the process may include operating a plurality of camera nodes. In a first camera allocation configuration, each of the plurality of camera nodes is operable to receive video data from a respective subset of the plurality of video cameras. The process may also include detecting a change in a plurality of available camera nodes in the video surveillance system. A second camera allocation configuration may include allocating the plurality of cameras to the available camera nodes after detecting the change in the plurality of available camera nodes.

[0104] In one example, the modification can be in response to a computational capability of the camera node exceeding a threshold, the computational capability of the camera node exceeding the threshold being in response to an increased computational load associated with video analytics being performed on video data by the camera node via a video processing module. The second camera assignment can include reassigning a video camera from the plurality of cameras to another camera node that has video analytics capabilities for the camera node to which the video camera was reassigned. For example, the camera assignment parameters can include a field of view for each of the plurality of cameras. The second camera assignment can include disassociating the camera node from a video camera that shares at least a partially overlapping field of view with another video camera associated with the camera node.

[0105] The embodiments described herein are implemented as logical steps in one or more computer systems. Logical operations can be implemented (1) as a sequence of processor-implemented steps executed in one or more computer systems and (2) as interconnected machine or circuit modules within one or more computer systems. The implementation is a matter of choice, depending on the performance requirements of the computer system utilized. Therefore, the logical operations that make up the embodiments described herein are variously referred to as operations, steps, objects, or modules. Furthermore, it should be understood that unless otherwise explicitly claimed or the claim language inherently requires a specific order, the logical operations can be performed in any order.

[0106] Although the present invention has been shown and described in detail in the drawings and the foregoing description, such illustration and description should be regarded as illustrative and not restrictive. For example, certain embodiments described above may be combined with other described embodiments and / or arranged in other ways (e.g., process elements may be performed in other sequences). Therefore, it should be understood that only preferred embodiments and variations thereof have been shown and described, and that protection is intended for all changes and modifications that come within the spirit of the invention.

[0107] Further examples

[0108] Example 1. A method for processing video data in a video surveillance system, comprising:

[0109] capturing video data at a plurality of video cameras;

[0110] transmitting the video data from the plurality of video cameras to at least one camera node via a communication network according to a first camera allocation configuration;

[0111] executing a video processing module at a camera node to process the video data received at the camera node;

[0112] monitoring camera allocation parameters at the camera nodes, wherein the camera allocation parameters are based at least in part on the video data received at the at least one camera node; and

[0113] The first camera allocation configuration is modified to a second camera allocation configuration different from the first camera allocation configuration based at least in part on the camera allocation parameters.

[0114] Example 2. The method of Example 1, further comprising:

[0115] operating a plurality of camera nodes, wherein in the first camera allocation configuration, each camera node of the plurality of camera nodes is operable to receive video data from a respective subset of the plurality of video cameras; and

[0116] detecting a change in a plurality of available camera nodes in the video surveillance system; and

[0117] The second camera allocation configuration includes allocating the multiple video cameras to the available camera nodes after the change of the multiple available camera nodes is detected.

[0118] Example 3. A method as described in Example 1, wherein the modification is in response to the computing power of the camera node exceeding a threshold, and the computing power of the camera node exceeding the threshold is in response to an increased computing load associated with video analysis performed by the camera node on the video data through the video processing module.

[0119] Example 4. The method of Example 3, wherein the second camera allocation comprises reallocating a video camera from the plurality of cameras to another camera node having video analytics capabilities for the camera node to which the video camera was reallocated.

[0120] Example 5. The method of Example 1, wherein the camera allocation parameters include a field of view of each of the plurality of cameras.

[0121] Example 6. The method of Example 5, wherein the second camera assignment comprises disassociating the camera node from a video camera that shares at least a partially overlapping field of view with another video camera associated with the camera node.

[0122] Example 7. A video surveillance system comprising:

[0123] a plurality of video cameras operable to capture video data;

[0124] at least one camera node operable to receive video data from one or more of the plurality of video cameras according to a first camera allocation configuration;

[0125] a video processing module executed at the at least one camera node to process the video data received at the at least one camera node;

[0126] a master node in operable communication with the at least one camera node to monitor a camera allocation parameter based at least in part on the video data received at the at least one camera node, and in response to a change in the camera allocation parameter, modify the first camera allocation configuration to a second camera allocation configuration different from the first camera allocation configuration.

[0127] Example 8. The system of Example 7, further comprising:

[0128] a plurality of camera nodes, wherein in the first camera allocation configuration, each camera node of the plurality of camera nodes is operable to receive video data from a respective subset of the plurality of video cameras; and

[0129] The master node is operable to detect changes in the plurality of available camera nodes in the video surveillance system, and the second camera allocation configuration includes allocating the plurality of video cameras to the available camera nodes after detecting changes in the plurality of available camera nodes.

[0130] Example 9. A system as described in Example 7, wherein the master node modifies the first camera allocation configuration to the second camera allocation configuration in response to the computing power of the camera node exceeding a threshold, and the computing power of the camera node exceeds the threshold in response to an increased computing load associated with video analysis performed by the camera node on the video data through the video processing module.

[0131] Example 10. The system of Example 9, wherein the second camera assignment comprises reassigning a video camera from the plurality of cameras to another camera node having video analytics capabilities for the camera node to which the video camera was reassigned.

[0132] Example 11. The system of Example 7, wherein the camera allocation parameters include a field of view of each of the plurality of cameras.

[0133] Example 12. The system of Example 11, wherein the second camera assignment comprises disassociating the camera node from a video camera that shares at least a partially overlapping field of view with another video camera associated with the camera node.

[0134] Example 13. One or more tangible processor-readable storage media embodied with instructions for executing, on one or more processors and circuits of a device, a process for processing video material in a video surveillance system, the process comprising:

[0135] capturing video data at a plurality of video cameras;

[0136] transmitting the video data from the plurality of video cameras to at least one camera node via a communication network according to a first camera allocation configuration;

[0137] executing a video processing module at a camera node to process the video data received at the camera node;

[0138] monitoring camera allocation parameters at the camera nodes, wherein the camera allocation parameters are based at least in part on the video data received at the at least one camera node; and

[0139] The first camera allocation configuration is modified to a second camera allocation configuration different from the first camera allocation configuration based at least in part on the camera allocation parameters.

[0140] Example 14. The one or more tangible processor-readable storage media of Example 13, wherein the process further comprises:

[0141] operating a plurality of camera nodes, wherein in the first camera allocation configuration, each camera node of the plurality of camera nodes is operable to receive video data from a respective subset of the plurality of video cameras; and

[0142] detecting a change in a plurality of available camera nodes in the video surveillance system; and

[0143] The second camera allocation configuration includes allocating the multiple video cameras to the available camera nodes after the change of the multiple available camera nodes is detected.

[0144] Example 15. One or more tangible processor-readable storage media as described in Example 13, wherein the modification is in response to the computing power of the camera node exceeding a threshold, and the computing power of the camera node exceeding the threshold is in response to an increased computational load associated with video analysis performed by the camera node on the video data through the video processing module.

[0145] Example 16. One or more tangible processor-readable storage media as described in Example 15, wherein the second camera allocation includes reallocating a video camera from the plurality of cameras to another camera node having video analysis capabilities for the camera node to which the video camera was reallocated.

[0146] Example 17. The one or more tangible processor-readable storage media of Example 13, wherein the camera allocation parameters include a field of view of each camera of the plurality of cameras.

[0147] Example 18. One or more tangible processor-readable storage media as described in Example 17, wherein the second camera assignment includes disassociating the camera node from a video camera that shares at least partially overlapping fields of view with another video camera associated with the camera node.

Claims

1. A method for processing video data in a video surveillance system, comprising: capturing video data at a plurality of video cameras; transmitting the video data from the plurality of video cameras to a plurality of camera nodes via a communication network according to a first camera allocation configuration; executing a video processing module at the plurality of camera nodes to process the video data received at the plurality of camera nodes; monitoring camera allocation parameters at the plurality of camera nodes, wherein the camera allocation parameters are based at least in part on the video data received at the plurality of camera nodes; as well as In response to a computing capability of a given camera node among the plurality of camera nodes exceeding a threshold, the first camera allocation configuration is modified to a second camera allocation configuration different from the first camera allocation configuration, wherein the second camera allocation provides load balancing of allocation parameters across all available camera nodes, the all available camera nodes including the given camera node and at least one other camera node among the plurality of camera nodes.

2. The method of claim 1, further comprising: operating the plurality of camera nodes, wherein in the first camera allocation configuration, each camera node of the plurality of camera nodes is operable to receive video data from a respective subset of the plurality of video cameras; as well as detecting changes in a plurality of available camera nodes in the video surveillance system; as well as The second camera allocation configuration includes allocating the multiple video cameras to the available camera nodes after the change of the multiple available camera nodes is detected.

3. The method according to claim 1, wherein The computing capacity of the given camera node exceeds a threshold in response to an increased computational load associated with video analytics performed by the given camera node on the video data via the video processing module.

4. The method of claim 3, wherein the second camera assignment comprises reallocating a video camera of the plurality of video cameras to another camera node having video analytics capabilities for the given camera node to which the video camera was reallocated. The method of claim 1 , wherein the camera allocation parameters include a field of view of each of the plurality of video cameras.

6. The method of claim 5, wherein the second camera assignment comprises disassociating the camera node from a video camera that shares at least a partially overlapping field of view with another video camera associated with the camera node.

7. A video surveillance system comprising: a plurality of video cameras operable to capture video data; a plurality of camera nodes operable to receive video data from one or more video cameras of the plurality of video cameras according to a first camera allocation configuration; a video processing module executed at the plurality of camera nodes to process the video data received at the plurality of camera nodes; a master node in operable communication with the plurality of camera nodes to monitor camera allocation parameters based at least in part on the video data received at the plurality of camera nodes, and in response to a change in the camera allocation parameters due to a computing capability of a given camera node among the plurality of camera nodes exceeding a threshold, modifying the first camera allocation configuration to a second camera allocation configuration different from the first camera allocation configuration, wherein the second camera allocation provides load balancing of the allocation parameters across all available camera nodes, the all available camera nodes including the given camera node and at least one other camera node among the plurality of camera nodes.

8. The system of claim 7, further comprising: wherein in the first camera allocation configuration, each camera node of the plurality of camera nodes is operable to receive video data from a respective subset of the plurality of video cameras; and The master node is operable to detect changes in the plurality of available camera nodes in the video surveillance system, and the second camera allocation configuration includes allocating the plurality of video cameras to the available camera nodes after detecting changes in the plurality of available camera nodes.

9. The system of claim 7, wherein: The computing capacity of the given camera node exceeds a threshold in response to an increased computational load associated with video analytics performed by the given camera node on the video data via the video processing module.

10. The system of claim 9, wherein the second camera assignment comprises reassigning a video camera of the plurality of video cameras to another camera node having video analytics capabilities for the given camera node to which the video camera was reassigned.

11. The system of claim 7, wherein the camera assignment parameters include a field of view of each of the plurality of video cameras.

12. The system of claim 11, wherein the second camera assignment comprises disassociating the camera node from a video camera that shares at least a partially overlapping field of view with another video camera associated with the camera node.

13. One or more tangible processor-readable storage media embodied with instructions for executing on one or more processors and circuits of a device a process for processing video material in a video surveillance system, the process comprising: capturing video data at a plurality of video cameras; transmitting the video data from the plurality of video cameras to a plurality of camera nodes via a communication network according to a first camera allocation configuration; executing a video processing module at the plurality of camera nodes to process the video data received at the plurality of camera nodes; monitoring camera allocation parameters at the plurality of camera nodes, wherein the camera allocation parameters are based at least in part on the video data received at the plurality of camera nodes; as well as In response to a computing capability of a given camera node among the plurality of camera nodes exceeding a threshold, the first camera allocation configuration is modified to a second camera allocation configuration different from the first camera allocation configuration, wherein the second camera allocation provides load balancing of allocation parameters across all available camera nodes, the all available camera nodes including the given camera node and at least one other camera node among the plurality of camera nodes.

14. The one or more tangible processor-readable storage media of claim 13, the process further comprising: operating a plurality of camera nodes, wherein in the first camera allocation configuration, each camera node of the plurality of camera nodes is operable to receive video data from a respective subset of the plurality of video cameras; as well as detecting changes in a plurality of available camera nodes in the video surveillance system; as well as The second camera allocation configuration includes allocating the multiple video cameras to the available camera nodes after the change of the multiple available camera nodes is detected.

15. One or more tangible processor-readable storage media as recited in claim 13, wherein: The computing capacity of the given camera node exceeds a threshold in response to an increased computational load associated with video analytics performed by the given camera node on the video data via the video processing module.

16. One or more tangible processor-readable storage media as described in claim 15, wherein the second camera assignment includes reassigning a video camera of the plurality of video cameras to another camera node having video analytics capabilities for the given camera node to which the video camera was reassigned.

17. The one or more tangible processor-readable storage media of claim 13, wherein the camera allocation parameters include a field of view of each of the plurality of video cameras.

18. One or more tangible processor-readable storage media as described in claim 17, wherein the second camera assignment includes disassociating the camera node from a video camera that shares at least partially overlapping fields of view with another video camera associated with the camera node.

Images