Reasoning resource allocation optimization method and system for multi-camera real-time video monitoring

Abstract

The invention discloses a reasoning resource allocation optimization method and system for multi-camera real-time video monitoring, and relates to the field of multi-camera real-time monitoring. According to the invention, through reasonable reasoning of resource allocation and combination of a video frame rate dynamic adjustment strategy, optimization processing of video streams of a plurality of cameras is realized. Particularly, in the method, different reasoning resources are allocated according to the priorities of the detection tasks of different cameras, so that the cameras with abnormal event detection requirements obtain more computing resources, and the cameras without abnormal events use fewer resources, thereby achieving the balance between the optimal performance and the computing resources.

Description

Technical Field

[0001] The present invention relates to the field of multi-camera real-time monitoring, and more specifically to a method and system for optimizing inference resource allocation for multi-camera real-time video monitoring. This method, applied in multi-camera real-time monitoring systems, is particularly suitable for optimizing resource allocation for video stream processing, target detection, and event response when GPU / NPU computing resources are limited. This method has broad applications in public security monitoring, industrial site monitoring, traffic management, and other fields. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and may not constitute prior art.

[0003] Prior art research on edge computing-based video surveillance systems and surveillance systems for resource-constrained environments primarily focuses on optimizing target detection algorithms, improving behavior recognition algorithms, and implementing inference resource allocation strategies for cloud-edge collaboration. While existing methods have made some progress in optimizing computing resources in single-camera scenarios, no effective solutions have been publicly reported for multi-camera collaborative scheduling scenarios, particularly regarding the technical challenge of dynamically adjusting computing resource allocation across multiple cameras based on detection results after an abnormal event is triggered. Current systems generally lack a coordinated mechanism between abnormal event response and dynamic computing resource scheduling, resulting in suboptimal overall resource utilization in multi-camera systems. Summary of the Invention

[0004] The purpose of the present invention is to address the problems existing in the prior art and provide an inference resource allocation optimization method and system for multi-camera real-time video surveillance. The method utilizes a GPU / NPU-based edge intelligent computing platform, combined with target detection algorithms (such as the YOLO series), to achieve efficient processing and real-time response of multi-camera video streams in the detection scene, ensuring that abnormal events (such as fires and intrusions) can be efficiently detected under limited computing power, and improving the overall efficiency of the system through a reasonable resource allocation mechanism.

[0005] The technical solutions of the present invention are as follows:

[0006] A method for optimizing inference resource allocation for multi-camera real-time video surveillance, comprising:

[0007] Step S1: Obtain video stream data sent by multiple cameras, each video stream containing serialized image frames;

[0008] Step S2: sampling the video stream sent by each camera through a dynamically adjusted decimator, wherein the decimator extracts image frames from the video stream according to the allocated sampling interval;

[0009] Step S3: sending the extracted image frames to the GPU / NPU inference card for inference calculation based on the target detection algorithm, wherein the target detection algorithm is used to identify abnormal events;

[0010] Step S4: Analyze the inference results through the anomaly identifier to determine whether there is an abnormal event and the camera number of the abnormal event source;

[0011] Step S5: Dynamically adjust the sampling interval of the corresponding camera according to the judgment result of step S4.

[0012] Furthermore, the video stream data collected by each camera is sampled by the extractor, and the obtained image frames are pre-processed and placed in the image buffer queue.

[0013] Furthermore, the total number of images sent to the image buffer queue by the extractor is F per second. input Zhang, GPU / NPU inference card can infer detection F per second ai images, F input =F ai .

[0014] Furthermore, the step S3 includes:

[0015] Step S31: the GPU / NPU inference card reads the image from the image buffer queue;

[0016] Step S32: Calling the target detection algorithm to perform inference calculation based on the underlying GPU / NPU hardware resources;

[0017] Step S33: Send the inference calculation result to the inference result queue.

[0018] Furthermore, the step S4 includes:

[0019] Step S41: performing a dequeue operation in the inference result queue, i.e., taking out the inference result of the image from the inference result queue;

[0020] Step S42: Determine whether an abnormal event exists through the abnormality identifier, perform identification analysis on the inference result, and determine whether an abnormal event is detected.

[0021] Furthermore, the step S5 includes:

[0022] If there is an abnormal event, the abnormality identifier needs to further determine which cameras the abnormal event comes from, and then enter the resource adjustment link to reduce the sampling interval of the extractor corresponding to the camera that detected the abnormal event, and increase the sampling interval of the extractors corresponding to the other cameras;

[0023] If there are no abnormal events, skip the resource adjustment step and directly record and output the inference results;

[0024] Regardless of whether an abnormal event is detected, the final result will be output and recorded to facilitate subsequent processing or alarm notification.

[0025] Furthermore, dynamically adjusting the sampling interval of the corresponding camera according to the judgment result of step S4 includes:

[0026] Reduce the sampling interval used by cameras that detect an anomaly, and increase the sampling interval used by cameras that do not detect an anomaly.

[0027] Furthermore, when all cameras do not detect abnormal events or all detect abnormal events, the sampling interval T of each camera is ζ Expressed as:

[0028]

[0029] Where: N represents the total number of cameras, and C represents the total computing resources.

[0030] Furthermore, when some cameras detect abnormal events, resource allocation is dynamically adjusted;

[0031] Assume that m cameras detect abnormal events, and the remaining Nm cameras do not detect abnormal events;

[0032] Assume that the priority of each camera that detects an abnormal event is P i , the priority of ordinary cameras is P j =1, where P i >1;

[0033] The total computing resources C are allocated to m abnormal event cameras and Nm normal cameras;

[0034] The sampling interval of camera i that detects an abnormal event and camera j that does not detect an abnormal event can be expressed as:

[0035] For camera i that detects an abnormal event:

[0036]

[0037] For camera j that does not detect any abnormal events:

[0038]

[0039] Where: R i and R j They represent the proportion of computing resources obtained by cameras that detect abnormal events and cameras that do not detect abnormal events.

[0040] The present invention also proposes an inference resource allocation optimization system for multi-camera real-time video monitoring, comprising:

[0041] The pre-processing module includes multiple extractors, pre-processing units and image buffer queues corresponding to the number of cameras;

[0042] Inference module, including GPU / NPU inference card, equipped with GPU / NPU accelerated object detection algorithm;

[0043] The post-processing module includes an anomaly discriminator, a resource allocator, an output recording unit, and an inference result queue;

[0044] The resource allocator dynamically adjusts the sampling interval of the corresponding camera according to the judgment result of the anomaly identifier; the sampling interval used by the camera that detects an abnormal event is reduced, and the sampling interval used by the camera that does not detect an abnormality is increased.

[0045] Compared with the existing technology, the beneficial effects of the present invention are:

[0046] 1. This invention optimizes the processing of video streams from multiple cameras through rational inference resource allocation, combined with a dynamic video frame rate adjustment strategy. Specifically, by allocating different inference resources based on the detection task priorities of different cameras, cameras that need to detect abnormal events (such as fires and intrusions) receive more computing resources, while cameras without abnormal events use fewer resources, thus achieving an optimal balance between performance and computing resources.

[0047] 2. The inference resource allocation strategy proposed in the present invention can dynamically adjust resource allocation according to the detection task priority of each camera. In particular, when GPU / NPU inference resources are limited, more computing resources can be allocated to cameras that detect abnormal events (such as fire, intrusion, etc.). By reducing the inference frame rate of non-critical tasks (such as routine monitoring), the allocation of computing resources is optimized, thereby improving the detection efficiency and response speed of abnormal events under limited resources. Compared with traditional video surveillance systems, the dynamic allocation mechanism of the present invention significantly improves resource utilization and ensures the real-time processing capability of high-priority tasks.

[0048] 3. The present invention combines the video stream sequence number and the dynamic frame sampling strategy to ensure that in a multi-camera environment, important cameras (such as fire monitoring cameras) can obtain more image data when abnormal events are detected, thereby obtaining more computing resources for reasoning. For ordinary cameras, their computing load is reduced by increasing the sampling interval. Compared with the prior art, the intelligent feedback mechanism introduced in the present invention allocates computing resources more accurately, which not only improves the accuracy of detection, but also avoids excessive consumption of system resources, ensuring the efficient operation of the entire system. This resource allocation method based on event feedback has significant advantages in resource-constrained environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is an inference resource allocation optimization method and system schematic for multi-camera real-time video surveillance;

[0050] Figure 2 This is the processing flow chart of the post-processing module. DETAILED DESCRIPTION

[0051] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

[0052] The features and performance of the present invention are further described in detail below with reference to the embodiments.

[0053] Example 1

[0054] First, to facilitate understanding of the technical solution of the present invention, the following glossary is provided:

[0055] A video stream can be understood as a data frame that is constantly sending image data. This data stream is constantly sending image data packets to the receiving port. Each data packet contains image data and a sequence number (which can be understood as the frame number).

[0056] The decimator detects the sequence number (frame number) in the data packets sent by the video stream and extracts image data based on a sampling interval. If the corresponding camera detects an abnormal event, the sampling interval of its sampler will be shortened, and the camera will extract more image data. This will increase the amount of image data stored in the image buffer queue, and it will be allocated more computing resources.

[0057] It should also be noted that the implementation of this embodiment requires resource constraints. Specifically, the number of image frames transmitted by the multi-camera video stream being tested must be greater than the number of inference frames supported by the GPU / NPU. This means that the inference capability is insufficient to perform inference detection on all the image data from all cameras. For example, there are three cameras, each sending image data to the monitoring system using a video streaming protocol (such as RTSP) that streams 30 images per second (a total of 90 images per second), while the GPU or NPU inference card in the inference component only supports detecting 60 images per second. This allows the feedback mechanism of this embodiment to take effect. If a camera detects an anomaly (a fire), 30 images are allocated to that camera, while 15 images are allocated to each of the other two cameras. This means that the sampling interval for the camera that detected the danger will be reduced, while the sampling interval for the other two cameras will be increased. This allocation strategy tilts inference resources toward the camera that detected the danger while remaining within the image detection capacity limit of the inference card, thus optimizing the allocation of computing resources.

[0058] See also Figure 1 This embodiment proposes an inference resource allocation optimization system for multi-camera real-time video monitoring, including:

[0059] The pre-processing module includes multiple extractors, pre-processing units and image buffer queues corresponding to the number of cameras;

[0060] Inference module, including GPU / NPU inference card, equipped with GPU / NPU accelerated object detection algorithm;

[0061] The post-processing module includes an anomaly discriminator, a resource allocator, an output recording unit, and an inference result queue;

[0062] The resource allocator dynamically adjusts the sampling interval of the corresponding camera according to the judgment result of the anomaly identifier; the sampling interval used by the camera that detects an abnormal event is reduced, and the sampling interval used by the camera that does not detect an abnormality is increased.

[0063] Based on the above system, this embodiment also proposes an inference resource allocation optimization method for multi-camera real-time video monitoring, which specifically includes the following steps:

[0064] Step S1: Obtain the video stream data sent by multiple cameras, where each video stream contains serialized image frames;

[0065] Step S2: Sample the video streams sent by each camera through a dynamically adjusted extractor. The extractor extracts image frames from the video stream according to the assigned sampling interval;

[0066] Step S3: Send the extracted image frames into the GPU / NPU inference card for inference calculation based on the object detection algorithm, and the object detection algorithm is used to identify abnormal events;

[0067] Step S4: Analyze the inference results through an anomaly discriminator to determine whether there are abnormal events and the camera numbers of the sources of the abnormal events;

[0068] Step S5: Dynamically adjust the sampling interval of the corresponding camera according to the judgment result of Step S4.

[0069] In this embodiment, specifically, after the video stream data collected by each camera is sampled by the extractor, the obtained image frames are preprocessed and then put into the image buffer queue.

[0070] In this embodiment, specifically, the total number of image frames sent by the extractor to the image buffer queue is F input frames per second, and the GPU / NPU inference card can infer and detect F ai frames of images per second, and F input = F ai ;

[0071] That is, the applicable scenario of the present invention is a multi-camera real-time video monitoring system with limited GPU / NPU inference resources, that is, an inference system composed of n (n≥2) cameras with a frame rate of F and a GPU / NPU inference card with limited resources. Among them, limited inference resources mean that the GPU / NPU inference card can only infer and detect F ai frames of images per second, which is less than the nF frames of images sent by n cameras to the monitoring system per second, that is, F ai < nF. This is to make full use of the inference resources of the GPU / NPU inference card and avoid queue congestion in the image buffer queue.

[0072] In this embodiment, specifically, Step S3 includes:

[0073] Step S31: The GPU / NPU inference card reads images from the image buffer queue;

[0074] Step S32: Invoke the object detection algorithm (such as the YOLO series) based on the underlying GPU / NPU hardware resources for inference calculation;

[0075] Step S33: Send the inference calculation result to the inference result queue.

[0076] In this embodiment, specifically, Figure 2 As shown, the step S4 includes:

[0077] Step S41: performing a dequeue operation in the inference result queue, i.e., taking out the inference result of the image from the inference result queue;

[0078] Step S42: Determine whether an abnormal event (such as fire, intrusion, etc.) exists through the abnormality identifier, perform identification analysis on the inference result, and determine whether an abnormal event is detected.

[0079] In this embodiment, specifically, step S5 includes:

[0080] If there is an abnormal event, the abnormality identifier needs to further determine which cameras the abnormal event comes from, and then enter the resource adjustment link to reduce the sampling interval of the extractor corresponding to the camera that detected the abnormal event, and increase the sampling interval of the extractors corresponding to the other cameras;

[0081] If there are no abnormal events, skip the resource adjustment step and directly record and output the inference results;

[0082] Regardless of whether an abnormal event is detected, the final result will be output and recorded to facilitate subsequent processing or alarm notification.

[0083] In this embodiment, specifically, dynamically adjusting the sampling interval of the corresponding camera according to the judgment result of step S4 includes:

[0084] Reduce the sampling interval used by cameras that detect an anomaly, and increase the sampling interval used by cameras that do not detect an anomaly.

[0085] In this embodiment, it should be noted that the resource allocator can automatically allocate computing resources (control the sampler's frame interval) to each camera based on parameters such as the system's computing resources, the number of cameras, and the source of the abnormal event (which camera it comes from). There are three main cases, and the corresponding mathematical models are as follows:

[0086] 1. All cameras detected no abnormal events:

[0087] In this case, the priority of all cameras is P i =1, that is, their priority is the standard value. The system will evenly distribute the computing resources (the maximum number of images that the GPU / NPU inference card can infer and detect per second) C to all N cameras, and the computing resource ratio R obtained by each camera i for:

[0088]

[0089] Therefore, the sampling interval T of each camera is ζ It can be expressed as:

[0090]

[0091] In this case, the sampling interval for all cameras is the same and is This ensures balanced distribution of system resources; T0 represents the default sampling interval of the sampler, that is, the standard sampling interval when there are no abnormal events.

[0092] 2. All cameras detect abnormal events:

[0093] When all cameras detect an abnormal event, the system will increase the priority P for each camera. i >1, usually P i The increase in is the same because each camera faces the same abnormal event.

[0094] Assuming that the priority of each camera is increased by the same multiple P, the resource allocation ratio of each camera will become:

[0095]

[0096] At this time, the sampling interval T of each camera ζ will be:

[0097]

[0098] In this case, since each camera has the same priority, the system will still distribute resources evenly, the same as the first case.

[0099] 3. Some cameras detect abnormal events:

[0100] When only some cameras detect abnormal events, the system needs to dynamically adjust resource allocation to prioritize the response capabilities of these cameras. Suppose there are m cameras that detect abnormal events, and the remaining Nm cameras do not detect abnormal events. For the camera that detects abnormal events, its priority P i >1, for normal cameras, its priority is P j =1.

[0101] 1) Resource allocation:

[0102] For cameras that detect abnormal events, they will get more computing resources, with priority P i >1, so the resource allocation ratio R i Will increase.

[0103] For cameras that do not detect abnormal events, the resource ratio R j will decrease.

[0104] 2) Dynamic resource allocation:

[0105] Assume that the priority of each camera for abnormal event detection is P i >1, the priority of ordinary cameras is P j =1;

[0106] The total computing resources C of the system are allocated to m abnormal event cameras and Nm normal cameras.

[0107] To ensure that high-priority cameras get more computing resources, the following allocation strategy can be used:

[0108] For abnormal event cameras (priority P i ):

[0109]

[0110] For ordinary cameras (priority P j =1):

[0111]

[0112] Then, the sampling interval of the camera that detects an abnormal event and the camera that does not detect an abnormal event (ordinary camera) can be expressed as:

[0113] For abnormal event cameras:

[0114]

[0115] For ordinary cameras:

[0116]

[0117] That is, when all cameras do not detect abnormal events or detect abnormal events, the sampling interval of each camera is balanced and is

[0118] When some cameras detect abnormal events, the cameras with abnormal events will obtain more resources, resulting in a decrease in their sampling intervals and an increase in the sampling intervals of ordinary cameras.

[0119] The above-described embodiments merely represent specific implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of protection of the present application. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the technical concept of the present application, and all such variations and improvements fall within the scope of protection of the present application.

[0120] This background section is provided to generally present the context of the invention, and the work of the presently named inventors, the work to the extent described in this background section, and aspects of the description in this section that did not constitute prior art at the time of filing are neither explicitly nor implicitly admitted to be prior art to the present invention.

Claims

1. A method for optimizing inference resource allocation for multi-camera real-time video monitoring, characterized in that: include: Step S1: Obtain video stream data sent by multiple cameras, each video stream containing serialized image frames; Step S2: sampling the video stream sent by each camera through a dynamically adjusted decimator, wherein the decimator extracts image frames from the video stream according to the allocated sampling interval; Step S3: sending the extracted image frames to the GPU / NPU inference card for inference calculation based on the target detection algorithm, wherein the target detection algorithm is used to identify abnormal events; Step S4: Analyze the inference results through the anomaly identifier to determine whether there is an abnormal event and the camera number of the abnormal event source; Step S5: Dynamically adjust the sampling interval of the corresponding camera according to the judgment result of step S4.

2. The inference resource allocation optimization method for multi-camera real-time video monitoring according to claim 1 is characterized in that: The video stream data collected by each camera is sampled by the extractor, and the obtained image frames are pre-processed and placed in the image buffer queue.

3. The inference resource allocation optimization method for multi-camera real-time video monitoring according to claim 2 is characterized in that: The total number of images sent to the image buffer queue by the extractor is F per second input Zhang, GPU / NPU inference card can infer detection F per second ai images, F input =F ai .

4. The method for optimizing inference resource allocation for multi-camera real-time video monitoring according to claim 2, wherein: The step S3 comprises: Step S31: the GPU / NPU inference card reads the image from the image buffer queue; Step S32: Calling the target detection algorithm to perform inference calculation based on the underlying GPU / NPU hardware resources; Step S33: Send the inference calculation result to the inference result queue.

5. The inference resource allocation optimization method for multi-camera real-time video monitoring according to claim 4 is characterized in that: The step S4 comprises: Step S41: performing a dequeue operation in the inference result queue, i.e., taking out the inference result of the image from the inference result queue; Step S42: Determine whether an abnormal event exists through the abnormality identifier, perform identification analysis on the inference result, and determine whether an abnormal event is detected.

6. The method for optimizing inference resource allocation for multi-camera real-time video monitoring according to claim 5, characterized in that: The step S5 comprises: If there is an abnormal event, the abnormality identifier needs to further determine which cameras the abnormal event comes from, and then enter the resource adjustment link to reduce the sampling interval of the extractor corresponding to the camera that detected the abnormal event, and increase the sampling interval of the extractors corresponding to the other cameras; If there are no abnormal events, skip the resource adjustment step and directly record and output the inference results; Regardless of whether an abnormal event is detected, the final result will be output and recorded to facilitate subsequent processing or alarm notification.

7. The inference resource allocation optimization method for multi-camera real-time video monitoring according to claim 1 is characterized in that: Dynamically adjusting the sampling interval of the corresponding camera according to the judgment result of step S4 includes: Reduce the sampling interval used by cameras that detect an anomaly, and increase the sampling interval used by cameras that do not detect an anomaly.

8. The inference resource allocation optimization method for multi-camera real-time video monitoring according to claim 1 is characterized in that: When all cameras do not detect abnormal events or all detect abnormal events, the sampling interval T of each camera is ζ Expressed as: Where: N represents the total number of cameras, and C represents the total computing resources.

9. The inference resource allocation optimization method for multi-camera real-time video monitoring according to claim 8 is characterized in that: When some cameras detect abnormal events, resource allocation is dynamically adjusted; Assume that m cameras detect abnormal events, and the remaining Nm cameras do not detect abnormal events; Assume that the priority of each camera that detects an abnormal event is P i , the priority of ordinary cameras is P j =1, where P i >1; The total computing resources C are allocated to m abnormal event cameras and Nm normal cameras; The sampling interval of camera i that detects an abnormal event and camera j that does not detect an abnormal event can be expressed as: For camera i that detects an abnormal event: For camera j that does not detect any abnormal events: Where: R i and R j They represent the proportion of computing resources obtained by cameras that detect abnormal events and cameras that do not detect abnormal events.

10. A reasoning resource allocation optimization system for multi-camera real-time video monitoring, characterized in that: include: The pre-processing module includes multiple extractors, pre-processing units and image buffer queues corresponding to the number of cameras; Inference module, including GPU / NPU inference card, equipped with GPU / NPU accelerated object detection algorithm; The post-processing module includes an anomaly discriminator, a resource allocator, an output recording unit, and an inference result queue; The resource allocator dynamically adjusts the sampling interval of the corresponding camera according to the judgment result of the anomaly identifier; the sampling interval used by the camera that detects an abnormal event is reduced, and the sampling interval used by the camera that does not detect an abnormality is increased.

Images