A Video Patrol Task Priority Scheduling System Integrating Real-Time Data

Abstract

The present invention provides a video inspection task priority scheduling system integrating real-time data, comprising: arranging cameras and sensors in different monitoring areas; associating and storing video frames and sensing data with the area numbers where they are located; generating abnormal summary information based on a multi-modal anomaly detection model according to the input video frames; polling and inspecting the video frames of the monitoring areas according to the video task inspection priorities; when sensing data anomalies occur, obtaining the abnormal summary information of the video frames in the current monitoring area, triggering the retrieval of the video frames of the associated monitoring targets in the associated monitoring areas according to the generation frequency of the abnormal summary information, and using the monitoring target feature extraction module to extract the associated abnormal summary information; and respectively adjusting the video inspection task priorities of the corresponding monitoring areas according to the generation frequencies of the abnormal summary information and the associated abnormal summary information. The present invention can not only improve the monitoring efficiency, reduce the false alarm rate, but also provide more reliable decision-making support in emergency situations.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent monitoring, and relates to a video patrol task priority scheduling system integrating real-time data. Background Art

[0002] With the development of technology and the progress of society, video surveillance systems have become an indispensable part of fields such as public security, industrial production, and traffic management. Traditional video surveillance mainly relies on manual patrols to detect abnormal situations. This method is not only inefficient but also difficult to cope with the challenges of the massive data brought by large-scale surveillance networks. In order to improve the intelligence level and response speed of surveillance systems, automated video patrol systems integrating real-time data have gradually become a research hotspot.

[0003] In existing video surveillance practices, fixed cameras are usually used to monitor specific areas, and abnormal behaviors are identified through manual or simple automatic alarm mechanisms. However, this mode has several obvious deficiencies: First, it highly depends on human resources. Especially when multiple monitoring points need to be continuously monitored, it is easy to cause fatigue and omissions. Second, for complex events in dynamic scenarios (such as crowd gatherings, traffic accidents, etc.), it is difficult to make accurate judgments only relying on image information. Finally, when facing emergencies, due to the lack of an effective priority scheduling mechanism, resources are often not concentrated in the most critical places in a timely manner, affecting the effect of emergency handling.

[0004] Therefore, how to provide a video patrol system that can integrate real-time data and support intelligent priority scheduling is an urgent problem for those skilled in the art. Summary of the Invention

[0005] In view of this, the present invention proposes a video patrol task priority scheduling system integrating real-time data, which can not only improve the monitoring efficiency, reduce the false alarm rate, but also provide more reliable decision-making support in emergencies, and plays an irreplaceable role in ensuring social security and promoting economic development.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] The present invention discloses a video patrol task priority scheduling system integrating real-time data, including: a camera, a sensor, a multi-source data monitoring memory, a monitoring target feature extraction module, a video patrol task module, a regional chain trigger module, and a dynamic priority scheduling module; wherein,

[0008] The camera and the sensor are arranged in different monitoring areas. The camera is used to obtain video frames of the monitoring target, and the sensor is used to obtain sensing data of the monitoring target;

[0009] The multi-source data monitoring memory associates and stores the video frames and the sensing data with the area number after adding time stamps;

[0010] The monitoring target feature extraction module is used to load a multi-modal anomaly detection model, which is trained based on the feature space distribution of the monitoring target under normal working conditions as training samples, and the multi-modal anomaly detection model generates anomaly summary information according to the input video frames;

[0011] The video inspection task module polls the video frames in each monitoring area according to the video task inspection priority, including generating a warning message according to the anomaly summary information generated by the monitoring target feature extraction module for the video frames at the polling time node;

[0012] The area chain trigger module establishes an area association map based on the abnormal spatio-temporal chain relationship between monitoring targets; determines whether the sensing data is abnormal, and if so, calls the monitoring target feature extraction module to obtain the anomaly summary information of the video frames in the current monitoring area, generates a frequency trigger according to the anomaly summary information to retrieve the video frames of the associated monitoring targets within the same time period in the associated monitoring area, and extracts the associated anomaly summary information by using the monitoring target feature extraction module;

[0013] The dynamic priority scheduling module adjusts the video inspection task priorities of the corresponding monitoring areas respectively according to the generation frequencies of the anomaly summary information and the associated anomaly summary information.

[0014] Preferably, the monitoring target includes a static monitoring target; in the multi-source data monitoring memory, there is stored a dynamic feature template library based on the state evolution of the target device throughout its life cycle, and a target device feature sample library obtained by extracting image segments of the normal state of the target device from the historical video frame sequence for feature annotation, and the dynamic feature template library and the target device feature sample library are used for the recognition training of the static monitoring target by the multi-modal anomaly detection model.

[0015] Preferably, the multi-modal anomaly detection model adopts a dual-channel feature extraction architecture, including:

[0016] Video feature extraction channel: A three-stage cascaded encoder structure is used to process the input video frames; among them, the first-stage encoder extracts the basic visual features of the target device based on the improved ResNet-50 network; the second-stage encoder captures the temporal change features of the basic visual features of the target device through 3D convolution kernels; the third-stage encoder combines the device life cycle data in the dynamic feature template library to generate a video frame feature vector including aging compensation;

[0017] Sensor feature extraction channel: Adopt a temporal feature pyramid network architecture; among them, first perform sliding window normalization processing on the original sensor signal; then use the multi-head attention mechanism to extract cross-sensor coupling correlation features; perform dynamic time warping matching on real-time sensor data and historical normal data to generate a sensor feature vector containing the morphological deviation features between the real-time sensor data and the historical normal mode.

[0018] After fusing the video frame feature vector and the sensor feature vector, input them into an anomaly score generator to generate anomaly summary information.

[0019] Preferably, before fusing the video frame feature vector and the sensor feature vector, it further includes: constructing a three-dimensional spatio-temporal feature cube and performing matrix fusion on the spatial information of the video frame sequence and the temporal change trend of the sensor.

[0020] Preferably, the monitoring target includes a dynamic monitoring target; in the multi-source data monitoring memory, there is stored a target object feature sample library obtained by annotating the image segments of the target scene extracted from the historical video frame sequence for the identification training of the multi-modal anomaly detection model for static monitoring targets.

[0021] Preferably, the steps of the multi-modal anomaly detection model include:

[0022] Use the object detection algorithm to identify the interested moving objects in the video frame and obtain the spatial features of the interested moving objects.

[0023] For the detected interested moving objects, apply the multi-object tracking algorithm to track their movement trajectories between consecutive frames and obtain the time series features.

[0024] Combine the extracted spatial features and the time series features to form a spatio-temporal feature representation, use the recurrent neural network to capture the behavior patterns of the interested moving objects, and input them into the anomaly score generator to generate anomaly summary information.

[0025] Preferably, the steps for the regional chain trigger module to establish the regional association map include:

[0026] Construct a directed weighted graph model:

[0027] G=(V,E,W);

[0028] In the formula, V represents the set of monitoring area nodes, E represents the set of edges of the chain relationship between regions, and W ∈ R {N×N} is the dynamic chain weight matrix;

[0029] The update strategy of the dynamic chain weight matrix is:

[0030] ;

[0031] where w ij represents the conditional probability that an anomaly in area i triggers an anomaly in area j, α represents the forgetting factor, represents the actual chain anomaly indication function from area i to j in the k-th event, T represents the length of the sliding time window, and t represents the number of updates.

[0032] Preferably, the area chain trigger module is used to perform the following steps:

[0033] Determine whether the sensor data in the monitored area A is abnormal. If so, extract the video frames of the previous time period corresponding to the abnormal time node from the multi-source data monitoring memory, and input them to the monitoring target feature extraction module to generate abnormal summary information;

[0034] Determine whether the generation frequency of the abnormal summary information corresponding to the monitored area A reaches a first threshold. If so, obtain the associated monitored area B according to the area association map, and retrieve from the multi-source data monitoring memory the video frames of the associated monitored area B within the same time period as the latest update time point of the abnormal summary information of the monitored area A.

[0035] Preferably, the dynamic priority scheduling module defines multiple priority levels and assigns corresponding resource weights to each level; the execution steps include:

[0036] The priority scheduling steps for the video inspection task of the monitored area A include:

[0037] Count the generation frequency f1 of the abnormal summary information corresponding to the monitored area A and compare it with the first threshold Threshold1:

[0038] If f1 ≥ Threshold1, then execute the queue-jumping mechanism for the inspection task of the monitored area A according to the preset rules for the priority of the inspection task of the monitored area A;

[0039] The priority scheduling steps for the video inspection task of the associated monitored area B:

[0040] Count the generation frequency f2 of the associated abnormal summary information corresponding to the associated monitored area B within the same time period and compare it with the second threshold Threshold2:

[0041] If f2 ≥ Threshold2, then execute the queue-jumping mechanism for the inspection task of the associated monitored area B according to the preset rules for the priority of the inspection task of the associated monitored area B.

[0042] Preferably, the warning message includes a spatio-temporal event graph of the abnormal occurrence video frame, including a three-dimensional coordinate marking frame of the abnormal occurrence position superimposed on the video picture, a time-axis sequence of the abnormal evolution process, and an associated sensor data change curve.

[0043] As can be seen from the above technical solutions, compared with the prior art, the beneficial effects of the present invention include:

[0044] The multi-modal anomaly detection model trained based on the feature space distribution of the monitoring target under normal working conditions can effectively identify abnormal situations in the video frame. This dual-channel feature extraction architecture (video feature extraction channel and sensor feature extraction channel) combines visual information and sensing data, greatly improving the accuracy and reliability of anomaly detection.

[0045] The regional chain trigger module establishes a regional association graph based on the abnormal spatio-temporal chain relationship between monitoring targets. When an anomaly occurs in a certain monitoring area, other associated monitoring areas can be automatically triggered for inspection according to the chain relationship, timely discovering and handling potential risks, and greatly reducing the possibility of missed reports.

[0046] The system is applicable to both static equipment status monitoring and the analysis of the behavior patterns of dynamic objects in the scene, has a wide range of applications, and can be applied to multiple fields such as industrial production safety monitoring and urban traffic management.

[0047] In summary, the present invention not only improves the intelligence level of the monitoring system, but also greatly enhances its ability to cope with complex environments, which has important practical significance for improving the social safety management level. Description of the Drawings

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained according to the provided drawings;

[0049] Figure 1 It is a schematic diagram of a video inspection task priority scheduling system integrating real-time data provided by an embodiment of the present invention;

[0050] Figure 2 It is a processing flow chart of the multi-modal anomaly detection model for static monitoring targets provided by an embodiment of the present invention;

[0051] Figure 3 It is a processing flow chart of the multi-modal anomaly detection model for dynamic monitoring targets provided by an embodiment of the present invention;

[0052] Figure 4 This is the flowchart of the dynamic priority scheduling provided by the embodiment of the present invention. Detailed implementation manners

[0053] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0054] The embodiment of the present invention provides a video inspection task priority scheduling system that integrates real-time data, as Figure 1 shown, including: a camera, a sensor, a multi-source data monitoring memory, a monitoring target feature extraction module, a video inspection task module, a regional chain trigger module, and a dynamic priority scheduling module; cameras and sensors are arranged in different monitoring areas. The camera is used to obtain video frames of the monitoring target, and the sensor is used to obtain sensing data of the monitoring target; the multi-source data monitoring memory associates and stores the video frames and sensing data with the area number after adding time stamps; the monitoring target feature extraction module is used to load a multi-modal anomaly detection model. The multi-modal anomaly detection model is trained based on the feature space distribution of the monitoring target under normal working conditions as training samples. The multi-modal anomaly detection model generates anomaly summary information according to the input video frames; the video inspection task module polls the video frames in each monitoring area according to the video task inspection priority, including generating a warning message according to the anomaly summary information generated by the monitoring target feature extraction module for the video frames at the polling time node; the regional chain trigger module establishes a regional association map based on the abnormal spatio-temporal chain relationship between monitoring targets, which is used to reflect the possible abnormal monitoring targets in other areas caused by the abnormality in a certain monitoring area; it judges whether the sensing data is abnormal. If so, it calls the monitoring target feature extraction module to obtain the anomaly summary information of the video frames in the current monitoring area, generates a frequency according to the anomaly summary information to trigger the retrieval of the video frames of the associated monitoring targets in the same time period in the associated monitoring area, and uses the monitoring target feature extraction module to extract the associated anomaly summary information; the dynamic priority scheduling module adjusts the video inspection task priorities of the corresponding monitoring areas respectively according to the generation frequencies of the anomaly summary information and the associated anomaly summary information.

[0055] In one embodiment, the monitoring targets include static monitoring targets, such as the damage monitoring of road safety protection facilities, the icing monitoring of transmission lines, and the fault monitoring of road traffic lights; the multi-source data monitoring memory stores a dynamic feature template library based on the state evolution of the target device throughout its life cycle, and a target device feature sample library obtained by extracting image segments of the normal state of the target device from historical video frame sequences and performing feature annotation. The dynamic feature template library and the target device feature sample library are used for the recognition training of static monitoring targets by the multi-modal anomaly detection model.

[0056] In this embodiment, the multi-modal anomaly detection model adopts a dual-channel feature extraction architecture, including:

[0057] Video feature extraction channel: A three-stage cascaded encoder structure is used to process the input video frames; among them, the first-stage encoder extracts the basic visual features of the target device based on the improved ResNet-50 network;

[0058] The second-stage encoder captures the temporal change features of the basic visual features of the target device through 3D convolutional kernels;

[0059] The third-stage encoder combines the device life cycle data in the dynamic feature template library to generate a video frame feature vector including aging compensation;

[0060] Sensor feature extraction channel: Adopt a temporal feature pyramid network architecture; among them, first perform sliding window normalization processing on the original sensor signal; then use the multi-head attention mechanism to extract cross-sensor coupling correlation features; perform dynamic time warping matching on the real-time sensor data and historical normal data to generate a sensor feature vector including the morphological deviation features between the real-time sensor data and the historical normal mode;

[0061] Feature fusion is performed on the video frame feature vector and the sensor feature vector, and the expression is:

[0062] ;

[0063] In the formula, represents the video frame feature vector, represents the sensor feature vector, represents the video frame feature vector weight, represents the sensor feature vector weight, σ represents the Sigmoid activation function, ⊙ represents the Hadamard product, and b represents the adjustment factor.

[0064] Input to the anomaly score generator to generate anomaly summary information, and the expression is:

[0065] ;

[0066] In the formula, F normalIt is a normal state feature template, τ is the dynamic threshold, and k is the sensitivity coefficient.

[0067] In this embodiment, before fusing the video frame feature vector and the sensor feature vector, it further includes: constructing a three-dimensional spatio-temporal feature cube, and performing matrix fusion on the spatial information of the video frame sequence and the temporal change trend of the sensor.

[0068] In one embodiment, the monitoring target includes a dynamic monitoring target, such as the monitoring of illegal vehicles and pedestrians in a monitoring area; the multi-source data monitoring memory stores a target object feature sample library obtained by annotating the image segments of the target scene extracted from the historical video frame sequence for the identification training of the multi-modal anomaly detection model for static monitoring targets.

[0069] In one embodiment, the steps performed by the multi-modal anomaly detection model include:

[0070] Using a target detection algorithm to identify the interesting moving objects in the video frame to obtain the spatial features of the interesting moving objects;

[0071] For the detected interesting moving objects, applying a multi-target tracking algorithm to track their motion trajectories between consecutive frames to obtain time series features;

[0072] Combining the extracted spatial features and time series features to form a spatio-temporal feature representation, using a recurrent neural network to capture the behavior patterns of the interesting moving objects, and inputting them into an anomaly score generator to generate anomaly summary information.

[0073] In one embodiment, the steps for the regional chain trigger module to establish a regional association map include:

[0074] Constructing a directed weighted graph model:

[0075] G=(V,E,W);

[0076] In the formula, V represents the set of monitoring area nodes, E represents the set of edges of the chain relationship between regions, and W∈R {N×N} is a dynamic chain weight matrix;

[0077] The update strategy of the dynamic chain weight matrix is:

[0078] ;

[0079] In the formula, w ij represents the conditional probability that the anomaly in region i causes the anomaly in region j, α represents the forgetting factor, represents the actual chain anomaly indicator function from region i to j in the kth event, T represents the length of the sliding time window, and t represents the number of updates.

[0080] In one embodiment, the area chain trigger module is used to perform the following steps:

[0081] Determine whether the sensor data in monitoring area A is abnormal. If so, extract the video frames of the previous time period corresponding to the abnormal time node from the multi-source data monitoring memory, and input them to the monitoring target feature extraction module to generate abnormal summary information; if not, continue to perform video patrol according to the current video patrol task priority.

[0082] Determine whether the generation frequency of the abnormal summary information corresponding to monitoring area A reaches threshold one. If so, obtain the associated monitoring area B according to the area association map, and retrieve the video frames of the associated monitoring area B in the same time period as the latest update time point of the abnormal summary information of monitoring area A from the multi-source data monitoring memory.

[0083] In one embodiment, the dynamic priority scheduling module defines multiple priority levels and assigns corresponding resource weights to each level; the execution steps include:

[0084] The priority scheduling steps for the video patrol task in monitoring area A include:

[0085] Statistically calculate the generation frequency f1 of the abnormal summary information corresponding to monitoring area A and compare it with threshold one Threshold1:

[0086] If f1 ≥ Threshold1, implement the queue-jumping mechanism for the patrol task priority of monitoring area A according to the preset rules; if f1 < Threshold1, continue to perform video patrol according to the current video patrol task priority.

[0087] The priority scheduling steps for the video patrol task in the associated monitoring area B:

[0088] Statistically calculate the generation frequency f2 of the associated abnormal summary information corresponding to the associated monitoring area B in the same time period and compare it with threshold two Threshold2:

[0089] If f2 ≥ Threshold2, implement the queue-jumping mechanism for the patrol task priority of the associated monitoring area B according to the preset rules; if f2 < Threshold2, continue to perform video patrol according to the current video patrol task priority.

[0090] The execution process of this embodiment requires the cooperation of the area chain trigger module, the monitoring target feature extraction module, and the dynamic priority scheduling module. The overall implementation process is as follows:

[0091] First, determine whether the sensor data in the monitoring area A is abnormal. If so, extract the video frames of the previous time period and input them into the monitoring target feature extraction module to generate abnormal summary information A. Adjust the video task inspection priority of the monitoring area A according to whether the frequency of generating abnormal summary information A within a set time interval (such as within a month or within a day) reaches threshold one. At the same time, obtain associated monitoring targets according to the area chain trigger module, locate their associated monitoring area B, and extract the video frames of the current time period (the time period when the sensor data in the monitoring area A is abnormal) of the associated monitoring area B and input them into the monitoring target feature extraction module to generate associated abnormal summary information B. Adjust the video task inspection priority of the associated monitoring area B according to whether the frequency of generating associated abnormal summary information B reaches threshold two.

[0092] In one embodiment, the warning message includes a spatio-temporal event graph of the abnormal occurrence video frame, including a three-dimensional coordinate marking box of the abnormal occurrence position presented by overlay on the video screen, a time-axis sequence of the abnormal evolution process, and an associated sensor data change curve.

[0093] The following gives a scenario description of the damage monitoring of road safety protection facilities:

[0094] In urban traffic management, the integrity of road safety protection facilities (such as guardrails, crash barriers, isolation belts, etc.) is crucial for ensuring traffic safety. However, due to being exposed to the natural environment for a long time, these facilities are vulnerable to various factors and may be damaged, such as vehicle collisions and material aging. The traditional regular inspection method is inefficient and difficult to detect potential risks in a timely manner. Therefore, the priority scheduling example of the present invention is as follows:

[0095] Deploy high-definition cameras at key sections to capture the status of the protection facilities. At the same time, install vibration sensors and displacement sensors inside or around the protection facilities to monitor changes in their structural status. All the collected data is time-stamped by the multi-source data monitoring memory and stored in association with the area number where it is located for subsequent analysis and processing.

[0096] Adopt a three-level cascade encoder structure to process the input video frames and identify the basic visual features of the protection facilities and their change trends. For example, when it is detected that the guardrail has obvious displacement or fracture (spatial feature change > 10% / s), it is regarded as an abnormality. After performing sliding window normalization processing on the original sensor signals, use the multi-head attention mechanism to extract cross-sensor coupling correlation features, perform dynamic time warping matching on the real-time data and the historical normal mode, and generate a sensor feature vector containing morphological deviation features. Combine the video frame feature vector and the sensor feature vector to construct a three-dimensional spatio-temporal feature cube, and calculate whether it is necessary to generate abnormal summary information through an abnormal score generator.

[0097] For road safety protection facilities that are normal under regular circumstances, the initial priority of their video inspection tasks is set to P base = 1. Whenever the system detects a summary of abnormal damage to the protection facilities, an alarm is triggered, and relevant departments are notified to go for inspection and repair. If ≥2 summaries of abnormal damage occur cumulatively in a certain section within one month, the priority of its video inspection task is raised to P new = 3. According to the road facility association map established by the regional chain trigger module (assuming the weight WAB between guardrail A and its adjacent guardrail B is 0.6), a re-inspection of relevant facilities within a range of 200 meters around is automatically triggered. For the facilities subject to the associated re-inspection, if the frequency of occurrence of their abnormal summaries exceeds 3 times per month, the priority of their inspection tasks is raised from P base = 2 to P new = 3.

[0098] When a certain guardrail (such as guardrail A) is determined to be a high-risk area (the priority rises from P base = 1 to P new = 3), the system will reallocate resources. For example, the camera originally assigned to the low-priority section C is temporarily transferred to this section, increasing the inspection frequency (from once per minute to once every 10 seconds).

[0099] At the same time, according to the information obtained by the regional chain trigger module, the real-time video stream of the adjacent guardrail B to guardrail A is automatically retrieved, and its inspection frequency is increased (adjusted from once every 30 seconds to once every 10 seconds) to ensure that any possible chain damage problems can be detected and handled in a timely manner.

[0100] The following gives a scenario description of transmission line icing monitoring:

[0101] Power companies usually manage a vast transmission network with a wide coverage area, which means that monitoring resources (such as cameras, sensors, computing power, etc.) are limited. If the monitoring tasks are not regulated by priority, it may lead to waste of resources or key areas not being monitored in a timely manner. Through the priority regulation mechanism, the system can adjust the resource allocation strategy according to the actual situation, concentrating more resources on areas with higher icing risks to ensure that these areas are inspected more frequently and meticulously.

[0102] Infrared cameras with a resolution of 1920×1080@30fps are deployed on insulator strings, and inclination sensors are installed on transmission lines. When ice ridges are found to grow through video feature extraction (spatial feature change > 15% / s) and the data of the inclination sensor deviates from the historical pattern, an icing abnormal summary is generated, and the initial priority of the video inspection task is P base= 1. In the actual scenario, an alarm is executed every time an icing anomaly summary is generated, and the ice removal operation is performed on the insulator strings in the corresponding area. When 3 icing anomaly summary data continuously appear within a month, the priority of the video inspection task for this scenario is increased to P new = 3. At the same time, according to the associated graph, the video re-inspection of the icing monitoring on the adjacent 200-meter wire segment is automatically triggered, and the frequency of the icing anomaly summary data of the wire segment is also counted. Here, the icing anomaly summary data of the wire segment is obtained by inspecting according to the execution cycle of the original wire segment area video inspection task. If the frequency exceeds 5 times / month, the priority adjustment operation is performed, such as it can be adjusted to P new = 2.

[0103] The following is the scenario description of the sudden gathering event monitoring in the subway station:

[0104] In the sudden gathering event in the subway station, although the management department needs to comprehensively monitor the entire station, the priority control of the monitoring video is the key to ensuring efficient emergency response and reasonable resource allocation. Dozens, hundreds or even hundreds of cameras are usually deployed in the subway station. When a sudden gathering event occurs, if all stations are inspected routinely in a unified manner, it is impossible to accurately screen and process the video stream of the key area in real time. If all areas run at the highest priority, it may lead to system overload and cause the emergency response to lag due to the dispersion of resources. Therefore, the priority scheduling example of the present invention is as follows:

[0105] When the pedestrian density in Area A suddenly increases to 3 people / ㎡ (Score = 3.2), a high-risk alarm is triggered. According to the associated graph, the weight W between Area A and Security Checkpoint B AB = 0.7, and the real-time video of Area B is automatically retrieved.

[0106] The priority of Area A is increased from P base = 2 to P new = 5 (processed by jumping the queue); the priority of Area B is increased from P base = 3 to P new = 4.4 (rounded to level 4).

[0107] The system turns the cameras originally assigned to Area C (priority 3) to Area A, and increases the inspection frequency of Area B (from 30 seconds / time to 10 seconds / time). Area A is increased from P base = 2 to P new = 5, realizing the processed by jumping the queue, ensuring the priority jump of the high-risk area, meeting the requirements of the emergency response plan for the timeliness of resource allocation. The inspection frequency of Area B is changed from 30 seconds / time to 10 seconds / time, realizing the dynamic resource weight allocation and optimizing the spatio-temporal utilization rate of the hardware resources.

[0108] The above has introduced in detail the video patrol task priority scheduling system that integrates real-time data. In this embodiment, specific examples are used to elaborate on the principle and implementation manner of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention. At the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present invention.

[0109] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be obvious to those skilled in the art. The general principles defined in this embodiment can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown in this embodiment, but rather to the widest scope consistent with the principles and novel features disclosed in this embodiment.

Claims

1. A video patrol task priority scheduling system integrating real-time data, characterized in that, Including: A camera, sensors, a multi-source data monitoring memory, a monitoring target feature extraction module, a video patrol task module, a regional chain trigger module, and a dynamic priority scheduling module; among them, The camera and sensors are arranged in different monitoring areas. The camera is used to obtain video frames of the monitoring target, and the sensors are used to obtain sensing data of the monitoring target; The multi-source data monitoring memory associates and stores the video frames and the sensing data with the area number after adding time stamps; The monitoring target feature extraction module is used to load a multi-modal anomaly detection model. The multi-modal anomaly detection model is trained based on the feature space distribution of the monitoring target under normal working conditions as training samples. The multi-modal anomaly detection model generates anomaly summary information according to the input video frames; The video patrol task module polls and patrols the video frames in each monitoring area according to the video task patrol priority, including generating a warning message according to the anomaly summary information generated by the monitoring target feature extraction module for the video frames at the polling time node; The regional chain trigger module establishes a regional association map based on the abnormal spatio-temporal chain relationship between monitoring targets; judges whether the sensing data is abnormal. If so, it calls the monitoring target feature extraction module to obtain the anomaly summary information of the video frames in the current monitoring area, generates a frequency trigger according to the anomaly summary information to retrieve the video frames of associated monitoring targets in the same time period in the associated monitoring area, and uses the monitoring target feature extraction module to extract associated anomaly summary information; the regional chain trigger module is used to perform the following steps: Judge whether the sensor data in monitoring area A is abnormal. If so, extract the video frames of the previous time period corresponding to the abnormal time node from the multi-source data monitoring memory and input them into the monitoring target feature extraction module to generate anomaly summary information; Judge whether the generation frequency of the anomaly summary information corresponding to monitoring area A reaches threshold one. If so, obtain the associated monitoring area B according to the regional association map, and retrieve from the multi-source data monitoring memory the video frames of the associated monitoring area B in the same time period as the latest update time point of the anomaly summary information of monitoring area A; The dynamic priority scheduling module adjusts the video patrol task priorities of the corresponding monitoring areas according to the generation frequencies of the anomaly summary information and the associated anomaly summary information respectively.

2. The video inspection task priority scheduling system integrating real-time data according to claim 1, wherein The monitoring target includes static monitoring targets; the multi-source data monitoring memory stores a dynamic feature template library based on the state evolution of the target device throughout its life cycle, and a target device feature sample library obtained by extracting and feature-labeling image segments of the normal state of the target device from the historical video frame sequence. The dynamic feature template library and the target device feature sample library are used for the multi-modal anomaly detection model to identify and train static monitoring targets.

3. The video inspection task priority scheduling system integrating real-time data according to claim 2, characterized in that, The multi-modal anomaly detection model adopts a two-channel feature extraction architecture, including: Video feature extraction channel: A three - cascaded encoder structure is used to process the input video frames. Among them, the first - stage encoder extracts the basic visual features of the target device based on the improved ResNet - 50 network; the second - stage encoder captures the temporal change features of the basic visual features of the target device through 3D convolutional kernels; the third - stage encoder combines the device life - cycle data in the dynamic feature template library to generate a video - frame feature vector including aging compensation. Sensor feature extraction channel: A temporal feature pyramid network architecture is adopted. Among them, first, the original sensor signals are processed by sliding - window normalization; then, a multi - head attention mechanism is used to extract cross - sensor coupling and correlation features; the real - time sensor data is dynamically time - warped and matched with the historical normal data to generate a sensor feature vector including the morphological deviation features between the real - time sensor data and the historical normal mode. After fusing the video - frame feature vector and the sensor feature vector, it is input into an anomaly scoring generator to generate anomaly summary information.

4. The video inspection task priority scheduling system integrating real-time data according to claim 3, characterized in that, Before fusing the video - frame feature vector and the sensor feature vector, it also includes: constructing a three - dimensional spatio - temporal feature cube, and matrix - fusing the spatial information of the video - frame sequence and the temporal change trend of the sensor.

5. The video inspection task priority scheduling system integrating real-time data according to claim 1, wherein The monitoring target includes dynamic monitoring targets; in the multi - source data monitoring memory, there is a target object feature sample library obtained by annotating the image segments of the target scene extracted from the historical video - frame sequence for the recognition training of static monitoring targets by the multi - modal anomaly detection model.

6. The video inspection task priority scheduling system integrating real-time data according to claim 5, characterized in that, The steps executed by the multi - modal anomaly detection model include: Using a target detection algorithm to identify the interested moving objects in the video frame and obtain the spatial features of the interested moving objects. For the detected interested moving objects, applying a multi - target tracking algorithm to track their motion trajectories between consecutive frames and obtain time - series features. Combining the extracted spatial features and the time - series features to form a spatio - temporal feature representation, using a recurrent neural network to capture the behavior patterns of the interested moving objects, and inputting them into an anomaly scoring generator to generate anomaly summary information.

7. A video inspection task priority scheduling system integrating real-time data according to claim 1, characterized in that, The steps for the regional chain - trigger module to establish a regional association map include: Constructing a directed weighted graph model: G=(V,E,W); Wherein, V represents the set of monitoring area nodes, E represents the set of edges of the inter-regional chain relationship, and W ∈ R {N×N} is the dynamic chain weight matrix; The dynamic chain weight matrix update strategy is: ； where w ij represents the conditional probability that an anomaly in region i causes an anomaly in region j, α represents the forgetting factor, represents the actual cascading anomaly indicator function from region i to j in the k-th event, T represents the length of the sliding time window, and t represents the number of updates.

8. The video inspection task priority scheduling system integrating real-time data according to claim 1, wherein, The dynamic priority scheduling module defines multiple priority levels and assigns corresponding resource weights to each level. The execution steps include: The priority scheduling steps for the video inspection task in monitoring area A include: Counting the generation frequency f1 of the corresponding anomaly summary information in monitoring area A and comparing it with the first threshold Threshold1: If f1≥Threshold1, then execute the queue - jumping mechanism for the inspection task in monitoring area A according to the preset rules for the inspection task in monitoring area A. The priority scheduling steps for the video inspection task associated with monitoring area B: Counting the generation frequency f2 of the corresponding anomaly summary information in the associated monitoring area B within the same time period and comparing it with the second threshold Threshold2: If f2 ≥ Threshold2, the priority of the inspection task for monitoring area B will execute the queue-jumping mechanism for the inspection task of monitoring area B according to the preset rules.

9. The video inspection task priority scheduling system for fusing real-time data according to claim 1, characterized in that, The warning message includes the spatio-temporal event graph of the abnormal occurrence video frame, and also includes a three-dimensional coordinate marking box at the abnormal occurrence position superimposed and presented on the video screen, a time-axis sequence of the abnormal evolution process, and an associated sensor data change curve.

Images