Campus panoramic security monitoring management system fusing deep learning

Abstract

The application discloses a campus panoramic security monitoring management system fusing deep learning, and belongs to the technical field of image communication monitoring.The campus monitoring area is divided into multiple sub-monitoring areas, and sub-panoramic splicing is independently carried out;deep learning is applied to the whole chain of "abnormal sensing->trajectory prediction->sub-panoramic splicing decision->execution";abnormal targets are identified in real time by deep learning based on sub-panoramic video, a predicted safety distance sequence of the abnormal targets crossing the boundary of the area is predicted, event-driven progressive fusion splicing is triggered, and the splicing range is dynamically adjusted by deep reinforcement learning, so that the global monitoring resource is fundamentally changed from "one-time full splicing" to "on-demand hierarchical splicing+event-driven expansion+dynamic shrinkage and release", the calculation load and bandwidth occupation are reduced, the monitoring blind area and visual field jump are eliminated in advance, and the real-time performance, expandability and intelligent level of the campus security system are comprehensively improved.

Description

Technical Field

[0001] This invention relates to the field of image communication and monitoring technology, and more specifically, to a campus panoramic security monitoring and management system that integrates deep learning. Background Technology

[0002] Campus safety is a major concern for society. Traditional campus security monitoring systems mainly rely on discretely deployed fixed cameras and manual monitoring, which have prominent problems such as many blind spots, slow response, and fragmented information. With the development of deep learning technology, some systems have begun to introduce video intelligent analysis algorithms to achieve personnel identification and abnormal behavior detection, but most are still limited to single-camera scenarios and lack cross-camera identity association and panoramic situational awareness capabilities.

[0003] For example, the patent application with publication number CN121075083A involves a campus security management system based on deep learning, which achieves risk scoring and alarms through access control identity collection, cross-identity association of multiple cameras, and trajectory offset analysis. However, this solution still has the following shortcomings:

[0004] (1) The monitoring screen is still presented in discrete single camera images, without forming a unified panoramic view. Security personnel need to switch frequently between multiple screens, which can easily cause visual fatigue and information omission.

[0005] (2) It mainly relies on "trajectory deviation" to indirectly assess risk, and fails to directly identify specific abnormal behaviors such as fighting, falling, and climbing;

[0006] (3) When an abnormal target moves within the campus, although the solution can link the identity across cameras, the images of each camera are still displayed independently. There is no mechanism to perform panoramic fusion of video images from adjacent areas. Security personnel need to manually switch the images to track the target’s cross-area movement, resulting in poor tracking continuity and obvious jumps in field of view.

[0007] To address the aforementioned issues, some traditional technologies propose multi-channel video panoramic stitching schemes that fuse images from multiple cameras into a single panoramic image. However, this involves performing a one-time global stitching of all cameras across the entire campus, resulting in high computational load and bandwidth consumption. Furthermore, the stitched panoramic image contains a large amount of information from irrelevant areas, leading to a waste of computational resources. In addition, these schemes lack event-driven mechanisms, making it impossible to dynamically adjust the stitching range based on the location of abnormal events and the movement trend of abnormal targets. They also struggle to achieve smooth expansion of the monitoring range when abnormal targets move across regions.

[0008] Therefore, to address the aforementioned technical issues, we propose a campus panoramic security monitoring and management system that integrates deep learning. This system solves problems such as discrete monitoring images, poor cross-regional tracking continuity, lack of predictive field of view expansion, and waste of computing resources in existing technologies by using independent sub-panoramic stitching based on regional division and event-driven progressive panoramic stitching. Summary of the Invention

[0009] The purpose of this invention is to address practical technical deficiencies, and a campus panoramic security monitoring and management system integrating deep learning is provided.

[0010] The objective of this invention can be achieved through the following technical solution: a campus panoramic security monitoring and management system integrating deep learning, comprising:

[0011] The area division and sub-panorama stitching module is used to divide the campus monitoring area into multiple sub-monitoring areas according to the geographical distribution of the campus. At least one camera is deployed in each sub-monitoring area. A virtual global coordinate system is established based on the global coordinate system in the preset school panoramic design map. The transformation matrix from the sub-area coordinate system to the virtual global coordinate system is pre-calibrated for each sub-monitoring area. The video images captured by the cameras in each sub-monitoring area are spatiotemporally stitched together to generate several sub-panorama video images.

[0012] The abnormal event judgment and alarm module is used to perform real-time deep learning analysis on each sub-panoramic video image, identify targets and track trajectories, determine whether an abnormal event has occurred in the sub-monitoring area, mark it as an event area if so, lock the abnormal target for risk assessment and generate an alarm signal.

[0013] The trajectory tracking and linkage monitoring module responds to alarm signals and uses a temporal deep learning network to obtain the predicted trajectory of the abnormal target within a preset time period based on the historical trajectory sequence of the abnormal target. Based on the predicted trajectory, it calculates the predicted safe distance and predicted crossing time between the abnormal event target and the edge of the event area, generates linkage monitoring signals for neighboring areas, and determines at least one sub-monitoring area that is geographically adjacent to the event area as a shared monitoring area.

[0014] The progressive panoramic stitching module is used to map the sub-panoramic video images of the shared monitoring area and the event area to the virtual global coordinate system after triggering the linkage monitoring signal of the neighboring area. The stitching boundary is determined according to the spatial adjacency relationship between the two in the virtual global coordinate system. At the stitching boundary, multi-band fusion, virtual canvas stitching, or three-dimensional depth map parallax compensation is used for fusion transition to generate a panoramic video stitched image. A deep learning-driven dynamic range adjustment strategy is used to adjust the range of the sub-monitoring area participating in real-time stitching for continuous tracking.

[0015] Furthermore, the region division module divides the monitoring regions according to at least one of the following criteria:

[0016] The campus is divided into geographical zones, including the school gate area, teaching building area, playground area, dormitory area, canteen area, and perimeter wall area; the physical deployment locations of cameras and the degree of overlap in the coverage areas of each camera; and the floor and room layout of campus buildings.

[0017] Furthermore, the sub-panorama stitching module employs a real-time stitching algorithm based on GPU parallel pipelines to spatiotemporally stitch together video images captured by cameras within each sub-monitoring area.

[0018] Furthermore, the specific process for determining whether an abnormal event has occurred within the sub-monitoring area includes:

[0019] S1. Use a target detection and tracking model to detect people in each frame of the sub-panoramic video image, assign a stable cross-frame ID to each detected person, and record the motion trajectory;

[0020] S2. The pose estimation model is used to extract the coordinates of key points of the human body in real time. The pose features are identified by calculating the angle between key points. The pose feature sequence of N consecutive frames is input into the pre-trained temporal behavior recognition model, and the abnormal behavior type and confidence level are output. The types include abnormal behaviors such as fighting, falling, climbing and crossing, intrusion into dangerous areas, gathering of people and prolonged stay.

[0021] S3. When the confidence level of a certain type of abnormal behavior exceeds the preset threshold, it is determined that an abnormal event has occurred in the current sub-monitoring area, marked as an event area, and the target is marked as an abnormal target.

[0022] Furthermore, the specific process for conducting risk assessments and generating alarm signals for abnormal targets includes:

[0023] T1. Locate the abnormal target in the event area, record the event start time, abnormal target ID and abnormal behavior type, and event area location, and summarize them into abnormal event content.

[0024] T2. Extract multi-dimensional risk features based on the content of abnormal events, including behavioral risk coefficient, target identity coefficient, dynamic trend coefficient, regional sensitivity coefficient, and spatiotemporal background coefficient.

[0025] T3. A weighted scoring model is used to calculate the risk score, and the risk level is determined based on the risk score, including low risk, medium risk, and high risk. Multi-level alarm signals are matched according to the risk level.

[0026] Furthermore, the specific process for generating neighboring area linkage monitoring signals includes:

[0027] Obtain the historical trajectory point sequence of the abnormal target within the past T seconds, input the historical trajectory point sequence into a pre-trained temporal deep learning network, which adopts the Transformer architecture, and output the predicted trajectory point sequence within the future prediction time.

[0028] Obtain the mathematical representation of the boundary line of the current event region, L_boundary. For each predicted point in the predicted trajectory point sequence, calculate the shortest Euclidean distance from the point to the boundary of the current event region and mark it as the safe distance to obtain the predicted safe distance sequence.

[0029] In the predicted safe distance sequence, it is detected whether there is a future moment when the safe distance is less than or equal to the preset linkage triggering distance threshold. If so, the first future arrival time that satisfies the safe distance ≤ the preset linkage triggering distance threshold is recorded, and the time difference between the future arrival time and the current time is calculated and marked as the predicted crossing time. When the predicted crossing time is less than the preset risk time threshold, a neighboring area linkage monitoring signal is generated.

[0030] Furthermore, the specific process for adaptively adjusting the range of sub-monitoring areas participating in real-time stitching for continuous tracking includes:

[0031] A deep reinforcement learning range adjustment model is constructed. The model takes the number of sub-regions in the current stitching range, the predicted safe distance sequence of abnormal targets, and the resource occupancy rate of edge computing nodes as state inputs, and expands the stitching range, shrinks the stitching range, and maintains the current stitching range as action outputs. The resource occupancy rate here includes at least one of GPU utilization, video memory occupancy rate, and memory occupancy rate.

[0032] The deep reinforcement learning model aims to minimize resource consumption while ensuring that abnormal targets are always within the stitching field of view, and drives model parameter updates through reward signals.

[0033] During real-time tracking, the current state is periodically input into the model, and corresponding operations are performed based on the actions output by the model:

[0034] If the output is to expand the stitching range, then according to the predicted movement direction of the abnormal target, the shared monitoring area is added to the current stitching range, triggering the fusion and stitching of the sub-panoramic video images of the shared monitoring area and the sub-panoramic video images of the current event area;

[0035] If the output is to reduce the stitching range, then perform a cross-region switching operation, remove the historical event region, remove it from the stitching, release computing resources, and update the shared monitoring region to the new event region;

[0036] If the output is to maintain the current splicing range, then the existing splicing area combination will remain unchanged;

[0037] Furthermore, the triggering conditions for cross-region switching are as follows: when an abnormal target completely leaves the current event area and enters the shared monitoring area, the abnormal target is not detected for 10 consecutive frames in the current event area, is detected for 5 consecutive frames in the shared monitoring area, and the depth of the abnormal target crossing the boundary line of the shared monitoring area exceeds the preset linkage triggering distance threshold, the current event area is marked as a historical event area, causing it to exit real-time stitching, and the shared monitoring area is updated to a new event area.

[0038] Compared with the prior art, the advantages of this invention are:

[0039] 1. This invention divides the campus monitoring area into multiple sub-monitoring areas according to geographical location. Each area independently performs spatiotemporal synchronous sub-panoramic stitching and establishes a virtual global coordinate system and pre-calibrates the transformation matrix from each sub-area to the global coordinate system. In real-time tracking mode, real-time deep learning analysis is performed on each sub-panoramic video image to identify targets and track trajectories, mark event areas and lock abnormal targets, and predict safe distance sequences to trigger linkage monitoring signals in neighboring areas to determine neighboring shared monitoring areas. For different spatial layout conditions, three fusion strategies are adopted: multi-band fusion, virtual canvas stitching and three-dimensional depth map parallax compensation, to seamlessly stitch the sub-panoramic images of the event area and the shared monitoring area into a unified panoramic image.

[0040] As abnormal targets move across regions, the system dynamically performs adaptive sub-panorama stitching adjustments, such as "expanding the stitching range, shrinking the stitching range, and maintaining the current stitching range." This achieves a fundamental shift in global monitoring resources from "one-time full stitching" to "on-demand layered stitching + event-driven expansion + dynamic shrinking and release." Compared to traditional global panorama stitching, this reduces real-time computing load and bandwidth consumption, significantly improving the scalability and deployment flexibility of the campus security monitoring system.

[0041] 2. This invention also integrates deep learning throughout the entire chain of "anomaly perception → trajectory prediction → sub-panorama stitching decision → execution": real-time deep learning analysis is performed on each sub-panorama video image to identify abnormal behavior and track targets. Temporal networks are used to predict the future trajectory of abnormal targets and the predicted safe distance sequence for crossing regional boundaries. A deep reinforcement learning model is used to dynamically decide the adjustment strategy of the stitching range, realizing a leap from passive fixed threshold linkage to active predictive adaptive linkage, and an upgrade from rule-driven stitching range adjustment to data-driven intelligent range adjustment. This allows for the early triggering of cross-regional panoramic stitching before abnormal targets enter adjacent areas, effectively eliminating monitoring blind spots and field of view jumps. At the same time, computing resources are dynamically allocated according to real-time resource occupancy to avoid overloading of edge nodes, comprehensively improving the system's response speed, resource utilization efficiency, and decision-making intelligence level in complex campus environments. Attached Figure Description

[0042] Figure 1 This is a system principle block diagram of the present invention;

[0043] Figure 2 This is a flowchart of the method of the present invention. Detailed Implementation

[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0045] Example 1: This invention discloses a campus panoramic security monitoring and management system that integrates deep learning. Please refer to [link / reference]. Figures 1-2 It includes a region division and sub-panorama stitching module, an abnormal event judgment and alarm module, a trajectory tracking and linkage monitoring module, a progressive panorama stitching module, an event tracing module, and a historical database connected to all modules.

[0046] The area division and sub-panorama stitching module is used to divide the campus monitoring area into multiple sub-monitoring areas according to the geographical distribution of the campus. Each sub-monitoring area is equipped with at least one camera. The video images captured by the cameras in each sub-monitoring area are spatiotemporally stitched together to generate several sub-panorama video images, which are then stored in the historical database.

[0047] The region division module divides the monitoring regions based on at least one of the following criteria:

[0048] The campus is divided into geographical zones, including the school gate area, teaching building area, playground area, dormitory area, canteen area, and perimeter wall area; the physical deployment locations of cameras and the degree of overlap in the coverage areas of each camera; the floor and room layout of campus buildings; all cameras are connected to an edge computing server cluster deployed in the campus network room via a gigabit campus network. The edge computing server uses 4 NVIDIA Jetson AGX Orin devices (each equipped with a 2048-core GPU, 32GB of video memory, and a 12-core ARM CPU), each responsible for real-time processing of different sub-monitoring areas.

[0049] The abnormal event judgment and alarm module is connected to the area division and sub-panorama stitching module. It is used to perform real-time deep learning analysis on each sub-panorama video image, identify targets and track trajectories, determine whether an abnormal event has occurred in the sub-monitoring area, mark the sub-monitoring area as the event area if so, lock the abnormal target for risk assessment, and generate an alarm signal based on the assessment results.

[0050] The sub-panorama stitching module employs a real-time stitching algorithm based on GPU parallel pipelines to spatiotemporally stitch together video images captured by cameras within each sub-monitoring area. Specifically:

[0051] Frame-level time synchronization of multiple cameras within the sub-monitoring area is achieved through PTP Precision Time Protocol or NTP Network Time Protocol, with the synchronization error controlled within 1ms.

[0052] The Zhang Zhengyou calibration method was used to calibrate the intrinsic and extrinsic parameters of each camera, and the video frames of each camera were mapped to a unified sub-region coordinate system.

[0053] A fast feature matching algorithm is used to extract feature points from video frames. A dynamic seam line is calculated using a graph cut-based minimum energy path algorithm. Multi-band fusion is performed using Laplacian pyramid weighted mixing. A single sub-panorama stitching is completed within 30ms, generating a smoothly transitioning sub-panorama video image. The sub-panorama video images are stored in the historical database in H.264 or H.265 encoding format according to time segments. The duration of each segment is preset to 1 minute or a preset time configuration.

[0054] The specific process for determining whether an abnormal event has occurred within a sub-monitoring area includes:

[0055] S1. Use a target detection and tracking model to detect people in each frame of the sub-panoramic video image, assign a stable cross-frame ID to each detected person, and record the motion trajectory;

[0056] The target detection and tracking model adopts YOLOv8x convolutional neural network and DeepSORT multi-target tracking algorithm. YOLOv8 target detection algorithm is the backbone network. It takes a single frame image of sub-panoramic video as input and outputs human body bounding box. It performs personnel identification and detection on each frame of sub-panoramic video image. DeepSORT multi-target tracking algorithm is based on 128-dimensional ReID features extracted by ResNet50 combined with Kalman filtering and Hungarian algorithm. It assigns a unique ID across frames to each detected person and records continuous motion trajectory.

[0057] S2. The pose estimation model is used to extract the coordinates of key points of the human body in real time. The pose features are identified by calculating the angle between key points. The pose feature sequence of N consecutive frames is input into the pre-trained temporal behavior recognition model, and the abnormal behavior type and confidence level are output. The types include abnormal behaviors such as fighting, falling, climbing and crossing, intrusion into dangerous areas, gathering of people and prolonged stay.

[0058] The pose estimation model uses the YOLOv8-Pose keypoint detection network, which shares a backbone network with the object detection network. The detection head outputs 17 human keypoint heatmaps. The model is trained under supervision by OKS loss. The input is the human bounding box clipping region, and the output is the coordinates and confidence of 17 keypoints: nose, left and right eyes, left and right ears, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, and left and right ankles. The joint angles are calculated based on the keypoints as angles to identify pose features.

[0059] The temporal behavior recognition model is a CNN-LSTM hybrid neural network. The CNN part consists of three convolutional layers (64→128→256 channels) and global average pooling, used to extract spatial features of single-frame pose features. The LSTM part is a two-layer long short-term memory network (256 hidden layer dimensions). It takes 16 consecutive frames of pose feature sequence as input and outputs temporal dependencies. Finally, it outputs the probability distribution of abnormal behaviors including fighting, falling, climbing, intrusion into dangerous areas, gathering of people, and prolonged stay after passing through fully connected layers and Softmax. The model is trained using 20,000 campus behavior segments (16 frames per segment), cross-entropy loss, Adam optimizer, and achieves a validation set accuracy of 94.3%. The processing time for 16 frames at the edge is about 8 milliseconds.

[0060] S3. When the confidence level of a certain type of abnormal behavior exceeds the preset threshold, it is determined that an abnormal event has occurred in the current sub-monitoring area, the sub-monitoring area is marked as an event area, and the target is marked as an abnormal target.

[0061] The specific process for conducting risk assessments and generating alarm signals for abnormal targets includes:

[0062] T1. Locate the abnormal target in the event area, record the event start time, abnormal target ID and abnormal behavior type, and event area location, and summarize them into abnormal event content.

[0063] T2. Extract multi-dimensional risk features based on the content of abnormal events, including behavioral risk coefficient, target identity coefficient, dynamic trend coefficient, regional sensitivity coefficient, and spatiotemporal background coefficient.

[0064] For example: Based on the identified abnormal behavior types, predefined danger coefficients are mapped (fighting 0.9, armed possession 1.0, falls 0.6, climbing 0.7, intrusion 0.8, gathering 0.5, loitering 0.3); target identity coefficients are determined by comparing facial recognition or ReID with a database of key personnel; if the target exists in a blacklist or key personnel list, the target identity coefficient is increased by 0.2–0.4; the rate of change of safe distance between the abnormal target and the edge of the event area is calculated to determine the dynamic trend coefficient; if the rate of change of safe distance is negative and the absolute value exceeds the speed threshold of 0.5 m / s, the dynamic trend coefficient is increased by 0.1–0.3; the area sensitivity coefficient is determined based on the event area type, and values ​​are assigned to the event area type: 0.9 for laboratories, rooftops, and power distribution rooms; 0.7 for school gates and perimeter walls; and 0.3 for ordinary areas; the spatiotemporal background coefficient is determined based on whether the current time belongs to a dangerous period (late at night, after school) and the recent frequency of events in the area, and a correction coefficient of 0–0.2 is calculated;

[0065] T3. A weighted scoring model is used to calculate the risk score, and the risk level is determined based on the risk score: low risk (0≤risk score<0.4), medium risk (0.4≤risk score<0.7), and high risk (0.7≤risk score≤1.0). Multi-level alarm signals are matched according to the risk level. For example, low risk corresponds to APP push notification, medium risk corresponds to SMS + large screen pop-up, and high risk corresponds to telephone voice + full screen pop-up + sound and light alarm. Video clips of 30 seconds before and after the abnormal event are automatically saved.

[0066] Example 2: The trajectory tracking and linkage monitoring module is connected to the abnormal event judgment and alarm module. After receiving the alarm signal, the temporal deep learning network is used to obtain the predicted trajectory of the abnormal target within a preset time period based on the historical trajectory sequence of the abnormal target. Based on the predicted trajectory, the predicted safe distance and predicted crossing time between the abnormal event target and the edge of the event area are calculated, and a linkage monitoring signal for the adjacent area is generated. At least one sub-monitoring area that is geographically adjacent to the event area is determined as a shared monitoring area.

[0067] The specific process for generating neighboring area linkage monitoring signals includes:

[0068] The historical trajectory point sequence of the abnormal target within the past T seconds is obtained. The historical trajectory point sequence is input into a pre-trained temporal deep learning network, which is a trajectory prediction model. It adopts the Transformer architecture and outputs the predicted trajectory point sequence within the future prediction time. The model is trained using 100,000 real moving trajectories collected on campus. The average displacement error and the final displacement error are used as the joint loss function. The single inference time at the edge is about 5 milliseconds.

[0069] Obtain the mathematical representation of the boundary line of the current event region, L_boundary. For each predicted point in the predicted trajectory point sequence, calculate the shortest Euclidean distance from the point to the boundary of the current event region and mark it as the safe distance to obtain the predicted safe distance sequence.

[0070] In the predicted safe distance sequence, it is detected whether there is a future moment when the safe distance is less than or equal to the preset linkage triggering distance threshold. If so, the first future arrival time that satisfies the safe distance ≤ the preset linkage triggering distance threshold is recorded, and the time difference between the future arrival time and the current time is calculated and marked as the predicted span time.

[0071] When the predicted span time is less than the preset risk time threshold, a neighboring area linkage monitoring signal is generated. Based on the distance between the predicted trajectory of the abnormal target and the area boundary, the progressive panoramic stitching module is used to dynamically stitch together the sub-panoramic views of the adjacent areas.

[0072] The progressive panoramic stitching module is connected to the trajectory tracking and linkage monitoring module. After triggering the linkage monitoring signal of the neighboring area, it maps the sub-panoramic video images of the shared monitoring area and the event area to the virtual global coordinate system. The stitching boundary is determined according to the spatial adjacency relationship between the two in the virtual global coordinate system. At the stitching boundary, multi-band fusion, virtual canvas stitching, or three-dimensional depth map parallax compensation is used to perform cross-regional fusion transition to generate a panoramic video stitching image. A deep learning-driven dynamic range adjustment strategy is used to adjust the range of the sub-monitoring area participating in real-time stitching for continuous tracking.

[0073] The specific process for adaptively adjusting the range of sub-monitoring areas participating in real-time stitching for continuous tracking includes:

[0074] A deep reinforcement learning range adjustment model is constructed. The model adopts the Deep Q-Network (DQN) algorithm. The state input is the number of sub-regions in the current stitching range, the predicted safe distance sequence of abnormal targets, and the resource utilization rate of edge computing nodes. The action output is to expand the stitching range, shrink the stitching range, or maintain the current stitching range. The resource utilization rate here includes at least one of GPU utilization, video memory utilization, and memory utilization.

[0075] Deep reinforcement learning models aim to minimize resource consumption while ensuring that anomalous targets are always within the stitching field of view, and drive model parameter updates through reward signals.

[0076] During real-time tracking, the current state is periodically input into the model, and corresponding operations are performed based on the actions output by the model:

[0077] If the output is to expand the stitching range, then according to the predicted movement direction of the abnormal target, the shared monitoring area is added to the current stitching range, triggering the fusion and stitching of the sub-panoramic video images of the shared monitoring area and the sub-panoramic video images of the current event area;

[0078] The specific method for cross-regional splicing is as follows:

[0079] Since there may be various different spatial layout relationships between different sub-monitoring areas, this application maps the sub-panoramic video images of the shared monitoring area and the sub-panoramic video images of the current event area onto the virtual global coordinate system according to the transformation matrix. First, the stitching boundary is determined according to the spatial adjacency relationship between the two images in the virtual global coordinate system, and the image connection at the stitching boundary is judged. Then, differentiated fusion strategies are designed for different situations:

[0080] If there is a pixel overlap area between the two images after mapping at the stitching boundary, the overlapping area is fused using a conventional smooth transition algorithm based on multi-band fusion, which is consistent with the spatiotemporal synchronous stitching method of video images in the same sub-monitoring area.

[0081] If the two images do not overlap pixels at the stitching boundary after mapping, a virtual canvas stitching method is used to place the two images side by side according to global coordinates, and the empty area in the middle of the display is filled by the existing scene completion algorithm.

[0082] If there are corners or doorways at the splicing boundary that cause perspective discontinuity, the fusion is completed after compensating for the parallax using a pre-acquired 3D depth map: the positional offset of pixels on both sides of the splicing boundary in 3D space is calculated based on the 3D depth map, and the parallax compensation transformation is performed on the two images to make the spatial structure on both sides of the corner or doorway visually continuously aligned. Then, multi-band fusion is used to smoothly transition the compensated overlapping area. Here, the 3D depth map is obtained by using a binocular stereo vision camera or depth sensor to perform 3D scanning of all corners and doorway areas in the campus during the system initialization phase.

[0083] If the output is to reduce the stitching range, then perform a cross-region switching operation, remove the historical event region, remove it from the stitching, release computing resources, and update the shared monitoring region to the new event region;

[0084] If the output is to maintain the current splicing range, then the existing splicing area combination will remain unchanged;

[0085] The triggering condition for cross-region switching is as follows: when an abnormal target completely leaves the current event region and enters the shared monitoring region, the abnormal target is not detected for 10 consecutive frames in the current event region and is detected for 5 consecutive frames in the shared monitoring region, and the depth of the abnormal target crossing the boundary line of the shared monitoring region exceeds the preset linkage triggering distance threshold, it indicates that the abnormal target is far away from the current event region and the distance is greater than the linkage triggering distance threshold. The current event region is marked as a historical event region, causing it to exit real-time stitching, releasing the GPU computing resources and bandwidth it occupies, updating the shared monitoring region to a new event region, and determining the next shared monitoring region based on the new predicted safe distance and predicted crossing time, and continuing to execute progressive stitching;

[0086] By triggering access to neighboring areas in advance, the monitoring range can be expanded, enabling predictive linkage rather than passive response. Through "event-driven cross-regional splicing", computing resources can be allocated on demand, and the monitoring range can be dynamically adjusted. The monitoring range can be dynamically expanded or contracted as the abnormal target moves, avoiding the waste of resources in global monitoring.

[0087] Example 3: The system also includes a historical database and an event tracing module. The event tracing module is communicatively connected to the historical database, and the historical database is connected to all the above modules to store the results output by all the above modules.

[0088] The core function of the event tracing module is to reconstruct the complete process of an abnormal event after it has ended. It is used to retrieve the corresponding sub-panoramic video image segments from the historical database based on the list of all regions marked as event areas and the time period after the abnormal event ends, and to reassemble them offline to generate a complete panoramic retrospective video of the event for post-event evidence reconstruction and analysis.

[0089] In summary, the campus monitoring area is divided into multiple sub-monitoring areas and each sub-panorama is stitched together independently. Deep learning is integrated into the entire chain of "anomaly perception → trajectory prediction → sub-panorama stitching decision → execution": real-time analysis of sub-panorama video identifies abnormal targets, marks event areas, predicts the predicted safe distance sequence of abnormal targets crossing the boundaries of event areas, obtains the predicted crossing time through the predicted safe distance sequence, and generates a neighboring area linkage monitoring signal when the predicted crossing time is less than a preset risk time threshold. This is based on a trigger event-driven progressive sub-panorama fusion stitching, and uses deep reinforcement learning to dynamically adjust the stitching range.

[0090] This enables a fundamental shift in global monitoring resources from "one-time full splicing" to "on-demand layered splicing + event-driven expansion + dynamic shrinking and release," reducing computing load and bandwidth consumption. At the same time, it achieves an upgrade from passive linkage to proactive predictive adaptive linkage, eliminating monitoring blind spots and field of view jumps in advance, and dynamically allocating computing power according to resource utilization, comprehensively improving the real-time performance, scalability, and intelligence level of the campus security system.

[0091] The above description is merely a preferred embodiment of the present invention; however, the scope of protection of the present invention is not limited thereto; any equivalent substitutions or modifications made by those skilled in the art within the technical scope disclosed in the present invention, based on the technical solution and its improved concept, should be covered within the scope of protection of the present invention.

Claims

1. A campus panoramic security monitoring and management system integrating deep learning, characterized by: include: The area division and sub-panorama stitching module is used to divide the campus monitoring area into multiple sub-monitoring areas according to the geographical distribution of the campus. At least one camera is deployed in each sub-monitoring area. A virtual global coordinate system is established based on the global coordinate system in the preset school panoramic design map. The transformation matrix from the sub-area coordinate system to the virtual global coordinate system is pre-calibrated for each sub-monitoring area. The video images captured by the cameras in each sub-monitoring area are spatiotemporally stitched together to generate multiple sub-panorama video images. The abnormal event judgment and alarm module is used to perform real-time deep learning analysis on each sub-panoramic video image, identify targets and track trajectories, determine whether an abnormal event has occurred in the sub-monitoring area, mark it as an event area if so, lock the abnormal target for risk assessment and generate an alarm signal. The trajectory tracking and linkage monitoring module responds to alarm signals and uses a temporal deep learning network to obtain the predicted trajectory of the abnormal target within a preset time period based on the historical trajectory sequence of the abnormal target. Based on the predicted trajectory, it calculates the predicted safe distance and predicted crossing time between the abnormal event target and the edge of the event area, generates linkage monitoring signals for neighboring areas, and determines at least one sub-monitoring area that is geographically adjacent to the event area as a shared monitoring area. The progressive panoramic stitching module is used to map the sub-panoramic video images of the shared monitoring area and the event area to the virtual global coordinate system after triggering the linkage monitoring signal of the neighboring area. The stitching boundary is determined according to the spatial adjacency relationship between the two in the virtual global coordinate system. At the stitching boundary, multi-band fusion, virtual canvas stitching, or three-dimensional depth map parallax compensation is used for fusion transition to generate a panoramic video stitched image. A deep learning-driven dynamic range adjustment strategy is used to adjust the range of the sub-monitoring area participating in real-time stitching for continuous tracking.

2. The campus panoramic security monitoring and management system integrating deep learning as described in claim 1, characterized in that: The region division module divides the monitoring regions according to at least one of the following criteria: The campus is divided into geographical zones, including the school gate area, teaching building area, playground area, dormitory area, canteen area, and perimeter wall area; the physical deployment locations of cameras and the degree of overlap in the coverage areas of each camera; and the floor and room layout of campus buildings.

3. The campus panoramic security monitoring and management system integrating deep learning according to claim 2, characterized in that: The sub-panorama stitching module uses a real-time stitching algorithm based on GPU parallel pipelines to spatiotemporally stitch together video images captured by cameras in each sub-monitoring area.

4. The campus panoramic security monitoring and management system integrating deep learning according to claim 3, characterized in that: The specific process for determining whether an abnormal event has occurred within a sub-monitoring area includes: S1. Use a target detection and tracking model to detect people in each frame of the sub-panoramic video image, assign a stable cross-frame ID to each detected person, and record the motion trajectory; S2. The pose estimation model is used to extract the coordinates of key points of the human body in real time. The pose features are identified by calculating the angle between key points. The pose feature sequence of N consecutive frames is input into the pre-trained temporal behavior recognition model, and the abnormal behavior type and confidence level are output. The types include abnormal behaviors such as fighting, falling, climbing and crossing, intrusion into dangerous areas, gathering of people and prolonged stay. S3. When the confidence level of a certain type of abnormal behavior exceeds the preset threshold, it is determined that an abnormal event has occurred in the current sub-monitoring area, marked as an event area, and the target is marked as an abnormal target.

5. The campus panoramic security monitoring and management system integrating deep learning according to claim 4, characterized in that: The specific process for conducting risk assessments and generating alarm signals for abnormal targets includes: T1. Locate the abnormal target in the event area, record the event start time, abnormal target ID, abnormal behavior type, and event area location, and summarize them into abnormal event content. T2. Extract multi-dimensional risk features based on the content of abnormal events, including behavioral risk coefficient, target identity coefficient, dynamic trend coefficient, regional sensitivity coefficient, and spatiotemporal background coefficient. T3. A weighted scoring model is used to calculate the risk score, and the risk level is determined based on the risk score, including low risk, medium risk, and high risk. Multi-level alarm signals are matched according to the risk level.

6. The campus panoramic security monitoring and management system integrating deep learning according to claim 5, characterized in that: The specific process for generating neighboring area linkage monitoring signals includes: Obtain the historical trajectory point sequence of the abnormal target within the past T seconds, input the historical trajectory point sequence into a pre-trained temporal deep learning network, which adopts the Transformer architecture, and output the predicted trajectory point sequence within the future prediction time. Obtain the mathematical representation of the boundary line of the current event region, L_boundary. For each predicted point in the predicted trajectory point sequence, calculate the shortest Euclidean distance from the point to the boundary of the current event region and mark it as the safe distance to obtain the predicted safe distance sequence. In the predicted safe distance sequence, it is detected whether there is a future moment when the safe distance is less than or equal to the preset linkage triggering distance threshold. If so, the first future arrival time that satisfies the condition of safe distance less than or equal to the preset linkage triggering distance threshold is recorded, and the time difference between the future arrival time and the current time is calculated and marked as the predicted crossing time. When the predicted crossing time is less than the preset risk time threshold, a neighboring area linkage monitoring signal is generated.

7. The campus panoramic security monitoring and management system integrating deep learning according to claim 6, characterized in that: The specific process for adaptively adjusting the range of sub-monitoring areas participating in real-time stitching for continuous tracking includes: A deep reinforcement learning range adjustment model is constructed. The model takes the number of sub-regions in the current stitching range, the predicted safe distance sequence of abnormal targets, and the resource occupancy rate of edge computing nodes as state inputs, and expands the stitching range, shrinks the stitching range, and maintains the current stitching range as action outputs. The resource occupancy rate here includes at least one of GPU utilization, video memory occupancy rate, and memory occupancy rate. During real-time tracking, corresponding operations are performed based on the actions output by the model: If the output is to expand the stitching range, then according to the predicted movement direction of the abnormal target, the shared monitoring area is added to the current stitching range, triggering the fusion and stitching of the sub-panoramic video images of the shared monitoring area and the sub-panoramic video images of the current event area; If the output is to reduce the stitching range, then perform a cross-region switching operation, remove the historical event region, remove it from the stitching, release computing resources, and update the shared monitoring region to the new event region; If the output is to maintain the current splicing range, then the existing splicing area combination will remain unchanged.

8. The campus panoramic security monitoring and management system integrating deep learning according to claim 7, characterized in that: The triggering conditions for cross-region switching are as follows: when an abnormal target completely leaves the current event area and enters the shared monitoring area, the abnormal target is not detected for 10 consecutive frames in the current event area, is detected for 5 consecutive frames in the shared monitoring area, and the depth of the abnormal target crossing the boundary line of the shared monitoring area exceeds the preset linkage triggering distance threshold, the current event area is marked as a historical event area, causing it to exit real-time stitching, and the shared monitoring area is updated to a new event area.

Images