A multi-modal campus monitoring method and system based on target preprocessing

Abstract

The application discloses a kind of multi-modal campus monitoring method and system based on target preprocessing, the described method includes the following steps: step 1: receiving high-resolution original video stream;Step 2: small target, middle target and large target detection are carried out by YOLO-v8 algorithm;Step 3: instance segmentation is carried out by Mask R-CNN algorithm, and pixel-level target mask is generated;Step 4: adaptive ROI extraction is carried out according to target size, and image quality inspection is carried out, finally output high-quality ROI image, ROI metadata, target category information, confidence information;Step 5: high-quality ROI image and scene description text are input into trained multi-modal large model, image information and text information are fused, and inference is completed;Step 6: the output of multi-modal large model is processed in JSON format, and structural data is extracted.

Description

Technical Field

[0001] This invention relates to the field of intelligent monitoring technology, and in particular to a multimodal campus monitoring method and system based on target preprocessing. Background Technology

[0002] With the rapid development of artificial intelligence technology, intelligent monitoring systems are playing an increasingly important role in campus security management. Traditional campus monitoring systems mainly rely on manual monitoring and simple motion detection algorithms, which suffer from problems such as low efficiency, high false alarm rates, and inability to perform semantic understanding. In recent years, the rise of deep learning technology has brought new development opportunities to intelligent monitoring.

[0003] Currently, campus surveillance and identification technologies mainly involve the following directions: 1. Surveillance systems based on traditional target detection. This type of solution uses the YOLOv3 algorithm for target detection and combines it with facial recognition technology to monitor campus personnel. Technical features: uses YOLOv3 for target detection, integrates a facial recognition module, supports real-time video stream processing, and has basic alarm functions. Shortcomings: limited to basic target detection and facial recognition, lacks deep semantic understanding, cannot perform intelligent analysis of target behavior and scenes, and has poor performance in detecting small targets. 2. Solutions based on optimized small target detection. This type of solution addresses the problem of small target detection in surveillance scenarios by improving detection performance through feature fusion and attention mechanisms. Technical features: designed a multi-scale feature fusion network, introduced an attention mechanism to enhance small target features, verified detection performance on surveillance datasets, and supports real-time detection applications. Shortcomings: focuses only on target detection tasks, lacks subsequent intelligent analysis, does not incorporate the semantic understanding capabilities of large models, and cannot generate structured analysis reports. 3. Visual understanding solutions based on large models: Some research institutions and companies have begun to explore the application of large models such as Qwen-VL to monitoring and analysis, but most of them remain at the proof-of-concept stage. Technical characteristics: Utilizes the powerful visual understanding capabilities of large models, can generate natural language descriptions, has certain reasoning and analysis capabilities, and supports the understanding of complex scenes. Shortcomings: Directly inputting the entire image leads to huge computational overhead, lacks optimization design for monitoring scenarios, and does not fully utilize the preprocessing advantages of object detection.

[0004] In summary, the main shortcomings of campus surveillance recognition technology that need to be addressed are as follows: 1. Low accuracy in small target recognition: Surveillance cameras are often far away, and targets occupy a small portion of the image. Traditional methods generally have low accuracy in recognizing small targets, easily leading to missed detections. 2. Significant waste of computational resources: Existing solutions either directly input the entire surveillance image into the algorithm for processing or only perform simple target detection, failing to fully utilize regions of interest, resulting in a large amount of ineffective computation. 3. Lack of deep semantic understanding: Traditional surveillance systems mainly remain at the level of target detection and simple classification, unable to deeply understand and analyze target behavior and scene semantics. 4. Insufficient utilization of multimodal information: Existing solutions rarely effectively combine visual and textual information. Even when multimodal fusion is used, it remains at a shallow feature stitching level, lacking deep reasoning capabilities.

[0005] In view of this, this invention is hereby proposed. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing technologies by proposing a multimodal campus monitoring method and system based on target preprocessing. This method improves the accuracy of small target recognition, optimizes computational efficiency, enhances semantic understanding capabilities, and achieves efficient multimodal fusion.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: A multimodal campus monitoring method based on target preprocessing includes the following steps: Step 1: Receive the high-resolution raw video stream and preprocess the raw video stream.

[0008] Step 2: Detect small, medium, and large targets using the YOLO-v8 algorithm and output the detection bounding box information.

[0009] Step 3: Perform instance segmentation using the Mask R-CNN algorithm to generate pixel-level target masks, which are used to extract the boundaries of the target region.

[0010] Step 4: Perform adaptive ROI extraction based on the target size, and check the image quality. Finally, output a high-quality ROI image, ROI metadata, target category information, and confidence information.

[0011] Step 5: Input the high-quality ROI image and scene description text into the trained multimodal large model, fuse the image information and text information, and complete the inference.

[0012] Step 6: Format the output of the multimodal large model into JSON format, extract the structural data, and generate a recognition report containing target information and scene analysis.

[0013] Furthermore, in step 1, the preprocessing of the original video stream is completed through the following steps: Step 1.1: Sample the original video stream to the preset frame rate.

[0014] Step 1.2: Scale the keyframe image to a preset standard size.

[0015] Step 1.3: Normalize the scaled keyframe images and convert them into a standardized Tensor format.

[0016] Furthermore, in step 2, CSPDarknet53 is used as the backbone network to extract features from the image; a feature pyramid network is used for feature fusion; and 80×80 pixel detectors, 40×40 pixel detectors, and 20×20 pixel detectors are used to predict and output the detection box information of small-sized targets, medium-sized targets, and large-sized targets from the feature maps extracted from the backbone network and the feature pyramid network, respectively.

[0017] Furthermore, in step 3, the input information includes detection box information and feature map information; the region of interest features are extracted through the ROI Align operation, and a pixel-level target mask is generated through the segmentation head.

[0018] Furthermore, step 4 includes the following steps: Step 4.1: Obtain the preprocessed image from Step 1, obtain the detection box information from Step 2, and obtain the pixel-level target mask from Step 3.

[0019] Step 4.2: Calculate the final extracted ROI region size using the following formula: S_roi = α * S_target + β; In the formula, S_roi is the size of the final extracted ROI region, α is the dynamic scaling factor, S_target is the size of the detected target, and β is the baseline margin.

[0020] Step 4.3: Determine the final ROI boundary based on the calculated S_roi; perform boundary checks to ensure that the ROI does not exceed the boundary of the preprocessed image in Step 1; extract the ROI region from the preprocessed image in Step 1; combine with the segmentation mask to accurately locate the target region and achieve ROI region extraction.

[0021] Step 4.4: Perform image quality inspection, including sharpness detection, target integrity verification, and lighting condition check.

[0022] Step 4.5: Perform quality filtering based on the image quality inspection in Step 4.4, and then perform standardization processing.

[0023] Step 4.6: Output high-quality ROI image, target category information, and confidence information.

[0024] Furthermore, in step 4.2, for small targets, α=1.8, for medium targets, α=1.5, and for large targets, α=1.2; for complex scenes, β=20 pixels; and for simple scenes, β=10 pixels.

[0025] Furthermore, step 4.5 includes the following steps: Step 4.5.1: Obtain a quality score by weighting the results of sharpness detection, target integrity verification, and lighting condition check.

[0026] Step 4.5.2: Remove ROIs with quality scores below the threshold.

[0027] Step 4.5.3: Adjust the ROI that has passed the quality inspection to a uniform size and perform size standardization.

[0028] Step 4.5.4: Unify image format and color space, and perform format standardization processing.

[0029] Furthermore, in step 6, the target information in the identification report includes personnel target information and object target information; the personnel target information includes: basic attribute information and behavioral status information, the basic attribute information includes: identity characteristics, appearance description, and biometric characteristics, and the behavioral status information includes current actions, interactive behaviors, and abnormal behaviors; the object target information includes: item identification and environmental facilities.

[0030] Furthermore, in step 6, the scene analysis in the identification report includes environmental scene analysis and behavioral scene analysis; environmental scene analysis includes spatial environment and temporal characteristics; behavioral scene analysis includes gathering patterns and security situation assessment.

[0031] To achieve the above objectives, the present invention also employs the following technical solution: A multimodal campus monitoring system based on target preprocessing is used to execute the method described in any one of the present invention, comprising: a video acquisition layer, an edge computing layer, a storage layer, a network transmission layer, and a display and early warning layer.

[0032] Compared with the prior art, the beneficial effects of this invention are as follows: (1) Improved small target recognition accuracy: Through accurate target detection and segmentation, high-quality regions of interest are extracted, providing clearer target images for subsequent recognition.

[0033] (2) Optimized computational efficiency: The ROI preprocessing strategy is adopted, and only the key areas are analyzed in depth, which greatly reduces invalid computation and improves the processing speed.

[0034] (3) Enhanced semantic understanding ability: Utilizing the powerful reasoning ability of multimodal large models, deep semantic understanding of target attributes, behaviors and scenarios is achieved.

[0035] (4) Achieved efficient multimodal fusion: The accurately extracted ROI image and scene context are directly input into the large model to achieve deep multimodal information fusion. Attached Figure Description

[0036] Figure 1 This is a flowchart of a multimodal campus monitoring method based on target preprocessing.

[0037] Figure 2 This is an architecture diagram of a multimodal campus monitoring method based on target preprocessing. Detailed Implementation

[0038] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0039] Example 1: A multimodal campus monitoring method based on target preprocessing, such as Figure 1 As shown, it includes the following steps: Step 1: Receive the high-resolution raw video stream and preprocess the raw video stream.

[0040] In this embodiment, the original video stream is the original surveillance video, with a resolution of 1920×1080×3. Furthermore, H.264 / H.265 encoding format can be used to ensure a balance between video quality and transmission efficiency. Additionally, the surveillance video frames can be adjusted to a standard input size (typically 640×640 pixels), normalizing pixel values ​​from the [0,255] range to the [0,1] range to ensure a uniform input data format and improve the model's generalization ability.

[0041] In this embodiment, the preprocessing of the original video stream is completed in step 1 through the following steps: Step 1.1: Sample the original video stream to the preset frame rate.

[0042] In this embodiment, existing frame sampling techniques can be used to sample the original video stream to a preset frame rate, such as 30fps.

[0043] Step 1.3: Scale the keyframe image to a preset standard size.

[0044] In this embodiment, the preset standard resolution can be 640×640×3.

[0045] Step 1.4: Normalize the scaled keyframe images and convert them into a standardized Tensor format.

[0046] In this embodiment, the preprocessing steps 1.1 to 1.4 can significantly improve processing efficiency while ensuring image quality.

[0047] Step 2: Detect small, medium, and large targets using the YOLO-v8 algorithm and output the detection bounding box information.

[0048] In this embodiment, step 2 is to detect targets in the image, which could be people or objects. Step 2 performs rapid target detection to accurately locate targets such as people and vehicles in the surveillance footage.

[0049] In this embodiment, in step 2, CSPDarknet53 is used as the backbone network to perform primary and multi-level feature extraction on the image; a feature pyramid network is used for feature fusion; and 80×80 pixel detectors, 40×40 pixel detectors, and 20×20 pixel detectors are used to predict and output the detection box information of small-sized targets, medium-sized targets, and large-sized targets from the feature maps extracted from the backbone network and the feature pyramid network, respectively.

[0050] In this embodiment, CSPDarknet53 is used as the backbone network to extract features from the image, that is, to extract multi-level feature representations from the input image. CSPDarknet53 can effectively capture the semantic information and detailed features of the image. Feature pyramid network is used for feature fusion to fuse features at different scales, so that the network can process targets of multiple sizes at the same time and improve detection accuracy.

[0051] In this embodiment, an 80×80 pixel detection head is used to predict the detection box, class number, and confidence score of small targets from the feature maps extracted from the backbone network and feature pyramid network. High-resolution feature maps can retain more detailed information and are suitable for detecting small objects in images. A 40×40 pixel detection head is used to predict the detection box, class number, and confidence score of medium-sized targets from the feature maps extracted from the backbone network and feature pyramid network. Medium-resolution feature maps strike a balance between computational efficiency and detection accuracy and are suitable for detecting medium-sized objects. A 20×20 pixel detection head is used to predict the detection box, class number, and confidence score of large targets from the feature maps extracted from the backbone network and feature pyramid network. Low-resolution feature maps have a larger receptive field and can effectively detect large objects in images.

[0052] Step 3: Perform instance segmentation using the Mask R-CNN algorithm to generate pixel-level target masks, which are used to extract the boundaries of the target region.

[0053] In this embodiment, the target region is the region of interest to the user; extracting the boundary of the target region provides accurate target boundary information for subsequent processing. Step 3 uses Mask R-CNN for pixel-level segmentation to obtain the precise boundary of the target.

[0054] In this embodiment, in step 3, the input information includes detection box information and feature map information; the region of interest features are extracted through the ROIAlign operation, and a pixel-level target mask is generated through the segmentation head.

[0055] In this embodiment, the detection box information includes: bounding box coordinates, target category, and confidence score, wherein the bounding box coordinates are derived from the target detection box [x1, y1, x2, y2] of YOLO-v8; the feature map information includes: feature map extracted by the backbone network, multi-scale feature representation fused by the feature pyramid network, and the image after preprocessing in step 1.

[0056] In this embodiment, the ROI Align operation extracts the features of the region of interest, further obtaining the precise pixel-level boundary of the target based on YOLO-v8 target detection, rather than just a rectangular bounding box; a pixel-level target mask is generated by the segmentation head to distinguish different instances of the same category, generating an independent segmentation mask for each target; finally, the ROI is accurately located, providing precise target boundaries for subsequent region of interest (ROI) extraction, ensuring the accuracy of ROI extraction.

[0057] In this embodiment, steps 2 and 3 employ a dual detection mechanism of YOLO-v8 + Mask R-CNN to achieve pixel-level accurate ROI extraction, thus solving the problem of missed detection of small targets in traditional methods.

[0058] Step 4: Perform adaptive ROI extraction based on the target size, and check the image quality. Finally, output a high-quality ROI image, ROI metadata, target category information, and confidence information.

[0059] In this embodiment, step 4 extracts the region of interest (ROI) and removes irrelevant background interference.

[0060] In this embodiment, step 4 includes the following steps: Step 4.1: Obtain the preprocessed image from Step 1, obtain the detection box information from Step 2, and obtain the pixel-level target mask from Step 3.

[0061] In this embodiment, step 4.1 provides a complete input data foundation for ROI extraction; ensuring the correspondence between the detection box, segmentation mask and the original image.

[0062] Step 4.2: Calculate the final extracted ROI region size using the following formula: S_roi = α * S_target + β; In the formula, S_roi is the size of the final extracted ROI region; α is the dynamic scaling factor, ranging from 1.2 to 2.0, which controls the size of the ROI region relative to the target and adjusts it dynamically according to the target size; S_target is the size of the detected target, which can be the width or height of the target bounding box, or the area of ​​the target bounding box; β is the baseline margin, ranging from 10 to 20 pixels, which ensures that the ROI region contains sufficient contextual information.

[0063] In this embodiment, in step 4.2, α=1.8 for small targets, α=1.5 for medium targets, and α=1.2 for large targets.

[0064] In this embodiment, step 4.2 intelligently adjusts the ROI size based on the target features to optimize the balance between computational efficiency and information preservation. For small targets (<50 pixels, such as mobile phones, keys, small items, etc.), α=1.8 to retain more contextual information and avoid losing important details; for medium-sized targets (50-150 pixels, such as backpacks, books, medium-sized equipment, etc.), it is used to achieve a balance between contextual information and processing efficiency; α=1.5. For large targets (>150 pixels, such as people, vehicles, large equipment, etc.), α=1.2 to avoid excessively large ROIs and reduce computational burden.

[0065] In this embodiment, in step 4.2, β = 20 pixels for complex scenes and β = 10 pixels for simple scenes.

[0066] In a practical application example, assuming a person is detected: target size (S_target) = 100 pixels, dynamic scaling factor (α) = 1.5, baseline margin (β) = 15 pixels, then the ROI size is calculated as: S_roi = 1.5 * 100 + 15 = 165 pixels.

[0067] Step 4.3: Determine the final ROI boundary based on the calculated S_roi; perform boundary checks to ensure that the ROI does not exceed the boundary of the preprocessed image in Step 1; extract the ROI region from the preprocessed image in Step 1; combine with the segmentation mask to accurately locate the target region and achieve ROI region extraction.

[0068] In this embodiment, step 4.3 accurately extracts the image region containing the target and necessary context, providing candidate ROIs for subsequent quality inspection.

[0069] In this embodiment, when the calculated ROI exceeds the image boundary, the boundary check automatically adjusts it to the image edge; pixel alignment ensures that the ROI coordinates are integer pixel values; and the original aspect ratio of the target is maintained where possible.

[0070] Step 4.4: Perform image quality inspection, including sharpness detection, target integrity verification, and lighting condition check.

[0071] In this embodiment, the image gradient magnitude is calculated (e.g., a direct calculation method using simple Sobel or Laplacian operators), and sharpness calculation and detection are performed based on the gradient information to filter out low-quality ROIs caused by blur or motion, ensuring the image quality for subsequent analysis. The continuity of the mask boundaries is verified (e.g., by directly determining whether a mask exists around a mask pixel), and the integrity of the segmented mask is verified, followed by target integrity verification, to ensure that the extracted ROI contains complete target information and avoid recognition errors caused by target truncation. An image brightness histogram is calculated (e.g., using a specific calculation method, first performing color space conversion, then calculating the histogram), and lighting condition checks are performed based on the histogram's illumination analysis to ensure that the ROI has good lighting conditions and filter out low-quality images caused by lighting problems.

[0072] Step 4.5: Perform quality filtering based on the image quality inspection in Step 4.4, and then perform standardization processing.

[0073] In this embodiment, step 4.5 ensures that the ROI quality of the input multimodal large model is consistent, thereby improving the accuracy and stability of subsequent inference.

[0074] In this embodiment, step 4.5 includes the following steps: Step 4.5.1: Obtain a quality score by weighting the results of sharpness detection, target integrity verification, and lighting condition check.

[0075] Step 4.5.2: Remove ROIs with quality scores below the threshold.

[0076] Step 4.5.3: Adjust the ROI that has passed the quality inspection to a uniform size and perform size standardization.

[0077] Step 4.5.4: Unify image format and color space, and perform format standardization processing.

[0078] Step 4.6: Output high-quality ROI image, target category information, and confidence information.

[0079] In this embodiment, the high-quality ROI image is a target region image that has undergone quality inspection and standardization; the target category information comes from the category label in the detection stage; the confidence information can be obtained by weighting the detection confidence and quality score.

[0080] In this embodiment, the ROI boundary is dynamically adjusted according to the characteristics of the monitoring scene to ensure that the optimal ROI can be obtained in different complexity scenarios. Complex and simple scenarios can be distinguished based on the number of targets per unit area, background texture and color changes, and the degree of overlap between targets. In complex scenarios, β=20 pixels to retain more environmental context and improve recognition accuracy. For simple scenarios, β=10 pixels to reduce irrelevant background and improve processing efficiency.

[0081] In this embodiment, steps 4.1-4.6 dynamically calculate the ROI size based on the target size, extracting a larger area for small targets and a smaller area for large targets. Margins are adaptively adjusted according to scene complexity, with larger margins for complex scenes and smaller margins for simple scenes. Multi-dimensional quality assessment is performed to ensure that image sharpness, target integrity, and lighting conditions meet requirements.

[0082] Step 5: Input the high-quality ROI image and scene description text into the trained multimodal large model, fuse the image information and text information, and complete the inference.

[0083] In this embodiment, the high-quality ROI image input in step 5 is the output of step 4; the scene description text can be edited based on the predefined description template of campus monitoring, such as location information: location descriptions such as "teaching building corridor", "library reading room", "campus square", etc.; time information: time period descriptions such as "class time", "break time", "night", etc.; environmental status: environmental descriptions such as "crowded", "quiet environment", "sufficient light", etc.

[0084] Step 6: Format the output of the multimodal large model into JSON format, extract the structural data, and generate a recognition report containing target information and scene analysis.

[0085] In this embodiment, the extracted structural data may include target attributes, behavioral analysis, and risk assessment.

[0086] In this embodiment, in step 6, the target information in the identification report includes personnel target information and object target information; the personnel target information includes: basic attribute information and behavioral status information, the basic attribute information includes: identity features, appearance description, and biometric features, and the behavioral status information includes current actions, interactive behaviors, and abnormal behaviors; the object target information includes: item identification and environmental facilities.

[0087] In this embodiment, the basic attributes of the personnel target information include: identity features such as age group, gender, and height estimation; appearance descriptions such as clothing color, style, and accessories; biometric features such as body shape; and behavioral state information such as current actions such as standing, walking, sitting, and running; interactive behaviors such as talking to others, acting alone, and acting in groups; and abnormal behaviors such as loitering, moving quickly, and gathering.

[0088] In this embodiment, in step 6, the scene analysis in the identification report includes environmental scene analysis and behavioral scene analysis; environmental scene analysis includes spatial environment and temporal characteristics; behavioral scene analysis includes gathering patterns and security situation assessment.

[0089] In this embodiment, the spatial environment of the environmental scenario analysis includes venue types and spatial layouts. Venue types include classrooms, libraries, canteens, dormitories, playgrounds, etc.; spatial layouts include open spaces, enclosed spaces, and passageways; time characteristics include class time, breaks between classes, meal times, and rest times; the gathering patterns in the behavioral scenario analysis include normal gatherings, abnormal gatherings, and emergency evacuations; the security situation assessment includes risk levels and abnormal events. Risk levels include low risk, medium risk, high risk, and emergency situations; abnormal events include fighting, theft, and security hazards.

[0090] This embodiment of a multimodal campus monitoring method based on target preprocessing solves the problems in the prior art: (1) Improves the accuracy of small target recognition: Through accurate target detection and segmentation, high-quality regions of interest are extracted, providing clearer target images for subsequent recognition. (2) Optimizes computational efficiency: By adopting the ROI preprocessing strategy, only key regions are analyzed in depth, greatly reducing invalid computation and improving processing speed. (3) Enhances semantic understanding ability: Utilizing the powerful reasoning ability of the multimodal large model, a deep semantic understanding of target attributes, behaviors, and scenes is achieved. (4) Achieves efficient multimodal fusion: The accurately extracted ROI image and scene context are directly input into the large model to achieve deep multimodal information fusion.

[0091] This embodiment of a multimodal campus monitoring method based on target preprocessing has the following key technical innovations: (1) Precise ROI extraction and preprocessing: Unlike traditional full-image processing, this embodiment accurately extracts the target region (ROI) through target detection and segmentation, significantly reducing the computational load while retaining complete target information; ROI standardization ensures the consistency of input to the large model. (2) End-to-end multimodal fusion: Abandoning the complex feature fusion architecture, the ROI image and scene text are directly input into the multimodal large model, making full use of the inherent multimodal understanding capability of the multimodal large model to achieve deep semantic fusion.

[0092] The multimodal campus monitoring method based on target preprocessing in this embodiment has the following core advantages: (1) Efficiency advantage: ROI preprocessing reduces the computational complexity from O(H×W) to O(N×h×w), where N is the number of targets, which is usually much smaller than the number of pixels in the whole image. At the same time, the processing speed is increased by 60% by utilizing the batch processing function of the model. (2) Accuracy advantage: Accurate ROI extraction ensures the integrity of target information. Combined with the powerful understanding ability of the large model, the small target recognition rate is increased by 32%, and the overall accuracy reaches more than 97%.

[0093] The multimodal campus monitoring method based on target preprocessing in this embodiment has the following technical innovation effects: (1) Simplified process: skips the complex feature extraction steps and directly utilizes the understanding ability of the multimodal large model. (2) Balance between accuracy and efficiency: ensures both recognition accuracy and improves processing efficiency.

[0094] The performance test data of the multimodal campus monitoring method based on target preprocessing in this embodiment are shown in Table 1: Table 1 Performance Test Data Small target detection accuracy 97.3% Medium target detection accuracy 98.5% Large Target Detection accuracy 99.2% Average detection time Time per frame 45ms ROI extraction Target integrity 99.8% ROI extraction Background interference percentage <8% Large Model Inference Single target time 120ms Large Model Inference Semantic understanding accuracy 96.7%

[0095] To verify the effectiveness of the method in this embodiment, comparative experiments were conducted under the following hardware and testing environments.

[0096] Hardware environment: Server configuration: NVIDIA RTX 4090 GPU + 64GB RAM; Storage system: 2TB NVMe SSD + 10TB HDD; Network environment: Gigabit Ethernet, supporting multiple video streams; Edge device: Jetson AGX Xavier edge computing node.

[0097] Software environment: Operating System: Ubuntu 20.04 LTS; Deep Learning Framework: PyTorch 2.0+; Database: MySQL 8.0+ Redis 6.0; Web Framework: FastAPI + Vue.js.

[0098] The comparative data are shown in Table 2: Table 2 Comparison Data Small target recognition rate 65% 97% Increased by 32% Processing speed 100% 160% Increase by 60% False alarm rate 12% 4% Reduced by 66% underreporting rate 8% 3% Reduced by 62%

[0099] Comparative experimental data demonstrates that compared to traditional methods (referring to existing campus surveillance identification schemes that do not employ target preprocessing and multimodal large model technology), this embodiment shows significant improvements in several key indicators. The small target recognition rate increased from 65% to 97%, a 32 percentage point improvement, which is particularly important for long-distance monitoring scenarios. Processing speed increased by 60%, meaning more surveillance footage can be processed, improving monitoring efficiency. In terms of reliability, the false alarm rate decreased from 12% to 4%, and the false negative rate decreased from 8% to 3%, reductions of 66% and 62% respectively. These improvements significantly enhance system reliability and reduce the need for manual intervention. These data fully demonstrate the advantages of this invention in recognition accuracy, processing efficiency, and system reliability, enabling the system to better meet the actual needs of campus surveillance and provide more reliable security.

[0100] Example 2: A multimodal campus monitoring system based on target preprocessing is used to execute the method of Embodiment 1, such as... Figure 2 As shown, it includes: video acquisition layer, edge computing layer, storage layer, network transmission layer, and display and early warning layer.

[0101] In this embodiment, the video capture layer uses high-definition network cameras to capture surveillance footage in real time. These cameras support 1080P and 4K high-definition resolution and are equipped with infrared night vision, enabling clear shooting even in complete darkness. The cameras can rotate 360 ​​degrees to ensure comprehensive monitoring. They are also equipped with an autofocus system that automatically adjusts the focus to maintain a clear image at all times.

[0102] In this embodiment, the edge computing layer is primarily responsible for processing and analyzing the acquired video data. The edge computing layer is equipped with a high-performance GPU accelerator card for running complex deep learning algorithms. It also features an FPGA coprocessor for rapidly processing real-time data. Large memory capacity allows it to handle multiple tasks simultaneously, much like the human brain can think about multiple problems at once. To ensure this "brain" can operate continuously and efficiently, a cooling system is also included to ensure the device remains stable during extended operation.

[0103] In this embodiment, the storage layer is responsible for storing all monitoring data. The storage layer uses high-speed solid-state drives (SSDs) to store real-time data, much like human short-term memory, allowing for rapid access. It also includes large-capacity hard disk drives (HDDs) to store historical data, similar to human long-term memory, capable of preserving information for longer periods. To protect these important "memories," a data backup module is also included, regularly backing up important data, much like how humans record important events. A data encryption module acts like a lock, protecting this data from unauthorized access.

[0104] In this embodiment, the network transport layer is responsible for transmitting information between the various modules (layers). Equipped with a gigabit network interface, it can rapidly transmit large amounts of data, much like the rapid response of the human nervous system. The wireless transmission module allows for flexible system deployment, much like a human's ability to move freely. The data encryption module ensures secure transmission, much like a human protecting important information. The network redundancy design acts as a backup system, ensuring the system can continue to function normally even if one part fails.

[0105] In this embodiment, the display and warning layer is responsible for presenting information to the user and issuing warnings. Equipped with a multi-screen display system, it can simultaneously display multiple video feeds, much like a human can focus on multiple things at once. When the system detects an anomaly, it issues warnings through various means such as audible and visual alarms, SMS notifications, and email pushes, just as humans express urgency in different ways. The hierarchical management function for warning information allows the system to take different warning measures based on the severity of the situation, just as humans respond differently to different urgency levels. The system also provides warning history query and statistical analysis functions to help users understand the system's operation, just as humans summarize lessons learned.

[0106] The multimodal campus monitoring system based on target preprocessing in this embodiment has the following beneficial effects: (1) The modular design supports flexible deployment, and the parameters of each module (layer) can be adjusted according to actual needs to adapt to different campus monitoring scenarios and hardware configurations. (2) The five core modules (layers) are closely connected to form a complete end-to-end processing link, which effectively avoids the overall performance loss caused by independent optimization of modules (layers) in traditional solutions.

[0107] This embodiment of a multimodal campus monitoring system based on target preprocessing has the following core innovations: (1) Layered architecture design innovation: five layers with clear division of labor: video acquisition layer → edge computing layer → storage layer → network transmission layer → display and early warning layer; the network transmission layer, as the central hub, has bidirectional connections with the other four layers, forming a star topology structure. This design is more flexible and efficient than the traditional linear processing flow. (2) Hardware combination innovation: edge computing layer innovation with a heterogeneous computing architecture of GPU + FPGA coprocessor, large-capacity memory + perfect heat dissipation system to ensure continuous high-performance operation. (3) Storage architecture innovation: dual-layer storage strategy of SSD (short-term memory) + HDD (long-term memory), triple data security protection of real-time storage + regular backup + data encryption. (4) Network transmission innovation: hybrid transmission mode of wired + wireless; network redundancy design to ensure system reliability.

[0108] In this embodiment, the original monitoring video input in step 1 is completed by video acquisition, and the preprocessing is completed by the edge computing layer; steps 2 and 3 are completed by the edge computing layer; step 4 is completed by the edge computing layer and the storage layer; step 5 is completed by the edge computing layer; the JSON in step 6 is completed by the edge computing layer and the network transmission layer; and the report generation and transmission in step 7 are completed by the display and early warning layer, the storage layer, and the network transmission layer.

[0109] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A multi-modal campus monitoring method based on target pre-processing, characterized in that, Includes the following steps: Step 1: Receive the high-resolution raw video stream and preprocess the raw video stream; Step 2: Detect small, medium, and large targets using the YOLO-v8 algorithm and output the detection bounding box information; Step 3: Perform instance segmentation using the Mask R-CNN algorithm to generate pixel-level target masks, which are used to extract the boundaries of the target region; In Step 3, the input information includes detection box information and feature map information; Step 4: Perform adaptive ROI extraction based on the target size, and check the image quality. Finally, output a high-quality ROI image, ROI metadata, target category information, and confidence information. Step 5: Input the high-quality ROI image and scene description text into the trained multimodal large model, fuse the image information and text information, and complete the inference; Step 6: Format the output of the multimodal large model into JSON format, extract the structural data, and generate a recognition report containing target information and scene analysis; In step 2, CSPDarknet53 is used as the backbone network to extract features from the image. Feature fusion is performed using a feature pyramid network. Using 80×80 pixel detectors, 40×40 pixel detectors, and 20×20 pixel detectors, the detection bounding boxes for small, medium, and large targets are predicted and output from the feature maps extracted from the backbone network and feature pyramid network, respectively. Step 4 includes the following steps: Step 4.1: Obtain the preprocessed image from Step 1, obtain the detection box information from Step 2, and obtain the pixel-level target mask from Step 3; Step 4.2: Calculate the final extracted ROI region size using the following formula: S_roi = α * S_target + β; In the formula, S_roi is the size of the final extracted ROI region, α is the dynamic scaling factor, S_target is the size of the detected target, and β is the baseline margin. Step 4.3: Determine the final ROI boundary based on the calculated S_roi; perform boundary checks to ensure that the ROI does not exceed the boundary of the preprocessed image from Step 1; extract the ROI region from the preprocessed image from Step 1. By combining target masks, the target region can be accurately located, and the ROI region can be extracted. Step 4.4: Perform image quality inspection, including sharpness detection, target integrity verification, and lighting condition check; Sharpness calculation and detection are performed based on gradient information to filter out low-quality ROIs caused by blur or motion; the integrity of the target mask is verified by verifying the continuity of the mask boundary, and then the target integrity is verified to ensure that the extracted ROI contains complete target information. Calculate the image brightness histogram and perform illumination condition checks based on the histogram illumination analysis to ensure that the ROI has good illumination conditions; Step 4.5: Perform quality filtering and standardization based on the image quality inspection in Step 4.4; Step 4.6: Output high-quality ROI image, target category information, and confidence information; In step 4.2, for small targets, α = 1.8; for medium targets, α = 1.5; and for large targets, α = 1.

2. For complex scenes, β = 20 pixels; For simple scenarios, β = 10 pixels.

2. The multi-modal campus monitoring method based on target pre-processing according to claim 1, characterized in that, In step 1, the preprocessing of the original video stream is completed through the following steps: Step 1.1: Sample the original video stream to the preset frame rate; Step 1.2: Scale the keyframe image to a preset standard size; Step 1.3: Normalize the scaled keyframe images and convert them into a standardized Tensor format. 3.The multi-modal campus monitoring method based on target pre-processing of claim 1, wherein, In step 3, the input information includes detection box information and feature map information; the ROIAlign operation is used to extract the features of the region of interest, and the segmentation head is used to generate a pixel-level target mask.

4. The multimodal campus monitoring method based on target preprocessing according to claim 1, characterized in that, Step 4.5 includes the following steps: Step 4.5.1: Obtain a quality score by weighting the results of sharpness detection, target integrity verification, and lighting condition check; Step 4.5.2: Remove ROIs with quality scores below the threshold; Step 4.5.3: Adjust the ROI that has passed quality inspection to a uniform size and perform size standardization processing; Step 4.5.4: Unify image format and color space, and perform format standardization processing.

5. The multimodal campus monitoring method based on target preprocessing according to claim 1, characterized in that, In step 6, the target information in the identification report includes personnel target information and object target information; Personnel target information includes: basic attribute information and behavioral status information. Basic attribute information includes: identity characteristics, appearance description, and biometric characteristics. Behavioral status information includes current actions, interactive behaviors, and abnormal behaviors. Object target information includes: object recognition and environmental facilities.

6. The multimodal campus monitoring method based on target preprocessing according to claim 1, characterized in that, In step 6, the scene analysis in the identification report includes environmental scene analysis and behavioral scene analysis; environmental scene analysis includes spatial environment and temporal characteristics. Behavioral scenario analysis includes aggregation patterns and security situation assessment.

7. A multimodal campus monitoring system based on target preprocessing, characterized in that, The method for performing any one of claims 1-6 includes: a video acquisition layer, an edge computing layer, a storage layer, a network transmission layer, and a display and early warning layer.

Images