Multi-mode campus monitoring method and system based on target preprocessing

Abstract

The invention discloses a multi-mode campus monitoring method and system based on target preprocessing. The method comprises the following steps: step 1, receiving a high-resolution original video stream; 2, detecting a small target, a medium target and a large target through a YOLO-v8 algorithm; step 3, instance segmentation is carried out through a Mask R-CNN algorithm, and a pixel-level target mask is generated; 4, performing adaptive ROI extraction according to the size of the target, performing image quality inspection, and finally outputting a high-quality ROI image, ROI metadata, target category information and confidence information; 5, inputting the high-quality ROI image and the scene description text into the trained multi-modal large model, fusing image information and text information, and completing reasoning; and 6, performing JSON formatting processing on the output of the multi-modal large model, and extracting structural data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent monitoring, and particularly relates to a multi-modal campus monitoring method and system based on target preprocessing. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, intelligent monitoring systems play an increasingly important role in campus safety management. Traditional campus monitoring systems mainly rely on manual monitoring and simple motion detection algorithms, which have problems such as low efficiency, high false alarm rate, and inability to understand semantics. In recent years, the rise of deep learning technology has brought new development opportunities for intelligent monitoring.

[0003] Current campus monitoring and recognition technologies mainly involve the following directions: 1. Monitoring system based on traditional target detection, which uses YOLOv3 algorithm for target detection and combines face recognition technology to realize campus personnel monitoring; technical features: uses YOLOv3 for target detection, integrates face recognition module, supports real-time video stream processing, and has basic alarm function; shortcomings: limited to basic target detection and face recognition, lacks deep semantic understanding, cannot intelligently analyze target behavior and scene, and has poor detection effect on small targets. 2. Optimization scheme based on small target detection, which aims at the problem of small target detection in monitoring scenes and improves detection performance through feature fusion and attention mechanism; technical features: designs a multi-scale feature fusion network, introduces attention mechanism to enhance small target features, verifies detection effect on monitoring data set, and supports real-time detection application; shortcomings: only focuses on target detection task, lacks subsequent intelligent analysis, does not combine the semantic understanding ability of large models, and cannot generate structured analysis reports. 3. Visual understanding scheme based on large models, some research institutions and companies have begun to explore the application of Qwen-VL and other large models in monitoring analysis, but most of them are still at the stage of conceptual verification; technical features: uses the powerful visual understanding ability of large models, can generate natural language description, has certain reasoning and analysis ability, and supports understanding of complex scenes; shortcomings: direct input of full image leads to huge computational overhead, lacks optimization design for monitoring scenes, and does not fully utilize the preprocessing advantages of target detection.

[0004] In summary, the main shortcomings of campus surveillance recognition technology that need to be addressed are as follows: 1. Low accuracy in small target recognition: Surveillance cameras are often far away, and targets occupy a small portion of the image. Traditional methods generally have low accuracy in recognizing small targets, easily leading to missed detections. 2. Significant waste of computational resources: Existing solutions either directly input the entire surveillance image into the algorithm for processing or only perform simple target detection, failing to fully utilize regions of interest, resulting in a large amount of ineffective computation. 3. Lack of deep semantic understanding: Traditional surveillance systems mainly remain at the level of target detection and simple classification, unable to deeply understand and analyze target behavior and scene semantics. 4. Insufficient utilization of multimodal information: Existing solutions rarely effectively combine visual and textual information. Even when multimodal fusion is used, it remains at a shallow feature stitching level, lacking deep reasoning capabilities.

[0005] In view of this, this invention is hereby proposed. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing technologies by proposing a multimodal campus monitoring method and system based on target preprocessing. This method improves the accuracy of small target recognition, optimizes computational efficiency, enhances semantic understanding capabilities, and achieves efficient multimodal fusion.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: A multimodal campus monitoring method based on target preprocessing includes the following steps: Step 1: Receive the high-resolution raw video stream and preprocess the raw video stream.

[0008] Step 2: Detect small, medium, and large targets using the YOLO-v8 algorithm and output the detection bounding box information.

[0009] Step 3: Perform instance segmentation using the Mask R-CNN algorithm to generate pixel-level target masks, which are used to extract the boundaries of the target region.

[0010] Step 4: Perform adaptive ROI extraction based on the target size, and check the image quality. Finally, output a high-quality ROI image, ROI metadata, target category information, and confidence information.

[0011] Step 5: Input the high-quality ROI image and scene description text into the trained multimodal large model, fuse the image information and text information, and complete the inference.

[0012] Step 6: Format the output of the multimodal large model into JSON format, extract the structural data, and generate a recognition report containing target information and scene analysis.

[0013] Further, the preprocessing of the original video stream in step 1 is completed by the following steps: Step 1.1: Sample the original video stream to a preset frame rate.

[0014] Step 1.2: Scale the key frame image to a preset standard size.

[0015] Step 1.3: Normalize the scaled key frame image and convert it to a standardized Tensor format.

[0016] Further, in step 2, CSPDarknet53 is used as the backbone network to extract features from the image; a feature pyramid network is used for feature fusion; an 80x80 pixel detection head, a 40x40 pixel detection head, and a 20x20 pixel detection head are used to predict the detection box information of small, medium, and large size targets from the feature maps extracted from the backbone network and the feature pyramid network.

[0017] Further, in step 3, the input information includes detection box information and feature map information; ROI Align operation is used to extract region of interest features, and a segmentation head is used to generate pixel-level target masks.

[0018] Further, step 4 includes the following steps: Step 4.1: Obtain the preprocessed image of step 1, obtain the detection box information of step 2, and obtain the pixel-level target mask of step 3.

[0019] Step 4.2: Calculate the size of the final extracted ROI region by the following formula: S_roi=α*S_target+β; In the formula, S_roi is the size of the final extracted ROI region, α is a dynamic scaling coefficient, S_target is the size of the detected target, and β is a reference margin.

[0020] Step 4.3: Determine the final ROI boundary according to the calculated S_roi; boundary checking ensures that the ROI does not exceed the boundary of the preprocessed image of step 1; extract the ROI region from the preprocessed image of step 1; combine the segmentation mask to accurately locate the target region and achieve ROI region extraction.

[0021] Step 4.4: Perform image quality inspection, including clarity detection, target integrity verification, and lighting condition check.

[0022] Step 4.5: Perform quality filtering and standardization processing according to the image quality inspection of step 4.4.

[0023] Step 4.6: Output high-quality ROI images, target class information, and confidence information.

[0024] Further, in the step 4.2, for small-size targets, alpha=1.8, for medium-size targets, alpha=1.5, for large-size targets, alpha=1.2, for complex scenes, beta=20 pixels, and for simple scenes, beta=10 pixels.

[0025] Further, the step 4.5 includes the following steps: Step 4.5.1: obtaining a quality score according to the sharpness detection, target integrity verification, and lighting condition check weighting.

[0026] Step 4.5.2: removing ROIs with a quality score lower than a threshold.

[0027] Step 4.5.3: adjusting the ROIs passing the quality test to a uniform size for size standardization processing.

[0028] Step 4.5.4: unifying the image format and color space for format standardization processing.

[0029] Further, in the step 6, the target information in the identification report includes personnel target information and object target information; the personnel target information includes basic attribute information and behavior state information, the basic attribute information includes identity features, appearance description, and biological features, and the behavior state information includes current actions, interaction behaviors, and abnormal behaviors; and the object target information includes article identification and environmental facilities.

[0030] Further, in the step 6, the scene analysis in the identification report includes environmental scene analysis and behavior scene analysis; the environmental scene analysis includes spatial environment and time characteristics; and the behavior scene analysis includes aggregation patterns and security posture assessment.

[0031] To achieve the above-mentioned purposes, the application also adopts the following technical solutions: A multi-modal campus monitoring system based on target preprocessing is used to execute any one of the methods provided by the application, and includes a video acquisition layer, an edge computing layer, a storage layer, a network transmission layer, and a display and early warning layer.

[0032] Compared with the prior art, the application has the following advantages: (1) Improved small target recognition accuracy: through accurate target detection and segmentation, high-quality regions of interest are extracted to provide clearer target images for subsequent recognition.

[0033] (2) Optimized computing efficiency: the ROI preprocessing strategy is adopted to perform deep analysis only on key areas, greatly reducing invalid calculations and improving processing speed.

[0034] (3) Enhanced semantic understanding ability: Utilize the powerful reasoning ability of multi-modal large models to achieve deep semantic understanding of target attributes, behaviors, and scenes.

[0035] (4) Efficient multi-modal fusion is achieved: Directly input the accurately extracted ROI images and scene context into the large model to achieve deep multi-modal information fusion. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 A flowchart of a multi-modal campus monitoring method based on target preprocessing.

[0037] Figure 2 A block diagram of a multi-modal campus monitoring method based on target preprocessing. DETAILED DESCRIPTION

[0038] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all.

[0039] Embodiment one: A multi-modal campus monitoring method based on target preprocessing, as shown in Figure 1 includes the following steps: Step 1: Receive a high-resolution raw video stream and pre-process the raw video stream.

[0040] In this embodiment, the raw video stream is the original monitoring video, and the resolution can be 1920x1080x3. In addition, H.264 / H.265 encoding format can be used to ensure the balance between video quality and transmission efficiency. In addition, the monitoring video frame can be adjusted to a standard input size (usually 640x640 pixels), and the pixel value can be normalized from the range [0, 255] to the range [0, 1] to ensure uniform input data format and improve the generalization ability of the model.

[0041] In this embodiment, the pre-processing of the raw video stream in step 1 is completed by the following steps: Step 1.1: Sample the raw video stream to a preset frame rate.

[0042] In this embodiment, existing frame sampling techniques can be used to sample the raw video stream to a preset frame rate, such as 30fps.

[0043] Step 1.3: Scale the key frame image to a preset standard size.

[0044] In this embodiment, the preset standard resolution can be 640x640x3.

[0045] Step 1.4: Normalize the scaled key frame images and convert them to standardized Tensor format.

[0046] In this embodiment, the preprocessing steps of steps 1.1-1.4 can significantly improve processing efficiency while ensuring image quality.

[0047] Step 2: Detect small, medium, and large targets using the YOLO-v8 algorithm and output bounding box information.

[0048] In this embodiment, step 2 is used to detect targets in the image, which can be people or objects. Through step 2, fast target detection is performed to accurately find personnel, vehicles, and other targets in the monitoring image.

[0049] In this embodiment, in step 2, CSPDarknet53 is used as the backbone network to perform primary and multi-level feature extraction on the image; a feature pyramid network is used for feature fusion; an 80x80 pixel detection head, a 40x40 pixel detection head, and a 20x20 pixel detection head are used to extract feature maps from the backbone network and the feature pyramid network to predict and output detection box information for small, medium, and large targets, respectively.

[0050] In this embodiment, CSPDarknet53 is used as the backbone network to extract features from the input image, i.e., to extract multi-level feature representations from the input image. CSPDarknet53 can effectively capture semantic information and detailed features of the image; a feature pyramid network is used for feature fusion, which fuses features of different scales to enable the network to handle targets of multiple sizes simultaneously and improve detection accuracy.

[0051] In this embodiment, an 80x80 pixel detection head is used to predict and output detection boxes, class numbers, and confidence levels for small targets from the feature maps extracted by the backbone network and the feature pyramid network. High-resolution feature maps can retain more detailed information and are suitable for detecting small objects in images; a 40x40 pixel detection head is used to predict and output detection boxes, class numbers, and confidence levels for medium targets from the feature maps extracted by the backbone network and the feature pyramid network. Medium-resolution feature maps strike a balance between computational efficiency and detection accuracy, making them suitable for detecting medium-sized objects; a 20x20 pixel detection head is used to predict and output detection boxes, class numbers, and confidence levels for large targets from the feature maps extracted by the backbone network and the feature pyramid network. Low-resolution feature maps have a larger receptive field and can effectively detect large objects in images.

[0052] Step 3: Perform instance segmentation using the Mask R-CNN algorithm to generate pixel-level target masks for extracting the boundaries of target regions.

[0053] In this embodiment, the target region is the region of interest of the user; extracting the boundary of the target region provides accurate target boundary information for subsequent processing. Step 3 uses Mask R-CNN for pixel-level segmentation to obtain the accurate boundary of the target.

[0054] In this embodiment, in step 3, the input information includes bounding box information and feature map information; the ROIAlign operation is used to extract the region of interest feature, and the segmentation head is used to generate the pixel-level target mask.

[0055] In this embodiment, the bounding box information includes: boundary box coordinates, target category and confidence score, wherein the boundary box coordinates come from the target detection box [x1, y1, x2, y2] of YOLO-v8; the feature map information includes: the feature map extracted by the backbone network, the multi-scale feature representation fused by the feature pyramid network, and the image after preprocessing in step 1.

[0056] In this embodiment, the ROI Align operation is used to extract the region of interest feature, and on the basis of YOLO-v8 target detection, the accurate pixel-level boundary of the target is further obtained, not just the rectangular boundary box; the segmentation head is used to generate the pixel-level target mask, and different instance objects of the same category are distinguished, and an independent segmentation mask is generated for each target; finally, the ROI is accurately positioned, providing accurate target boundary for subsequent region of interest (ROI) extraction, ensuring the accuracy of ROI extraction.

[0057] In this embodiment, steps 2 and 3 use the YOLO-v8+Mask R-CNN dual detection mechanism to realize pixel-level accurate ROI extraction and solve the small target missing detection problem of traditional methods.

[0058] Step 4: Adaptive ROI extraction according to target size and image quality inspection, finally output high-quality ROI image, ROI metadata, target category information and confidence information.

[0059] In this embodiment, step 4 extracts the region of interest (ROI) and removes irrelevant background interference.

[0060] In this embodiment, step 4 includes the following steps: Step 4.1: Obtain the preprocessed image of step 1, obtain the bounding box information of step 2, and obtain the pixel-level target mask of step 3.

[0061] In this embodiment, step 4.1 provides complete input data basis for ROI extraction; ensures the correspondence between the detection box, the segmentation mask and the original image.

[0062] Step 4.2: Calculate the size of the final extracted ROI region by the following formula: S_roi = a * S_target + b; In the formula, S_roi is the size of the final extracted ROI region; a is the dynamic scaling coefficient, with a value range of 1.2-2.0, and its function is to control the size of the ROI region relative to the target, and to dynamically adjust according to the target size; S_target is the detected target size, which can be the width or height of the target bounding box, or the area of the target bounding box; b is the reference margin, with a value range of 10-20 pixels, and its function is to ensure that the ROI region contains sufficient context information.

[0063] In this embodiment, in step 4.2, for small size targets, a = 1.8, for medium size targets, a = 1.5, and for large size targets, a = 1.2.

[0064] In this embodiment, step 4.2 intelligently adjusts the ROI size according to the target characteristics, optimizing the balance between calculation efficiency and information retention. For small size targets (<50 pixels, such as mobile phones, keys, small items, etc.), a = 1.8 is used to retain more context information and avoid losing important details; for medium size targets (50-150 pixels, such as backpacks, books, medium-sized devices, etc.), a balance is achieved between context information and processing efficiency; for large size targets (>150 pixels, such as personnel, vehicles, large devices, etc.), a = 1.2 is used to avoid excessive ROI and reduce computational burden.

[0065] In this embodiment, in step 4.2, for complex scenes, b = 20 pixels; for simple scenes, b = 10 pixels.

[0066] In a practical application example, assuming a person is detected: target size (S_target) = 100 pixels, dynamic scaling coefficient (a) = 1.5, reference margin (b) = 15 pixels, then the ROI size calculation is: S_roi = 1.5 * 100 + 15 = 165 pixels.

[0067] Step 4.3: Determine the final ROI boundary according to the calculated S_roi; boundary check to ensure that the ROI does not exceed the boundary of the preprocessed image in step 1; extract the ROI region from the preprocessed image in step 1; combine the segmentation mask to accurately locate the target region and achieve ROI region extraction.

[0068] In this embodiment, step 4.3 accurately extracts the image region containing the target and necessary context, providing a candidate ROI for subsequent quality inspection.

[0069] In this embodiment, the boundary check automatically adjusts to the image edge when the calculated ROI exceeds the image boundary; the ROI coordinates are ensured to be integer pixel values through pixel alignment; and the original aspect ratio of the target is maintained as much as possible.

[0070] Step 4.4: Perform image quality inspection, including sharpness detection, target integrity verification, and illumination condition check.

[0071] In this embodiment, the image gradient magnitude is calculated (such as a direct calculation method: using a simple Sobel or Laplacian operator), the sharpness is calculated and detected based on the gradient information to filter low-quality ROIs caused by blur or motion, and the image quality for subsequent analysis is ensured; the integrity of the segmentation mask is verified by verifying the continuity of the mask boundary (such as a judgment method: directly judging whether there is a mask around the mask pixel), and the target integrity verification is performed to ensure that the extracted ROI contains complete target information and avoids recognition errors caused by target truncation; the image brightness histogram is calculated (such as a specific calculation method: color space conversion is performed first, and then histogram calculation), and the illumination condition check is performed based on the histogram analysis to ensure that the ROI has good illumination conditions and filters low-quality images caused by illumination problems.

[0072] Step 4.5: Perform quality filtering according to the image quality inspection of step 4.4, and perform standardization processing.

[0073] In this embodiment, step 4.5 ensures that the ROI quality of the input multi-modal large model is consistent, improving the accuracy and stability of subsequent reasoning.

[0074] In this embodiment, step 4.5 includes the following steps: Step 4.5.1: Obtain a quality score according to the weighted sharpness detection, target integrity verification, and illumination condition check.

[0075] Step 4.5.2: Remove ROIs with a quality score below a threshold.

[0076] Step 4.5.3: Adjust the ROIs that pass the quality inspection to a uniform size for size standardization processing.

[0077] Step 4.5.4: Uniformize the image format and color space for format standardization processing.

[0078] Step 4.6: Output high-quality ROI images, target category information, and confidence information.

[0079] In this embodiment, the high-quality ROI image is a target region image that has passed quality inspection and standardization; the target category information comes from the category label in the detection stage; the confidence information is obtained by integrating the detection confidence and the quality score weighting.

[0080] In this embodiment, the ROI boundary is dynamically adjusted according to the characteristics of the monitoring scene, ensuring that the optimal ROI can be obtained in different complexity scenes; the complexity scene and the simple scene can be distinguished according to the number of targets in unit area, background texture and color change, and the degree of overlap between targets; when the complexity scene, β = 20 pixels, is used to retain more environmental context and improve recognition accuracy; for simple scenes, β = 10 pixels, is used to reduce irrelevant background and improve processing efficiency.

[0081] In this embodiment, steps 4.1-4.6, based on target size, dynamically calculate ROI size, small target extracts larger area, large target extracts smaller area. The margin is adjusted adaptively according to the complexity of the scene, a larger margin is left for complex scenes, and a smaller margin is left for simple scenes. Multi-dimensional quality assessment is performed to ensure that image clarity, target integrity and lighting conditions meet the requirements.

[0082] Step 5: Input the high-quality ROI image and the scene description text into the trained multi-modal large model, fuse the image information and the text information, and complete the inference.

[0083] In this embodiment, the high-quality ROI image input in step 5 is the output of step 4; the scene description text can be edited based on the pre-defined description template of the campus monitoring, such as location description: "teaching building corridor", "library reading room", "campus square" and the like; time period description: "class time", "intermission", "night" and the like; environment description: "dense flow of people", "quiet environment", "sufficient light" and the like.

[0084] Step 6: Perform JSON formatting processing on the output of the multi-modal large model, extract the structured data, and generate an identification report containing target information and scene analysis.

[0085] In this embodiment, the extracted structured data can include target attributes, behavior analysis and risk assessment, etc.

[0086] In this embodiment, in step 6, the target information in the identification report includes personnel target information and object target information; the personnel target information includes: basic attribute information and behavior state information, the basic attribute information includes: identity features, appearance description, biological features, the behavior state information includes current action, interaction behavior, abnormal behavior; the object target information includes: article identification and environmental facilities.

[0087] In this embodiment, in the basic attributes of the personnel target information, the identity features include: age range, gender, height estimate; the appearance description includes: clothing color, style, accessory items; the biological features include: body posture characteristics; in the behavior state information of the personnel target information, the current actions include: standing, walking, sitting, running, etc.; the interaction behaviors include: talking with others, solo activities, group activities; the abnormal behaviors include: wandering, rapid movement, gathering, etc.

[0088] In this embodiment, in step 6, the scene analysis in the recognition report includes environmental scene analysis and behavior scene analysis; the environmental scene analysis includes spatial environment and time characteristics; the behavior scene analysis includes gathering mode and security posture assessment.

[0089] In this embodiment, the spatial environment of the environmental scene analysis includes place type and spatial layout, the place type includes: classroom, library, cafeteria, dormitory, playground, etc.; the spatial layout includes: open space, closed space, passage area; the time characteristics include: class time, break time, meal time, rest time; the gathering mode in the behavior scene analysis includes: normal gathering, abnormal gathering, emergency evacuation; the security posture assessment includes risk level and abnormal event, the risk level includes: low risk, medium risk, high risk, emergency state, the abnormal event includes: fighting, stealing behavior, safety hazard.

[0090] The multi-modal campus monitoring method based on target preprocessing of the embodiment solves the problems in the prior art: (1) improves the small target recognition accuracy: through accurate target detection and segmentation, high-quality regions of interest are extracted to provide clearer target images for subsequent recognition. (2) Optimizes the calculation efficiency: adopt ROI preprocessing strategy, only analyze the key area deeply, greatly reduce the invalid calculation, improve the processing speed. (3) Enhance the semantic understanding ability: use the powerful reasoning ability of multi-modal large model to realize the deep semantic understanding of target attribute, behavior and scene. (4) Realize efficient multi-modal fusion: directly input the accurately extracted ROI image and scene context into the large model to realize deep multi-modal information fusion.

[0091] The multi-modal campus monitoring method based on target preprocessing of the embodiment has the following key technical innovations: (1) Accurate ROI extraction and preprocessing: Different from traditional full image processing, the embodiment accurately extracts the target area (ROI) through target detection and segmentation, which significantly reduces the calculation amount while retaining the complete target information; ROI standardization ensures the consistency of the large model input. (2) End-to-end multi-modal fusion: Discard the complex feature fusion architecture, directly input the ROI image and scene text into the multi-modal large model, fully utilize the inherent multi-modal understanding ability of the multi-modal large model to realize deep semantic fusion.

[0092] The multi-modal campus monitoring method based on target preprocessing of the embodiment has the following core advantages: (1) efficiency advantage: ROI preprocessing reduces the computational complexity from O(HxW) to O(Nxh x w), where N is the number of targets, which is usually much smaller than the total number of pixels, and the processing speed is improved by 60% by using the batch processing function of the model. (2) Precision advantage: accurate ROI extraction ensures the integrity of target information, combined with the powerful understanding ability of the large model, the small target recognition rate is improved by 32%, and the overall accuracy rate is more than 97%.

[0093] The multi-modal campus monitoring method based on target preprocessing of the embodiment has the following technical innovation effects: (1) process simplification: skipping the complex feature extraction step, directly using the understanding ability of the multi-modal large model. (2) precision and efficiency: both recognition accuracy and processing efficiency are guaranteed.

[0094] The multi-modal campus monitoring method based on target preprocessing of the embodiment has the following technical innovation effects: (1) process simplification: skipping the complex feature extraction step, directly using the understanding ability of the multi-modal large model. (2) precision and efficiency: both recognition accuracy and processing efficiency are guaranteed. Table 1 Performance test data Test item Index Value Small target detection Accuracy rate 97.3% Medium target detection Accuracy rate 98.5% Large target detection Accuracy rate 99.2% Average detection time Time consumption per frame 45 ms ROI extraction Target integrity 99.8% ROI extraction Background interference proportion <8% Large model inference Single target time consumption 120 ms Large model inference Semantic understanding accuracy rate 96.7%

[0095] To verify the effectiveness of the method of the embodiment, comparative experiments were conducted under the following hardware and test environment.

[0096] Hardware environment: Server configuration: NVIDIA RTX 4090 GPU + 64GB memory; storage system: 2TB NVMe SSD + 10TB HDD; network environment: gigabit Ethernet, supporting multi-channel video stream: edge device: Jetson AGX Xavier edge computing node.

[0097] Software environment: Operating system: Ubuntu 20.04 LTS; deep learning framework: PyTorch 2.0+; database: MySQL 8.0+ Redis 6.0; Web framework: FastAPI + Vue.js.

[0098] The comparison data is shown in Table 2: Table 2 Comparison data Comparison item Traditional method The method of the embodiment Promotion effect Small target recognition rate 65% 97% Increased by 32% Processing speed 100% 160% Increased by 60% False positive rate 12% 4% Reduced by 66% False negative rate 8% 3% Reduced by 62%

[0099] The comparative experimental data show that: compared with the traditional method (referring to the existing campus monitoring recognition scheme without using target preprocessing and multi-modal large model technology), the embodiment has significant improvement in multiple key indicators. The small target recognition rate is increased from 65% to 97%, an increase of 32 percentage points, which is particularly important for long-distance monitoring scenes. The processing speed is increased by 60%, which means that more monitoring pictures can be processed, improving the monitoring efficiency. In terms of reliability, the false positive rate is reduced from 12% to 4%, and the false negative rate is reduced from 8% to 3%, a decrease of 66% and 62% respectively. These improvements greatly improve the reliability of the system and reduce the need for manual intervention. These data fully demonstrate the advantages of the invention in recognition accuracy, processing efficiency and system reliability, enabling the system to better meet the actual needs of campus monitoring and provide more reliable security.

[0100] Embodiment two: A multi-modal campus monitoring system based on target preprocessing is used to perform the method of embodiment one, as shown in Figure 2 The video acquisition layer, edge computing layer, storage layer, network transmission layer and display and early warning layer are included.

[0101] In this embodiment, the video acquisition layer captures monitoring pictures in real time through high-definition network cameras. These cameras support 1080P and 4K high-definition resolution, and are also equipped with infrared night vision function, which can clearly capture even in pitch-black night. The camera can rotate 360 degrees to ensure that there is no dead angle in monitoring. It is also equipped with an automatic focusing system that can automatically adjust the focal length to keep the picture clear at all times.

[0102] In this embodiment, the edge computing layer is mainly responsible for processing and analyzing the collected video data. The edge computing layer is equipped with high-performance GPU acceleration cards for running complex deep learning algorithms. It is also equipped with FPGA co-processors that can quickly process real-time data. The large capacity of memory allows it to handle multiple tasks simultaneously, just like the human brain can think about multiple problems at the same time. In order to ensure that this "brain" can work efficiently and continuously, it is also equipped with a cooling system to ensure that the device remains stable during long-term operation.

[0103] In this embodiment, the storage layer is responsible for saving all monitoring data. The storage layer uses high-speed solid state drives to store real-time data, just like human short-term memory, which can be accessed quickly. It is also equipped with large-capacity mechanical hard drives for storing historical data, just like human long-term memory, which can save information for a longer period of time. In order to protect these important "memories", it is also equipped with a data backup module that backs up important data regularly, just like humans record important things. The data encryption module is like a lock that protects these data from unauthorized access.

[0104] In this embodiment, the network transmission layer is responsible for transmitting information between various modules (layers). Equipped with a gigabit network interface, it can quickly transmit large amounts of data, just like the rapid response of the human nervous system. The wireless transmission module allows the system to be deployed flexibly, just like humans can move freely. The data encryption module ensures the security of transmission, just like humans protect important information. The network redundancy design is like a backup system, so that the system can still work normally even if a part fails.

[0105] In this embodiment, the display and warning layer is responsible for displaying information to users and issuing warnings. Equipped with a multi-screen display system, it can display multiple videos simultaneously, just like humans can focus on multiple things at the same time. When the system detects abnormal conditions, it will issue warnings through various means such as sound and light alarms, SMS notifications, email pushes, and other means, just like humans express emergency situations through different ways. The warning information hierarchical management function allows the system to take different warning measures according to the severity of the situation, just like humans take different measures according to the urgency of the situation. The system also provides warning history record query and statistical analysis functions to help users understand the system's operation, just like humans summarize lessons learned.

[0106] The multi-modal campus monitoring system based on target preprocessing of this embodiment has the following beneficial effects: (1) Modular design supports flexible deployment, and parameters of each module (layer) can be adjusted according to actual needs to adapt to different campus monitoring scenarios and hardware configurations. (2) The five core modules (layers) are closely connected to form a complete end-to-end processing link, effectively avoiding the overall performance loss caused by independent optimization of modules (layers) in traditional solutions.

[0107] The multi-modal campus monitoring system based on target preprocessing of this embodiment has the following core innovations: (1) Layered architecture design innovation: five clear divisions: video acquisition layer → edge computing layer → storage layer → network transmission layer → display and warning layer; the network transmission layer serves as the central hub and is connected to the other four layers in both directions, forming a star topology. This design is more flexible and efficient than traditional linear processing flow. (2) Hardware combination innovation: heterogeneous computing architecture of GPU + FPGA coprocessor edge computing layer innovation, large capacity memory + perfect cooling system to ensure continuous high performance operation. (3) Storage architecture innovation: dual-layer storage strategy of SSD (short-term memory) + HDD (long-term memory), triple data security guarantee of real-time storage + regular backup + data encryption. (4) Network transmission innovation: hybrid transmission mode of wired + wireless; network redundancy design ensures system reliability.

[0108] In the present embodiment, the original monitoring video input of step 1 is completed by video acquisition, and the preprocessing is completed by the edge computing layer; steps 2 and 3 are completed by the edge computing layer; step 4 is completed by the edge computing layer and the storage layer; step 5 is completed by the edge computing layer; the JSON of step 6 is completed by the edge computing layer and the network transmission layer; the report generation and transmission of step 7 are completed by the display and early warning layer, the storage layer and the network transmission layer.

[0109] The above merely describes a preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can make equivalent replacements or changes according to the technical solution and the inventive concept of the present application within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A multimodal campus monitoring method based on target preprocessing, characterized in that, Includes the following steps: Step 1: Receive the high-resolution raw video stream and preprocess the raw video stream; Step 2: Detect small, medium, and large targets using the YOLO-v8 algorithm and output the detection bounding box information; Step 3: Perform instance segmentation using the Mask R-CNN algorithm to generate pixel-level target masks, which are used to extract the boundaries of the target region; Step 4: Perform adaptive ROI extraction based on the target size, and check the image quality. Finally, output a high-quality ROI image, ROI metadata, target category information, and confidence information. Step 5: Input the high-quality ROI image and scene description text into the trained multimodal large model, fuse the image information and text information, and complete the inference; Step 6: Format the output of the multimodal large model into JSON format, extract the structural data, and generate a recognition report containing target information and scene analysis.

2. The multimodal campus monitoring method based on target preprocessing according to claim 1, characterized in that, In step 1, the preprocessing of the original video stream is completed through the following steps: Step 1.1: Sample the original video stream to the preset frame rate; Step 1.2: Scale the keyframe image to a preset standard size; Step 1.3: Normalize the scaled keyframe images and convert them into a standardized Tensor format.

3. The multimodal campus monitoring method based on target preprocessing according to claim 1, characterized in that, In step 2, CSPDarknet53 is used as the backbone network to extract features from the image. Feature fusion is performed using a feature pyramid network; detection bounding boxes for small, medium, and large targets are predicted and output from the feature maps extracted from the backbone network and the feature pyramid network using 80×80 pixel, 40×40 pixel, and 20×20 pixel detectors, respectively.

4. The multimodal campus monitoring method based on target preprocessing according to claim 1, characterized in that, In step 3, the input information includes detection box information and feature map information; the region of interest features are extracted through the ROI Align operation, and a pixel-level target mask is generated through the segmentation head.

5. The multimodal campus monitoring method based on target preprocessing according to claim 1, characterized in that, Step 4 includes the following steps: Step 4.1: Obtain the preprocessed image from Step 1, obtain the detection box information from Step 2, and obtain the pixel-level target mask from Step 3. Step 4.2: Calculate the final extracted ROI region size using the following formula: S_roi = α * S_target + β; In the formula, S_roi is the size of the final extracted ROI region, α is the dynamic scaling factor, S_target is the size of the detected target, and β is the baseline margin. Step 4.3: Determine the final ROI boundary based on the calculated S_roi; perform boundary checks to ensure that the ROI does not exceed the boundary of the preprocessed image in Step 1; extract the ROI region from the preprocessed image in Step 1; combine with the segmentation mask to accurately locate the target region and achieve ROI region extraction; Step 4.4: Perform image quality inspection, including sharpness detection, target integrity verification, and lighting condition check; Step 4.5: Perform quality filtering based on the image quality inspection in Step 4.4, and then perform standardization processing; Step 4.6: Output high-quality ROI image, target category information, and confidence information.

6. A multimodal campus monitoring method based on target preprocessing according to claim 5, characterized in that, In step 4.2, for small targets, α = 1.8; for medium targets, α = 1.5; and for large targets, α = 1.

2. For complex scenes, β = 20 pixels; For simple scenarios, β = 10 pixels.

7. A multimodal campus monitoring method based on target preprocessing according to claim 5, characterized in that, Step 4.5 includes the following steps: Step 4.5.1: Obtain a quality score by weighting the results of sharpness detection, target integrity verification, and lighting condition check; Step 4.5.2: Remove ROIs with quality scores below the threshold; Step 4.5.3: Adjust the ROI that has passed quality inspection to a uniform size and perform size standardization processing; Step 4.5.4: Unify image format and color space, and perform format standardization processing.

8. The multimodal campus monitoring method based on target preprocessing according to claim 1, characterized in that, In step 6, the target information in the identification report includes personnel target information and object target information; Personnel target information includes: basic attribute information and behavioral status information. Basic attribute information includes: identity characteristics, appearance description, and biometric characteristics. Behavioral status information includes current actions, interactive behaviors, and abnormal behaviors. Object target information includes: object recognition and environmental facilities.

9. A multimodal campus monitoring method based on target preprocessing according to claim 1, characterized in that, In step 6, the scene analysis in the identification report includes environmental scene analysis and behavioral scene analysis; environmental scene analysis includes spatial environment and temporal characteristics. Behavioral scenario analysis includes aggregation patterns and security situation assessment.

10. A multimodal campus monitoring system based on target preprocessing, characterized in that, The method for performing any one of claims 1-9 includes: a video acquisition layer, an edge computing layer, a storage layer, a network transmission layer, and a display and early warning layer.

Images