An intelligent patrol and guard system and method fusing a size model

Abstract

The present application relates to the technical field of intelligent inspection, in particular to a smart inspection and prevention system fusing large and small models, applied to real-time monitoring and alarm in the communication pipeline construction scene, the present application aims to solve the problems of high false alarm rate, difficult information retrieval and low report generation efficiency in the existing communication pipeline construction monitoring system, the present application realizes real-time detection and tracking of the construction site target, deep analysis of abnormal events, intelligent question and answer of video content, and automatic generation of construction audit report by combining the cooperative working mechanism of lightweight small model and multi-modal large model, thereby improving the intelligent level of communication pipeline construction monitoring.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent inspection, and particularly relates to an intelligent inspection and surveillance system fusing large and small models, which is applied to real-time monitoring and alarm in a communication pipeline construction scene. BACKGROUND

[0002] With the rapid development of communication networks, communication pipelines play an important role in modern society. However, during the construction process, due to improper operation of construction equipment or human factors, communication pipelines are often damaged, leading to communication interruption and service quality degradation, and further causing economic losses and reduced user experience. Therefore, how to implement effective real-time monitoring of communication pipelines during the construction process and timely warn potential threats has become a technical problem that needs to be solved at present.

[0003] Traditional inspection and surveillance methods mainly rely on manual inspection, which is not only inefficient but also costly, and it is difficult to achieve all-weather and all-around monitoring. Although video monitoring-based solutions have begun to be applied to construction scenes, there are still the following main problems:

[0004] 1. High false alarm rate: Existing video monitoring systems lack effective false alarm filtering mechanisms, and a large number of false alarms increase the workload of maintenance personnel and reduce the reliability of the system.

[0005] 2. Difficult information retrieval: In a large amount of video data, users have difficulty quickly finding information related to specific events or targets, and cannot efficiently perform content queries and analysis.

[0006] 3. Report generation is cumbersome: Traditional inspection relies on manual preparation of inspection reports, which is time-consuming and labor-intensive, and key information is easily missed, affecting the quality and efficiency of the inspection work.

[0007] Currently, some technical solutions attempt to improve the intelligent level of construction monitoring through target detection, radar sensors, or vehicle-mounted monitoring devices. For example, some systems identify intruders through electronic fences and trigger alarms, but are often limited to monitoring a single modality (such as video), resulting in a high false alarm rate; other systems rely on human body radar sensors for tracking, which improves local accuracy but lacks the ability to detect multiple targets in complex construction scenes; and vehicle-mounted monitoring solutions achieve local alarms through position and state data, but are difficult to fully cover the diverse monitoring needs of construction areas.

[0008] In summary, existing technologies cannot meet the comprehensive needs of real-time, high-precision, and multi-modal information analysis in communication pipeline construction monitoring, and there is an urgent need for an intelligent inspection and surveillance system that fuses large and small models to improve detection accuracy, reduce false alarm rates, and achieve efficient information retrieval and automatic report generation. SUMMARY

[0009] The present application aims to solve the problems of high false alarm rate, difficult information retrieval and low report generation efficiency in existing communication pipeline construction monitoring systems, and proposes an intelligent patrol and prevention system integrating large and small models. Through the cooperative working mechanism of combining lightweight small models and multi-modal large models, real-time detection and tracking of construction site targets, deep analysis of abnormal events, intelligent question and answer of video content, and automatic generation of construction audit reports are realized, thereby improving the intelligent level of communication pipeline construction monitoring.

[0010] The technical scheme of the intelligent patrol and prevention system integrating large and small models of the present application comprises:

[0011] The video analysis module is used to receive the video stream transmitted by the construction monitoring camera in real time through the small model of the small parameter target detection model, analyze the video content, and generate preliminary analysis results and key frames.

[0012] The multi-modal fusion analysis module is used to receive the preliminary analysis results and key frames of the video analysis module, combine the multi-modal large model to perform deep analysis on the target object and video content, and generate analysis results and alarm information of abnormal events.

[0013] The data management module is used to store the analysis results and alarm information generated by the multi-modal fusion analysis module, and respond to the query request of the user.

[0014] The alarm generation module is used to generate alarm information in real time according to the abnormal event analysis results generated by the multi-modal fusion analysis module, and send it to the terminal device through the network.

[0015] The video question and answer module is used to realize the retrieval and question and answer of the user to the video content based on natural language processing technology.

[0016] The audit report generation module is used to generate the patrol and audit report of the construction site based on the prompt word template and the stored analysis data.

[0017] Preferably, the video analysis module detects the target object in real time through the small model, which uses YOLOv5-nano to optimize the target detection task using an improved loss function, including boundary box regression, classification and target confidence, and the function formula is as follows:

[0018] The boundary box regression loss CIoU is used to optimize the position and size of the detection box, and its formula is:

[0019] ;

[0020] : the intersection over union of the predicted box and the true box

[0021] : Euclidean distance between the center of the predicted box and the center of the ground truth box;

[0022] : Diagonal length of the minimum bounding box containing the predicted box and the ground truth box;

[0023] : Weight for aspect ratio consistency;

[0024] : Parameters for adjusting vv.

[0025] Classification loss for optimizing the predicted probability of the target class, formula:

[0026] ;

[0027] : True label (1 or 0);

[0028] : Probability predicted by the model.

[0029] Confidence loss for optimizing the predicted target confidence, formula:

[0030] ;

[0031] : True confidence (1 means there is a target, 0 means there is no target);

[0032] : Predicted target confidence.

[0033] Rules of the screening function:

[0034] Based on the intersection-over-union of the detection box and the communication pipeline area, targets with a confidence score greater than 0.8 will be considered as valid results;

[0035] Based on the confidence score of the target, results with a confidence score greater than 0.8 will be passed as key frames;

[0036] The output data format includes the detection box position, target class, confidence, and timestamp;

[0037] The total loss function of the small model is the weighted sum of the above three parts:

[0038] ;

[0039] and are weight parameters used to adjust the balance between classification loss and confidence loss.

[0040] ​Preferably, wherein the multi-modal fusion analysis module is used to receive the preliminary analysis results and key frames generated by the video analysis module, and based on these data, a deep analysis is conducted by combining a multi-modal large model to generate the analysis results of abnormal events and screened key frames, and the deep abnormal event analysis is completed through the following steps:

[0041] A, input data format: bounding box position, target category, confidence from small model; screening results from video analysis module: key frame image and corresponding timestamp;

[0042] B, input feature fusion: based on the spatial features of target position and size, the time features of key frame time sequence, and the text features of target description, multi-modal feature fusion is conducted; the dynamic adjustment formula of multi-modal feature weight is:

[0043] ;

[0044] wherein, is the weight of feature , and is the score of feature correlation with the event.

[0045] C, deep inference and classification: the fused features are analyzed in depth by the multi-modal large model to determine the type, time and position of the abnormal event, and the category, behavior description and detailed classification results of the target are generated.

[0046] Preferably, wherein the alarm generation module sends the alarm information to the terminal device of the construction management personnel through the 5G network.

[0047] The intelligent monitoring method of the application comprises the following steps:

[0048] A, video receiving: acquiring video stream through the monitoring camera of the construction site;

[0049] B, target detection: analyzing the video stream by using a small model to detect specific target objects and their motion states, and generating key frames and preliminary analysis results;

[0050] C, multi-modal fusion analysis: based on the preliminary analysis results, a deep analysis is conducted by combining a multi-modal large model to generate detailed analysis results of abnormal events;

[0051] D, alarm generation: identifying potential abnormalities and generating alarm information according to the results of multi-modal analysis, and sending the alarm information to relevant personnel through the network;

[0052] E, video question and answer: based on natural language processing technology, supporting users to search and ask questions about video content;

[0053] F, audit report generation: generating a site inspection audit report based on the prompt word template and stored analysis data.

[0054] Preferably, the target detection step includes detecting the type and motion state of the construction equipment through a small model and generating key frames.

[0055] Preferably, the multi-modal fusion analysis step combines multi-modal large models based on key frame images and preliminary analysis results to make event judgments and generate alarm information.

[0056] Preferably, the audit report generation step includes structuring the stored data and generating a report according to the user-set prompt word template.

[0057] In this application, the small model used in target detection adopts YOLOv5-nano, which uses an improved loss function to optimize the target detection task, including boundary box regression, classification and target confidence. The function formula is as follows:

[0058] 1. Loss function formula

[0059] YOLOv5-nano uses an improved loss function to optimize the target detection task, including boundary box regression, classification and target confidence. The key parts are as follows:

[0060] 1.1. Boundary box regression loss

[0061] YOLOv5-nano uses Complete Intersection over Union (CIoU) or Generalized Intersection over Union (GIoU) as the boundary box loss function to optimize the overlap of the detection box and the real box.

[0062] Formula (CIoU loss):

[0063] ;

[0064] : the intersection over union of the predicted box and the real box;

[0065] : the Euclidean distance between the center of the predicted box and the center of the real box;

[0066] : the diagonal length of the smallest enclosing box containing the predicted box and the real box;

[0067] : the weight of the aspect ratio consistency;

[0068] : Adjust the parameters of vv.

[0069] 1.2. Classification loss

[0070] The classification loss usually adopts Binary Cross-Entropy (BCE) to optimize the prediction of the target class.

[0071] Formula:

[0072] ;

[0073] : True label (1 or 0);

[0074] : Probability predicted by the model.

[0075] 1.3. Confidence loss

[0076] The loss of target confidence uses Binary Cross-Entropy (BCE).

[0077] Formula:

[0078] ;

[0079] : True confidence (1 indicates the presence of a target, 0 indicates the absence of a target);

[0080] : Predicted target confidence.

[0081] Total loss function formula:

[0082] The total loss of YOLOv5-nano is the weighted sum of the above three parts:

[0083] ;

[0084] and are weight parameters used to adjust the influence of classification loss and confidence loss.

[0085] 2. Prediction formula of Anchor Boxes

[0086] YOLOv5-nano uses a combination of Anchor-free and Anchor-based methods to predict target boxes, generating prediction boxes based on center points and offsets.

[0087] Formula:

[0088] , ;

[0089] , : center coordinates of the predicted bounding box

[0090] : Sigmoid function to limit the predicted value to the range [0, 1]

[0091] , : offset of the network output

[0092] , : top-left corner coordinates of the current grid cell

[0093] Width and height of the predicted bounding box

[0094] ;

[0095] , : width and height of the anchor

[0096] , : scale offset of the network output

[0097] 3. Sigmoid activation function

[0098] YOLOv5-nano uses a Sigmoid activation function to limit the classification and confidence outputs, with the formula as follows:

[0099] ;

[0100] : network output value

[0101] : maps the network output value to [0, 1]

[0102] 4. Non-Maximum Suppression (NMS)

[0103] Used to remove redundant detection boxes and keep the target box with the highest confidence.

[0104] Formula:

[0105] ;

[0106] : two candidate boxes

[0107] : Intersection over Union of two boxes.

[0108] When the IoU exceeds a set threshold, remove the candidate box with lower confidence.

[0109] By adopting YOLOv5-nano, the entire technical solution can effectively meet the core needs of real-time, accuracy and adaptability in communication pipeline construction monitoring, while providing a technical foundation for the scalability of the system (such as collaboration with large models).

[0110] After adopting this set of algorithms using YOLOv5-nano, the system can achieve the following core purposes and technical effects:

[0111] Real-time: Adapt to the rapidly changing environment of the construction site, ensuring the timeliness of detection and alarm.

[0112] Accuracy: Significantly reduce false positive rate and missed detection rate, improve system reliability.

[0113] Adaptability: Run on low-power edge devices, reduce dependence on cloud computing resources.

[0114] Collaboration: Combine with multi-modal analysis and natural language processing modules to build an efficient intelligent inspection system.

[0115] By adopting YOLOv5-nano, the entire technical solution can effectively meet the core needs of real-time, accuracy and adaptability in communication pipeline construction monitoring, while providing a technical foundation for the scalability of the system (such as collaboration with large models).

[0116] The intelligent inspection and prevention system of the present application by fusing large and small models significantly improves the intelligent level of monitoring in the communication pipeline construction scene. Compared with the prior art, the following technical effects are mainly realized:

[0117] Real-time target detection and efficient processing:

[0118] Lightweight small models (such as YOLOv5-nano) are deployed on edge devices to detect construction equipment, personnel and their motion states in real time. The optimized design of edge devices enables the video analysis module to achieve millisecond-level response (such as 30 frames per second processing speed), significantly improving the real-time performance of the system in dynamic construction environments.

[0119] High precision and low false positive rate of abnormal event recognition:

[0120] The multi-modal fusion analysis module combines multi-modal large models, conducts deep analysis on target behaviors and events through deep learning and data fusion technology. Based on the improved CIoU / GIoU loss function and multi-modal event judgment algorithm, the detection accuracy of abnormal events is improved, and the false positive rate and missed detection rate are significantly reduced.

[0121] Timeliness and reliability of alarm information:

[0122] After detecting potential threats or abnormal events on the construction site, the system can generate and send alarm information in real time through wireless networks such as 5G, ensuring that construction managers respond in a timely manner and reducing the risk of communication pipeline damage. The multi-layer alarm filtering mechanism further improves the accuracy of the alarm and reduces unnecessary interference.

[0123] Intelligent video content retrieval:

[0124] With the help of the video question and answer module, users can accurately retrieve video content through natural language. The system supports quick positioning and information return for specific targets, events or time periods, greatly improving the efficiency of massive video data utilization and the convenience of information retrieval.

[0125] Automatic audit report generation:

[0126] The audit report generation module automatically integrates stored analysis results into structured reports based on prompt word templates. Automated report generation avoids the inefficiency and error-prone problems of traditional manual writing, improving the quality and efficiency of audit work.

[0127] Flexible edge computing adaptability:

[0128] The system supports multiple edge devices (such as NVIDIA Jetson, Huawei Ascend, etc.), and through model pruning, quantization and other lightweight technologies, it adapts to the power limitations of different devices. The design of edge computing and cloud cooperation reduces the dependence on high-bandwidth networks, ensuring the stability of the system in unstable network environments.

[0129] Multi-scenario applicability and scalability:

[0130] This invention is not only applicable to communication pipeline construction scenarios, but also can be extended to other application fields that require target detection, information retrieval and report generation, such as infrastructure construction monitoring, factory equipment operation management and public safety inspection, etc.

[0131] Reduce construction management costs:

[0132] The system reduces the workload of manual inspection and audit through intelligent means, reducing labor costs, and through precise abnormal detection, reducing the risk of construction accidents and communication pipeline damage, thereby reducing the overall maintenance cost.

[0133] The application improves the quality of target detection results by optimizing the loss function of the small model, so that the generated key frames and preliminary analysis results have high confidence and accuracy. This optimization significantly reduces the computational burden of the multi-modal large model, improving the accuracy and efficiency of subsequent abnormal event analysis. For example, the target data filtered by IoU reduces more than 50% of redundant information, making the classification accuracy of abnormal events increase to 98%. The combination of loss function optimization and small model cooperation ensures the real-time and accuracy of the system, significantly reducing the false alarm rate of on-site construction monitoring. BRIEF DESCRIPTION OF DRAWINGS

[0134] Figure 1 is a structural block diagram of the application;

[0135] Figure 2 is a step schematic diagram of the application. DETAILED DESCRIPTION

[0136] Example 1:

[0137] A technical scheme of an intelligent inspection and prevention system fusing a small model includes:

[0138] A video analysis module is used to receive video streams transmitted by construction monitoring cameras in real time through a small model of a small parameter target detection model, analyze video content, and generate preliminary analysis results and key frames;

[0139] A multi-modal fusion analysis module is used to receive preliminary analysis results and key frames from the video analysis module, combine multi-modal large models to perform in-depth analysis on target objects and video content, and generate analysis results and alarm information of abnormal events;

[0140] A data management module is used to store analysis results and alarm information generated by the multi-modal fusion analysis module, and respond to user query requests;

[0141] An alarm generation module is used to generate alarm information in real time based on abnormal event analysis results generated by the multi-modal fusion analysis module, and send it to terminal equipment through a network;

[0142] A video question and answer module is used to realize user retrieval and question and answer of video content based on natural language processing technology;

[0143] An audit report generation module is used to generate an inspection and audit report of the construction site based on a prompt word template and stored analysis data.

[0144] Example 2:

[0145] The difference between this embodiment and example 1 is that this embodiment further includes:

[0146] The video analysis module detects target objects in real time through small models, which use YOLOv5-nano to optimize the target detection task using an improved loss function, including boundary box regression, classification, and target confidence,

[0147] The target detection of the video analysis module is accurately optimized by the optimized loss function, specifically including:

[0148] 1) The overlap between the detection box and the real target is optimized by the boundary box regression loss to ensure that the position and size of the detection box accurately fit the target;

[0149] 2) The classification loss improves the accuracy of target class prediction to ensure the consistency of the class of the detection result;

[0150] 3) The confidence loss optimizes the confidence score of the target existence, and filters out targets with a confidence greater than 0.8.

[0151] Based on the above optimization results, the video analysis module filters out key frames and preliminary analysis results related to the safety of the communication pipeline, including the accurate position of the target (IoU>0.5), the class (such as excavators, construction personnel), and the confidence score (such as confidence>0.8). These high-quality data after filtering are passed to the multi-modal fusion analysis module for further multi-modal inference of abnormal events in combination with video key frames, time series data, and text information.

[0152] The function formula used is as follows:

[0153] Boundary box regression loss CIoU, used to optimize the position and size of the detection box, its formula is:

[0154] ;

[0155] : Intersection over Union of predicted box and real box;

[0156] : Euclidean distance between the center of the predicted box and the center of the real box;

[0157] : Diagonal length of the minimum enclosing box containing the predicted box and the real box;

[0158] : Aspect ratio consistency weight;

[0159] : Adjust the parameters of vv.

[0160] Classification loss, used to optimize the prediction probability of the target class, formula:

[0161] ;

[0162] : true label (1 or 0);

[0163] : model predicted probability.

[0164] Confidence loss, used to optimize the prediction of target confidence, formula:

[0165] ;

[0166] : true confidence (1 means target, 0 means no target);

[0167] : predicted target confidence.

[0168] The rules of the screening function are:

[0169] Based on the intersection-over-union of the detection box and the communication pipeline area, The target is considered as a valid result; Based on the confidence score of the target, the result with a confidence greater than 0.8 is considered as a key frame;

[0170] The output data format includes detection box position, target category, confidence, and timestamp;

[0171] The total loss function of the small model is the weighted sum of the above three parts:

[0172]

[0173] ;

[0174] and are weight parameters, used to adjust the balance between classification loss and confidence loss. Preferably, the multi-modal fusion analysis module is used to receive the preliminary analysis results and key frames generated by the video analysis module, and based on these data, a deep analysis is performed in combination with a multi-modal large model to generate analysis results of abnormal events and screened key frames, and the deep abnormal event analysis is completed through the following steps:

[0175] A, input data format: detection box position, target category, confidence from the small model; screening results from the video analysis module: key frame image and corresponding timestamp;

[0176]

[0177] ​B. Input feature fusion: based on the spatial features of the target position and size, the temporal features of the keyframe time series, and the text features of the target description, multi-modal feature fusion is performed; the dynamic adjustment formula for multi-modal feature weight is:

[0178] ;

[0179] wherein, is the weight of the feature , and is the score of the feature correlation with the event;

[0180] C. Deep reasoning and classification: through multi-modal large model, the fused features are deeply analyzed to determine the type, time and location of the abnormal event, and the category, behavior description and detailed classification result of the target are generated.

[0181] Preferably, wherein the alarm generation module sends the alarm information to the terminal device of the construction management personnel through the 5G network.

[0182] Embodiment 3:

[0183] The intelligent monitoring method of the present embodiment comprises the following steps:

[0184] A. Video reception: acquire video stream through the monitoring camera of the construction site;

[0185] B. Target detection: analyze the video stream using a small model to detect specific target objects and their motion states, generate keyframes and preliminary analysis results;

[0186] C. Multi-modal fusion analysis: based on the preliminary analysis results, combine multi-modal large model for deep analysis to generate detailed analysis results of abnormal events;

[0187] D. Alarm generation: identify potential abnormalities and generate alarm information according to the results of multi-modal analysis, and send it to relevant personnel through the network;

[0188] E. Video question and answer: based on natural language processing technology, support users to search and ask questions about video content;

[0189] F. Generate inspection and audit report: based on the prompt word template and the stored analysis data, generate the inspection and audit report of the construction site.

[0190] Embodiment 4:

[0191] The difference between the present embodiment and embodiment 3 is that the present embodiment further comprises:

[0192] The target detection step comprises detecting the type and motion state of the construction equipment through a small model, and generating keyframes.

[0193] Preferably, the multi-modal fusion analysis step is based on key frame images and preliminary analysis results, combined with a multi-modal large model to make event determination and generate alarm information.

[0194] Preferably, the audit report generation step includes structured processing of stored data according to user-set prompt word templates and generating reports.

[0195] In the target detection in this application, the small model YOLOv5-nano uses an improved loss function to optimize the target detection task, including boundary box regression, classification and target confidence. The function formula used is as follows:

[0196] 1. Loss function formula

[0197] YOLOv5-nano uses an improved loss function to optimize the target detection task, including boundary box regression, classification and target confidence. The key parts are as follows:

[0198] 1.1. Boundary box regression loss

[0199] YOLOv5-nano uses Complete Intersection over Union (CIoU) or Generalized Intersection over Union (GIoU) as the boundary box loss function to optimize the overlap of the detection box and the real box.

[0200] Formula (CIoU loss):

[0201] ;

[0202] : Intersection over Union of predicted box and real box;

[0203] : Euclidean distance between the center of the predicted box and the center of the real box;

[0204] : Diagonal length of the smallest enclosing box containing the predicted box and the real box;

[0205] : Weight of aspect ratio consistency;

[0206] : Adjust the parameters of vv.

[0207] 1.2. Classification loss

[0208] The classification loss usually adopts Binary Cross-Entropy (BCE) to optimize the prediction of the target class.

[0209] Formula:

[0210] ;

[0211] : True label (1 or 0);

[0212] : Probability predicted by the model.

[0213] The confidence loss is used to optimize the prediction of the target confidence, and the formula is:

[0214] ;

[0215] : True confidence (1 indicates the presence of a target, and 0 indicates the absence of a target);

[0216] : Predicted target confidence.

[0217] Total loss function formula:

[0218] The total loss of YOLOv5-nano is the weighted sum of the above three parts:

[0219] ;

[0220] and are weight parameters used to adjust the influence of the classification loss and the confidence loss.

[0221] 2. Prediction formula of Anchor Boxes

[0222] YOLOv5-nano uses a combination of Anchor-free and Anchor-based methods to predict target boxes, generating prediction boxes based on the center point and offset.

[0223] Formula:

[0224] , ;

[0225] , : Center coordinates of the prediction box;

[0226] : Sigmoid function, used to limit the prediction value to the range [0, 1];

[0227] , : offset of network output

[0228] , : top-left coordinate of current grid cell

[0229] width and height of predicted box

[0230] ;

[0231] , : width and height of Anchor

[0232] , : scale offset of network output

[0233] 3. Sigmoid activation function

[0234] YOLOv5-nano uses Sigmoid activation function to limit the classification and confidence output, the formula is as follows:

[0235] ;

[0236] : value of network output

[0237] : map the value of network output to [0, 1]

[0238] 4. Non-Maximum Suppression (NMS)

[0239] Used to remove redundant detection boxes and keep the target box with the highest confidence.

[0240] Formula:

[0241] ;

[0242] : two candidate boxes

[0243] : IoU of two boxes

[0244] When IoU exceeds the set threshold, remove the candidate box with lower confidence.

[0245] Example 5:

[0246] In a certain communication pipeline construction site, in order to protect the safety of the communication pipeline, an intelligent inspection and prevention system based on large and small models is deployed, and the specific implementation is as follows:

[0247] 1. Video reception and preprocessing

[0248] The monitoring camera installed at the construction site transmits real-time video stream to the video analysis module of the edge device through wireless network (such as WiFi or 5G). After preprocessing, the video stream is used as input data for target detection by small model (YOLOv5-nano).

[0249] 2. Target detection and loss function optimization

[0250] The video analysis module uses a small parameter target detection model to accurately optimize the target detection results through an optimized loss function, which includes:

[0251] 1) Boundary box regression loss optimization: Use complete intersection over union (CIoU) loss to optimize the overlap between the predicted box and the real box, the formula is:

[0252] ;

[0253] Where ρ represents the Euclidean distance between the center points of the predicted box and the real box, c is the diagonal length of the minimum bounding box, and α and v are adjustment parameters and width-height ratio consistency weights, respectively.

[0254] 2) Classification loss optimization: Use binary cross-entropy loss (BCE) to optimize the prediction accuracy of target class, the formula is:

[0255] ;

[0256] Where y represents the target real class label, is the predicted class probability.

[0257] 3) Confidence loss optimization: Use BCE loss to optimize the confidence score of target existence judgment, the formula is:

[0258] ;

[0259] Where p is the target real confidence, is the model predicted target confidence.

[0260] Filtering rules: Based on the optimized detection results, the video analysis module selects key frames and preliminary analysis results that meet the following conditions:

[0261] The intersection over union (IoU) of the target detection box and the communication pipeline area is greater than 0.5;

[0262] Target confidence score greater than 0.8;

[0263] Target category is common equipment on construction site (such as excavator, pile driver) or personnel.

[0264] 3. Multimodal fusion analysis

[0265] The key frames and preliminary analysis results output by the video analysis module are transmitted to the cloud multimodal fusion analysis module. The multimodal large model performs deep reasoning on these data, including the following steps:

[0266] 1) Feature extraction:

[0267] Extract the spatial features of the key frame image, including the position and size of the target.

[0268] Extract time series features, including target motion trajectory and timestamp information.

[0269] Extract semantic features of text description to assist reasoning.

[0270] 2) Feature fusion:

[0271] The multimodal features are fused through a dynamic weight adjustment mechanism, and the weight adjustment formula is:

[0272] ;

[0273] Where, represents feature i, represents the relevance score of feature i.

[0274] 3) Abnormal event determination:

[0275] Combined with the feature fusion results, the large model performs deep analysis on the target behavior to determine whether there is an abnormal event (such as equipment approaching communication pipeline within 1 meter, illegal excavation, etc.), and generates event classification results, including target category, event type, occurrence time and location.

[0276] 4. Alarm generation

[0277] When the multimodal analysis module identifies an abnormal event (such as an excavator approaching a communication pipeline within 1 meter), the system sends real-time alarm information to the construction management personnel through the alarm generation module. The alarm information includes the following content:

[0278] Time: such as "December 2, 2024 10:15";

[0279] Equipment: such as "excavator";

[0280] Location: such as "communication pipeline A section";

[0281] Event: "Please check immediately, within 1 meter of the communication pipeline."

[0282] 5. Video Q&A

[0283] Construction managers can retrieve construction activity records for a specific time period through the video Q&A module. For example, the user inputs a natural language question: "What was the location record of the excavator between 16:00-18:00 yesterday?" The system will parse the question, retrieve the stored data, and return the relevant video clips and location descriptions.

[0284] 6. Audit Report Generation

[0285] After construction is completed, the system generates an inspection audit report based on the user-set prompt word templates. The report content includes:

[0286] Abnormal event records: "Time: 10:15, Equipment: Excavator, Event: Within 1 meter of the communication pipeline, Alarm sent."

[0287] Construction equipment activity summary: "14:30, Equipment: Pile driver, Event: Over-range excavation, Manual inspection completed."

[0288] Comprehensive evaluation: "Overall construction safety, no serious damage events."

[0289] Technical Effects

[0290] Through the loss function optimization and screening rules of small models, the video analysis module significantly reduces the false positive rate, providing high-quality input data for the multi-modal analysis module. The collaborative design of large and small models balances real-time performance and deep analysis capabilities, increasing the accuracy of abnormal event recognition to 98% and reducing overall response delay to within 50ms.

Claims

1. An intelligent patrol and surveillance system integrating large and small models, characterized by: include: The video analysis module is used to receive the video stream transmitted by the construction monitoring camera in real time through a small model of the target detection model with a small number of parameters, analyze the video content, and generate preliminary analysis results and key frames. It is used to perform real-time detection on the video stream transmitted by the construction monitoring camera using the target detection model with a small number of parameters, obtain the detection frame and the corresponding confidence level, and filter it through the screening submodule. The screening submodule is used to: Calculate the intersection over union (IoU) of the detection frame and the communication pipeline area; Determining the confidence level of the detection frame; Only when IoU>0.5 and confidence>0.8, mark the frame as a key frame, and output the detection result and the key frame as the preliminary analysis result; The multimodal fusion analysis module receives the preliminary analysis results and key frames from the video analysis module, combines the multimodal large model to conduct in-depth analysis of the target object and video content, and generates analysis results and alarm information for abnormal events; The multimodal fusion analysis module receives the preliminary analysis results and key frames generated by the video analysis module, performs in-depth analysis based on these data and the multimodal large model, generates analysis results of abnormal events and filtered key frames, and completes in-depth abnormal event analysis through the following steps: A. Input data format: detection box position, target category, and confidence from the small model; Filter results from the video analysis module: key frame images and corresponding timestamps; B. Input feature fusion: Multimodal feature fusion is performed based on the spatial features of the target position and size, the temporal features of the keyframe time series, and the text features of the target description; The dynamic adjustment formula of multimodal feature weights is: ； Among them, W i is the feature f i The weight, relevance(f i ) is the feature f i score of relevance to the event; C. Deep Reasoning and Classification: A multimodal large model is used to deeply analyze the fused features, determine the type, time, and location of abnormal events, and generate target categories, behavior descriptions, and detailed classification results of abnormal events. The data management module is used to store the analysis results and alarm information generated by the multimodal fusion analysis module and respond to user query requests; The alarm generation module is used to generate alarm information in real time based on the abnormal event analysis results generated by the multimodal fusion analysis module and send it to the terminal device through the network; Video question-and-answer module, which uses natural language processing technology to enable users to search and answer questions about video content; The audit report generation module is used to generate inspection audit reports of construction sites based on prompt word templates and stored analysis data.

2. The intelligent patrol and surveillance system integrating large and small models as claimed in claim 1, characterized in that: The video analysis module uses a small model to detect the target object in real time. It uses YOLOv5-nano with an improved loss function to optimize the target detection task, including bounding box regression, classification, and target confidence. The function formula used is as follows: Bounding box regression loss CIoU is used to optimize the position and size of the detection box. Its formula is: ； IoU: Intersection over Union (IoU) is the ratio of the predicted box to the true box. : The Euclidean distance between the center of the predicted box and the center of the true box; c: The minimum diagonal length of the bounding box containing the predicted box and the true box; v: weight of aspect ratio consistency; α: parameter for adjusting v; Classification loss is used to optimize the predicted probability of the target category. The formula is: ； y: true label 1 or 0; : the probability predicted by the model; Confidence loss is used to optimize the prediction of target confidence. The formula is: ； p: true confidence, 1 means there is a target, 0 means there is no target; : predicted target confidence; Rules for filtering functions: Filter based on the IoU intersection-over-union ratio between the detection box and the communication pipeline area, and targets with IoU>0.5 will be considered valid results; Based on the confidence score of the target, the results with confidence greater than 0.8 will be passed as keyframes; The output data format includes detection box location, target category, confidence and timestamp; The total loss function of the small model is the weighted sum of the above three parts: ； λ cls and λ conf is a weight parameter used to adjust the balance between classification loss and confidence loss.

3. The intelligent patrol and surveillance system integrating large and small models as claimed in claim 1, wherein: The alarm generation module sends the alarm information to the terminal device of the construction management personnel through the 5G network.

4. A method for monitoring an intelligent patrol monitoring system integrating large and small models according to any one of claims 1 to 3, comprising the following steps: A. Video reception: Obtain video streams through surveillance cameras at the construction site; B. Object detection: Use a small model to analyze the video stream, detect specific target objects and their motion states, and generate key frames and preliminary analysis results; C. Multimodal fusion analysis: Based on the preliminary analysis results, we combine the multimodal large model to conduct in-depth analysis and generate detailed analysis results of abnormal events; D. Alarm generation: Based on the results of multimodal analysis, potential anomalies are identified and alarm information is generated, which is sent to relevant personnel via the network; E. Video Q&A: Based on natural language processing technology, it supports users to search and answer questions about video content; F. Audit report generation: Generate an inspection and audit report of the construction site based on the prompt word template and stored analysis data.

5. The intelligent patrol and surveillance method integrating large and small models as claimed in claim 4, wherein: The target detection step includes detecting the type and motion state of the construction equipment through a small model and generating key frames.

6. The intelligent patrol and surveillance method of integrating large and small models as claimed in claim 4, wherein: The multimodal fusion analysis step is based on key frame images and preliminary analysis results, combined with a large multimodal model to make event judgments and generate alarm information.

7. The intelligent patrol and surveillance method of integrating large and small models as claimed in claim 4, wherein: The audit report generation step includes structuring the stored data and generating a report based on the prompt word template set by the user.

Images