Intelligent inspection staring system and staring method integrated with large and small models

Abstract

The invention relates to the technical field of intelligent inspection, in particular to an intelligent inspection staring system fusing large and small models, which is applied to real-time monitoring and alarming in a communication pipeline construction scene, and aims to solve the problems of high false alarm rate, difficulty in information retrieval and low report generation efficiency in an existing communication pipeline construction monitoring system. According to the invention, by combining a cooperative working mechanism of a lightweight small model and a multi-modal large model, real-time detection and tracking of a construction site target, deep analysis of an abnormal event, intelligent question and answer of video content and automatic generation of a construction auditing report are realized, so that the intelligent level of communication pipeline construction monitoring is improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent inspection technology, and in particular to an intelligent inspection and surveillance system integrating large and small models, which is applied to real-time monitoring and alarming in communication pipeline construction scenarios. Background Art

[0002] With the rapid development of communication networks, communication pipelines play an important role in modern society. However, during the construction process, communication pipelines are often damaged due to improper operation of construction equipment or human factors, resulting in communication interruptions and reduced service quality, which in turn causes economic losses and reduced user experience. Therefore, how to implement effective real-time monitoring of communication pipelines during construction and timely warn of potential threats has become a technical problem that needs to be solved urgently.

[0003] Traditional inspection and surveillance methods mainly rely on manual patrols, which is not only inefficient but also costly and difficult to achieve all-weather, all-round monitoring. Although video surveillance-based solutions have begun to be applied in construction scenarios, the following major problems still exist: 1. High false alarm rate: The existing video surveillance system lacks an effective false alarm filtering mechanism. A large number of false alarms increase the workload of maintenance personnel and reduce the reliability of the system.

[0004] 2. Difficulty in information retrieval: In massive video data, it is difficult for users to quickly find information related to specific events or targets, and it is impossible to efficiently query and analyze content.

[0005] 3. Report generation is cumbersome: Traditional inspections rely on manual preparation of inspection reports, which is time-consuming and labor-intensive, and is prone to missing key information, affecting the quality and efficiency of inspection work.

[0006] At present, some technical solutions attempt to improve the intelligence level of construction monitoring through target detection, radar sensors or vehicle-mounted monitoring equipment. For example, some systems identify intruders and trigger alarms through electronic fences, but are often limited to monitoring in a single mode (such as video), resulting in a high false alarm rate; other systems rely on human radar sensors for tracking, which improves local accuracy but lacks the ability to detect multiple targets in complex construction scenarios; there are also vehicle-mounted monitoring solutions that achieve local alarms through location and status data, but it is difficult to fully cover the diverse monitoring needs of the construction area.

[0007] To sum up, the existing technology cannot meet the comprehensive needs of real-time, high-precision, and multimodal information analysis in communication pipeline construction monitoring. There is an urgent need for an intelligent inspection and monitoring system that can integrate large and small models to improve detection accuracy, reduce false alarm rates, and realize efficient information retrieval and automatic report generation. Summary of the invention

[0008] The present invention aims to solve the problems of high false alarm rate, difficult information retrieval and low report generation efficiency in the existing communication pipeline construction monitoring system, and proposes an intelligent patrol and surveillance system that integrates large and small models. By combining the collaborative working mechanism of lightweight small models and multimodal large models, real-time detection and tracking of construction site targets, in-depth analysis of abnormal events, intelligent question and answer of video content, and automatic generation of construction audit reports are achieved, thereby improving the intelligent level of communication pipeline construction monitoring.

[0009] The technical solution of the intelligent patrol and surveillance system integrating large and small models of the present invention includes: The video analysis module is used to receive the video stream transmitted by the construction monitoring camera in real time through a small model of the target detection model with a small number of parameters, analyze the video content, and generate preliminary analysis results and key frames; The multimodal fusion analysis module is used to receive the preliminary analysis results and key frames of the video analysis module, conduct in-depth analysis of the target object and video content in combination with the multimodal large model, and generate analysis results and alarm information of abnormal events; The data management module is used to store the analysis results and warning information generated by the multimodal fusion analysis module and respond to user query requests; An alarm generation module is used to generate alarm information in real time based on the abnormal event analysis results generated by the multimodal fusion analysis module and send it to the terminal device through the network; Video question-and-answer module, which is used to enable users to search and answer questions about video content based on natural language processing technology; The audit report generation module is used to generate a construction site inspection audit report based on the prompt word template and stored analysis data.

[0010] Preferably, the video analysis module detects the target object in real time through a small model, and uses YOLOv5-nano to optimize the target detection task using an improved loss function, including bounding box regression, classification, and target confidence. The function formula used is as follows: Bounding box regression loss CIoU is used to optimize the position and size of the detection box. Its formula is: ; : The intersection of the predicted box and the true box; : The Euclidean distance between the center of the predicted box and the center of the true box; : The minimum diagonal length of the bounding box containing the predicted box and the true box; : The weight of aspect ratio consistency; : Adjust the parameters of vv.

[0011] Classification loss, used to optimize the predicted probability of the target category, is: ; : True label (1 or 0); : The probability predicted by the model.

[0012] Confidence loss is used to optimize the prediction of target confidence. The formula is: ; : True confidence (1 means there is a target, 0 means there is no target); : Target confidence of the prediction.

[0013] Rules for filtering functions: Based on detection frame and communication pipeline area Intersection and union are used for screening. Goals will serve as effective outcomes; Based on the confidence score of the target, the results with confidence greater than 0.8 will be passed as keyframes; The output data format includes the detection box position, target category, confidence and timestamp; The total loss function of the small model is the weighted sum of the above three parts: ; and is a weight parameter used to adjust the balance between classification loss and confidence loss.

[0014] Preferably, the multimodal fusion analysis module is used to receive the preliminary analysis results and key frames generated by the video analysis module, perform in-depth analysis based on these data in combination with the multimodal large model, generate analysis results of abnormal events and filtered key frames, and complete in-depth abnormal event analysis through the following steps: A. Input data format: detection box position, target category, and confidence from the small model; screening results from the video analysis module: key frame images and corresponding timestamps; B. Input feature fusion: Based on the spatial features of the target position and size, the temporal features of the key frame time series, and the text features of the target description, multimodal feature fusion is performed; the dynamic adjustment formula of the multimodal feature weight is: ; in, Features The weight of Features score for relevance to the event; C. Deep reasoning and classification: Through the multimodal large model, the fused features are deeply analyzed to determine the type, time and location of the abnormal event, and generate the target category, behavior description and detailed classification results of the abnormal event.

[0015] Preferably, the alarm generation module sends the alarm information to the terminal device of the construction management personnel through the 5G network.

[0016] The intelligent marking method of the present invention comprises the following steps: A. Video reception: Obtain video streams through surveillance cameras at the construction site; B. Target detection: Use a small model to analyze the video stream, detect specific target objects and their motion states, and generate key frames and preliminary analysis results; C. Multimodal fusion analysis: Based on the preliminary analysis results, the multimodal large model is combined for in-depth analysis to generate detailed analysis results of abnormal events; D. Alarm generation: Based on the results of multimodal analysis, potential anomalies are identified and alarm information is generated, which is sent to relevant personnel through the network; E. Video Q&A: Based on natural language processing technology, it supports users to search and answer questions about video content; F. Audit report generation: Generate a construction site inspection audit report based on the prompt word template and stored analysis data.

[0017] Preferably, the target detection step includes detecting the type and motion state of the construction equipment through a small model and generating a key frame.

[0018] Preferably, the multimodal fusion analysis step is based on key frame images and preliminary analysis results, combined with a multimodal large model to perform event judgment and generate alarm information.

[0019] Preferably, the audit report generation step includes structuring the stored data and generating a report according to a prompt word template set by the user.

[0020] In this application, the small model of target detection uses YOLOv5-nano and uses an improved loss function to optimize the target detection task, including bounding box regression, classification, and target confidence. The function formula used is as follows: 1. Loss function formula YOLOv5-nano uses an improved loss function to optimize target detection tasks, including bounding box regression, classification, and target confidence. The key parts are as follows: 1.1. Bounding Box Regression Loss YOLOv5-nano uses Complete Intersection over Union (CIoU) or Generalized Intersection over Union (GIoU) ​​as the bounding box loss function to optimize the overlap between the detection box and the true box.

[0021] Formula (CIoU loss): ; : The intersection-over-union ratio of the predicted box and the true box; : The Euclidean distance between the center of the predicted box and the center of the true box; : The minimum diagonal length of the bounding box containing the predicted box and the true box; : The weight of aspect ratio consistency; : Adjust the parameters of vv.

[0022] 1.2. Classification Loss Classification loss usually uses Binary Cross-Entropy (BCE) to optimize the prediction of the target category.

[0023] formula: ; : True label (1 or 0); : The probability predicted by the model.

[0024] 1.3. Confidence Loss The target confidence loss uses Binary Cross-Entropy (BCE).

[0025] formula: ; : True confidence (1 means there is a target, 0 means there is no target); : Target confidence of the prediction.

[0026] The total loss function formula is: The total loss of YOLOv5-nano is the weighted sum of the above three parts: ; and is a weight parameter used to adjust the impact of classification loss and confidence loss.

[0027] 2. Anchor Boxes prediction formula YOLOv5-nano uses a combination of Anchor-free and Anchor-based methods to predict the target box and generates a prediction box based on the center point and offset.

[0028] formula: , ; , : The center coordinate of the prediction box; : Sigmoid function, used to limit the predicted value to the range of [0, 1]; , : The offset of the network output; , : The upper left corner coordinate of the current grid cell.

[0029] Prediction box width and height: ; , : Anchor width and height; , : Scale offset of the network output.

[0030] 3. Sigmoid activation function YOLOv5-nano uses the Sigmoid activation function to restrict the classification and confidence outputs. The formula is as follows: ; : Network output value; : Map the network output value to [0, 1].

[0031] 4. Non-Maximum Suppression (NMS) Used to remove redundant detection boxes and retain the target box with the highest confidence.

[0032] formula: ; : Two candidate boxes; : The intersection-over-union ratio of two boxes.

[0033] When the IoU exceeds the set threshold, the candidate boxes with lower confidence are removed.

[0034] By adopting YOLOv5-nano, the entire technical solution can effectively meet the core requirements of real-time, accuracy and adaptability in communication pipeline construction monitoring, while providing a technical foundation for the scalability of the system (such as collaboration with large models).

[0035] After adopting this set of algorithms of YOLOv5-nano, the system can achieve the following core goals and technical effects: Real-time: Adapt to the rapidly changing environment of the construction site and ensure the timeliness of detection and alarm.

[0036] Accuracy: Significantly reduce false alarm and missed detection rates, and improve system reliability.

[0037] Adaptability: Run on low-computing edge devices to reduce dependence on cloud computing resources.

[0038] Synergy: Combined with multimodal analysis and natural language processing modules to build an efficient intelligent inspection system.

[0039] By adopting YOLOv5-nano, the entire technical solution can effectively meet the core requirements of real-time, accuracy and adaptability in communication pipeline construction monitoring, while providing a technical foundation for the scalability of the system (such as collaboration with large models).

[0040] The present invention significantly improves the intelligence level of monitoring in the construction scene of communication pipelines by integrating the intelligent inspection and monitoring system of large and small models. Compared with the prior art, it mainly achieves the following technical effects: Real-time object detection and efficient processing: The use of lightweight small models (such as YOLOv5-nano) deployed on edge devices can detect construction equipment, personnel and their motion status in real time. The optimized design of edge devices enables the video analysis module to achieve millisecond-level response (such as a processing speed of 30 frames per second), significantly improving the real-time performance of the system in a dynamic construction environment.

[0041] High accuracy and low false alarm rate in abnormal event identification: The multimodal fusion analysis module combines a large multimodal model and conducts in-depth analysis of target behaviors and events through deep learning and data fusion technology. Based on the improved CIoU / GIoU loss function and multimodal event determination algorithm, the detection accuracy of abnormal events is improved, and the false alarm rate and missed detection rate are significantly reduced.

[0042] Timeliness and reliability of warning information: After detecting potential threats or abnormal events at the construction site, the system can instantly generate and send alarm information through wireless networks (such as 5G), ensuring that construction managers can respond in a timely manner and reducing the risk of damage to communication pipelines. The multi-layer alarm filtering mechanism further improves the accuracy of alarms and reduces unnecessary interference.

[0043] Intelligent video content retrieval: With the help of the video question-answering module, users can accurately search for video content in natural language. The system supports rapid positioning and information return for specific targets, events or time periods, greatly improving the efficiency of using massive video data and the convenience of information retrieval.

[0044] Automatic audit report generation: The audit report generation module automatically integrates the stored analysis results into a structured report based on the prompt word template. Automated report generation avoids the inefficiency and error-proneness of traditional manual writing, and improves the quality and efficiency of audit work.

[0045] Flexible edge computing adaptability: The system supports a variety of edge devices (such as NVIDIA Jetson, Huawei Ascend, etc.), and adapts to the computing power limitations of different devices through lightweight technologies such as model pruning and quantization. The collaborative design of edge computing and cloud computing reduces the dependence on high-bandwidth networks and ensures the stability of the system when the network environment is unstable.

[0046] Multi-scenario applicability and scalability: The present invention is not only applicable to communication pipeline construction scenarios, but can also be extended to other application fields that require target detection, information retrieval and report generation, such as infrastructure construction monitoring, factory equipment operation management and public safety inspections.

[0047] Reduce construction management costs: The system reduces the workload of manual inspections and audits through intelligent means, reduces labor costs, and reduces the risks of construction accidents and communication pipeline damage through accurate anomaly detection, thereby reducing overall maintenance costs.

[0048] The present invention improves the quality of target detection results by optimizing the loss function of the small model, so that the generated key frames and preliminary analysis results have high confidence and accuracy. This optimization significantly reduces the computational burden of the multimodal large model and improves the accuracy and efficiency of subsequent abnormal event analysis. For example, the target data after IoU screening reduces more than 50% of redundant information, which increases the classification accuracy of abnormal events to 98%. The combination of loss function optimization and large and small model collaboration ensures the real-time and accuracy of the system and significantly reduces the false alarm rate of on-site construction monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a structural block diagram of the present invention; Figure 2 It is a schematic diagram of the steps of the present invention. DETAILED DESCRIPTION

[0050] Embodiment 1: A technical solution for an intelligent patrol and surveillance system integrating large and small models, including: The video analysis module is used to receive the video stream transmitted by the construction monitoring camera in real time through a small model of the target detection model with a small number of parameters, analyze the video content, and generate preliminary analysis results and key frames; The multimodal fusion analysis module is used to receive the preliminary analysis results and key frames of the video analysis module, conduct in-depth analysis of the target object and video content in combination with the multimodal large model, and generate analysis results and alarm information of abnormal events; The data management module is used to store the analysis results and warning information generated by the multimodal fusion analysis module and respond to user query requests; An alarm generation module is used to generate alarm information in real time based on the abnormal event analysis results generated by the multimodal fusion analysis module and send it to the terminal device through the network; Video question-and-answer module, which is used to enable users to search and answer questions about video content based on natural language processing technology; The audit report generation module is used to generate a construction site inspection audit report based on the prompt word template and stored analysis data.

[0051] Embodiment 2: The difference between this embodiment and embodiment 1 is that this embodiment further includes: The video analysis module detects the target object in real time through a small model, which uses YOLOv5-nano to optimize the target detection task, including bounding box regression, classification and target confidence, using an improved loss function. The object detection of the video analysis module is precisely optimized through an optimized loss function, including: 1) Optimize the overlap between the detection box and the real target through bounding box regression loss to ensure that the position and size of the detection box accurately fits the target; 2) Improve the accuracy of target category prediction through classification loss and ensure the category consistency of detection results; 3) Optimize the confidence score of the target through confidence loss and filter out targets with confidence greater than 0.8.

[0052] Based on the above optimization results, the video analysis module selects key frames related to communication pipeline safety and preliminary analysis results, including the precise location of the target (IoU>0.5), category (such as excavator, construction personnel) and confidence score (such as confidence>0.8). These filtered high-quality data are passed to the multimodal fusion analysis module, which further combines video key frames, time series data and text information for multimodal reasoning of abnormal events.

[0053] The function formula used is as follows: Bounding box regression loss CIoU is used to optimize the position and size of the detection box. Its formula is: ; : The intersection-over-union ratio of the predicted box and the true box; : The Euclidean distance between the center of the predicted box and the center of the true box; : The minimum diagonal length of the bounding box containing the predicted box and the true box; : The weight of aspect ratio consistency; : Adjust the parameters of vv.

[0054] Classification loss, used to optimize the predicted probability of the target category, is: ; : True label (1 or 0); : The probability predicted by the model.

[0055] Confidence loss is used to optimize the prediction of target confidence. The formula is: ; : True confidence (1 means there is a target, 0 means there is no target); : Target confidence of the prediction.

[0056] Rules for filtering functions: Based on detection frame and communication pipeline area Intersection and union are used for screening. Goals will serve as effective outcomes; Based on the confidence score of the target, the results with confidence greater than 0.8 will be passed as keyframes; The output data format includes the detection box position, target category, confidence and timestamp; The total loss function of the small model is the weighted sum of the above three parts: ; and is a weight parameter used to adjust the balance between classification loss and confidence loss.

[0057] Preferably, the multimodal fusion analysis module is used to receive the preliminary analysis results and key frames generated by the video analysis module, perform in-depth analysis based on these data in combination with the multimodal large model, generate analysis results of abnormal events and filtered key frames, and complete in-depth abnormal event analysis through the following steps: A. Input data format: detection box position, target category, and confidence from the small model; screening results from the video analysis module: key frame images and corresponding timestamps; B. Input feature fusion: Based on the spatial features of the target position and size, the temporal features of the key frame time series, and the text features of the target description, multimodal feature fusion is performed; the dynamic adjustment formula of the multimodal feature weight is: ; in, Features The weight of Features score for relevance to the event; C. Deep reasoning and classification: Through the multimodal large model, the fused features are deeply analyzed to determine the type, time and location of the abnormal event, and generate the target category, behavior description and detailed classification results of the abnormal event.

[0058] Preferably, the alarm generation module sends the alarm information to the terminal device of the construction management personnel through the 5G network.

[0059] Embodiment 3: The intelligent marking method of this embodiment includes the following steps: A. Video reception: Obtain video streams through surveillance cameras at the construction site; B. Target detection: Use a small model to analyze the video stream, detect specific target objects and their motion states, and generate key frames and preliminary analysis results; C. Multimodal fusion analysis: Based on the preliminary analysis results, the multimodal large model is combined for in-depth analysis to generate detailed analysis results of abnormal events; D. Alarm generation: Based on the results of multimodal analysis, potential anomalies are identified and alarm information is generated, which is sent to relevant personnel through the network; E. Video Q&A: Based on natural language processing technology, it supports users to search and answer questions about video content; F. Audit report generation: Generate a construction site inspection audit report based on the prompt word template and stored analysis data.

[0060] Embodiment 4: The difference between this embodiment and embodiment 3 is that this embodiment further includes: The target detection step includes detecting the type and motion state of the construction equipment through a small model and generating key frames.

[0061] Preferably, the multimodal fusion analysis step is based on key frame images and preliminary analysis results, combined with a multimodal large model to perform event judgment and generate alarm information.

[0062] Preferably, the audit report generation step includes structuring the stored data and generating a report according to a prompt word template set by the user.

[0063] In this application, the small model of target detection uses YOLOv5-nano and uses an improved loss function to optimize the target detection task, including bounding box regression, classification, and target confidence. The function formula used is as follows: 1. Loss function formula YOLOv5-nano uses an improved loss function to optimize target detection tasks, including bounding box regression, classification, and target confidence. The key parts are as follows: 1.1. Bounding Box Regression Loss YOLOv5-nano uses Complete Intersection over Union (CIoU) or Generalized Intersection over Union (GIoU) ​​as the bounding box loss function to optimize the overlap between the detection box and the true box.

[0064] Formula (CIoU loss): ; : The intersection-over-union ratio of the predicted box and the true box; : The Euclidean distance between the center of the predicted box and the center of the true box; : The minimum diagonal length of the bounding box containing the predicted box and the true box; : The weight of aspect ratio consistency; : Adjust the parameters of vv.

[0065] 1.2. Classification Loss Classification loss usually uses Binary Cross-Entropy (BCE) to optimize the prediction of the target category.

[0066] formula: ; : True label (1 or 0); : The probability predicted by the model.

[0067] Confidence loss is used to optimize the prediction of target confidence. The formula is: ; : True confidence (1 means there is a target, 0 means there is no target); : Target confidence of the prediction.

[0068] The total loss function formula is: The total loss of YOLOv5-nano is the weighted sum of the above three parts: ; and is a weight parameter used to adjust the impact of classification loss and confidence loss.

[0069] 2. Anchor Boxes prediction formula YOLOv5-nano uses a combination of Anchor-free and Anchor-based methods to predict the target box and generates a prediction box based on the center point and offset.

[0070] formula: , ; , : The center coordinate of the prediction box; : Sigmoid function, used to limit the predicted value to the range of [0, 1]; , : The offset of the network output;

[0071] , : The upper left corner coordinate of the current grid cell.

[0072] Prediction box width and height: ; , : Anchor width and height; , : Scale offset of the network output.

[0073] 3. Sigmoid activation function YOLOv5-nano uses the Sigmoid activation function to restrict the classification and confidence outputs. The formula is as follows: ; : Network output value.

[0074] : Map the network output value to [0, 1].

[0075] 4. Non-Maximum Suppression (NMS) Used to remove redundant detection boxes and retain the target box with the highest confidence.

[0076] formula: ; : Two candidate boxes.

[0077] : The intersection-over-union ratio of two boxes.

[0078] When the IoU exceeds the set threshold, the candidate boxes with lower confidence are removed.

[0079] Embodiment 5: At a communication pipeline construction site, in order to ensure the safety of the communication pipeline, a set of intelligent inspection and monitoring system based on the combination of large and small models was deployed. The specific implementation method is as follows: 1. Video reception and preprocessing The surveillance cameras installed at the construction site transmit real-time video streams to the video analysis module of the edge device through wireless networks (such as WiFi or 5G). After preprocessing, the video stream is used as input data for target detection by the small model (YOLOv5-nano).

[0080] 2. Object detection and loss function optimization The video analysis module uses a small-parameter target detection model and accurately optimizes the target detection results through an optimized loss function, including: 1) Bounding box regression loss optimization: The complete intersection over union (CIoU) loss is used to optimize the overlap between the predicted box and the true box. The formula is: ; Among them, ρ represents the Euclidean distance between the center point of the predicted box and the real box, c is the diagonal length of the minimum bounding box, α and v are the adjustment parameter and aspect ratio consistency weight respectively.

[0081] 2) Classification loss optimization: The prediction accuracy of the target category is optimized through binary cross entropy loss (BCE), the formula is: ; Among them, y represents the target true category label, is the predicted class probability.

[0082] 3) Confidence loss optimization: Use BCE loss to optimize the confidence score of target existence judgment. The formula is: ; Among them, p is the true confidence of the target, The target confidence predicted by the model.

[0083] Screening rules: Based on the optimized detection results, the video analysis module screens out key frames and preliminary analysis results that meet the following conditions: The intersection over union (IoU) between the target detection box and the communication pipeline area is greater than 0.5; The target confidence score is greater than 0.8; The target categories are equipment (such as excavators, pile drivers) or personnel commonly found on construction sites.

[0084] 3. Multimodal Fusion Analysis The key frames and preliminary analysis results output by the video analysis module are transmitted to the multimodal fusion analysis module in the cloud. The multimodal large model performs deep reasoning on these data, which includes the following steps: 1) Feature extraction: Extract the spatial features of the keyframe image, including the location and size of the target.

[0085] Extract time series features, including target motion trajectory and timestamp information.

[0086] Extract semantic features of text descriptions to assist reasoning.

[0087] 2) Feature fusion: Multimodal features are fused through a dynamic weight adjustment mechanism, and the weight adjustment formula is: ; in, represents feature i, represents the relevance score of feature i.

[0088] 3) Abnormal event determination: Combined with the feature fusion results, the large model conducts in-depth analysis of the target behavior to determine whether there are abnormal events (such as equipment close to communication pipelines, illegal excavation, etc.), and generates event classification results, including target category, event type, occurrence time and location.

[0089] 4. Alarm generation When the multimodal analysis module identifies an abnormal event (such as an excavator approaching within 1 meter of a communication pipeline), the system sends a real-time alarm message to the construction management personnel through the alarm generation module. The alarm message includes the following: Time: such as "December 2, 2024 10:15"; Equipment: such as "excavator"; Location: such as "communication pipeline section A"; Event: such as "Within 1 meter of the communication pipeline, please check immediately."

[0090] 5. Video Q&A Construction managers can use the video question-and-answer module to retrieve construction activity records for a specific time period. For example, a user enters a natural language question: "Where was the excavator's location recorded between 16:00 and 18:00 yesterday?" The system will parse the question, retrieve the stored data, and return the relevant video clips and location descriptions.

[0091] 6. Audit report generation After the construction is completed, the system generates an inspection and audit report based on the prompt word template set by the user. The report includes: Abnormal event record: such as "Time: 10:15, Equipment: Excavator, Event: Approaching within 1 meter of the communication pipeline, alarm has been sent."

[0092] Construction equipment activity summary: such as "14:30, equipment: pile driver, event: excavation beyond the scope, manual inspection completed."

[0093] Comprehensive evaluation: such as "The overall construction is safe, with no serious damage."

[0094] Technical Effects Through the optimization of the loss function and screening rules of the small model, the video analysis module significantly reduced the false alarm rate and provided high-quality input data for the multimodal analysis module. The collaborative design of the combination of large and small models takes into account both real-time and deep analysis capabilities, increasing the accuracy of abnormal event recognition to 98% and reducing the overall response delay to less than 50ms.

Claims

1. An intelligent patrol and surveillance system integrating large and small models, characterized in that: include: The video analysis module is used to receive the video stream transmitted by the construction monitoring camera in real time through a small model of the target detection model with a small number of parameters, analyze the video content, and generate preliminary analysis results and key frames; The multimodal fusion analysis module is used to receive the preliminary analysis results and key frames of the video analysis module, conduct in-depth analysis of the target object and video content in combination with the multimodal large model, and generate analysis results and alarm information of abnormal events; The data management module is used to store the analysis results and warning information generated by the multimodal fusion analysis module and respond to user query requests; An alarm generation module is used to generate alarm information in real time based on the abnormal event analysis results generated by the multimodal fusion analysis module and send it to the terminal device through the network; Video question-and-answer module, which is used to enable users to search and answer questions about video content based on natural language processing technology; The audit report generation module is used to generate a construction site inspection audit report based on the prompt word template and stored analysis data.

2. The intelligent patrol and surveillance system integrating large and small models as claimed in claim 1, characterized in that: The video analysis module detects the target object in real time through a small model. It uses YOLOv5-nano to optimize the target detection task using an improved loss function, including bounding box regression, classification, and target confidence. The function formula used is as follows: Bounding box regression loss CIoU is used to optimize the position and size of the detection box. Its formula is: ； : The intersection of the predicted box and the true box; : The Euclidean distance between the center of the predicted box and the center of the true box; : The minimum diagonal length of the bounding box containing the predicted box and the true box; : The weight of aspect ratio consistency; : Adjust the parameters of vv; Classification loss, used to optimize the predicted probability of the target category, is: ； : The true label is 1 or 0; : The probability predicted by the model; Confidence loss is used to optimize the prediction of target confidence. The formula is: ； : True confidence 1 means there is a target, 0 means there is no target; : Predicted target confidence; Rules for filtering functions: Based on detection frame and communication pipeline area Intersection and union are used for screening. Goals will serve as effective outcomes; Based on the confidence score of the target, the results with confidence greater than 0.8 will be passed as keyframes; The output data format includes the detection box position, target category, confidence and timestamp; The total loss function of the small model is the weighted sum of the above three parts: ； and is a weight parameter used to adjust the balance between classification loss and confidence loss.

3. The intelligent patrol and surveillance system integrating large and small models as claimed in claim 1, characterized in that: in, The multimodal fusion analysis module is used to receive the preliminary analysis results and key frames generated by the video analysis module, and conduct in-depth analysis based on these data combined with the multimodal large model to generate analysis results of abnormal events and filtered key frames. The in-depth abnormal event analysis is completed through the following steps: A. Input data format: detection box position, target category, and confidence from the small model; Filter results from the video analysis module: key frame images and corresponding timestamps; B. Input feature fusion: Multimodal feature fusion is performed based on the spatial features of the target position and size, the temporal features of the key frame time series, and the text features of the target description; The dynamic adjustment formula of multimodal feature weights is: ； in, Features The weight of Features score for relevance to the event; C. Deep reasoning and classification: Through the multimodal large model, the fused features are deeply analyzed to determine the type, time and location of the abnormal event, and generate the target category, behavior description and detailed classification results of the abnormal event.

4. The intelligent patrol and surveillance system integrating large and small models as claimed in claim 1, wherein: The alarm generation module sends the alarm information to the terminal device of the construction management personnel through the 5G network.

5. A method for monitoring an intelligent patrol monitoring system integrating large and small models as claimed in any one of claims 1 to 4, comprising the following steps: A. Video reception: Obtain video streams through surveillance cameras at the construction site; B. Target detection: Use a small model to analyze the video stream, detect specific target objects and their motion states, and generate key frames and preliminary analysis results; C. Multimodal fusion analysis: Based on the preliminary analysis results, the multimodal large model is combined for in-depth analysis to generate detailed analysis results of abnormal events; D. Alarm generation: Based on the results of multimodal analysis, potential anomalies are identified and alarm information is generated, which is sent to relevant personnel through the network; E. Video Q&A: Based on natural language processing technology, it supports users to search and answer questions about video content; F. Audit report generation: Generate a construction site inspection audit report based on the prompt word template and stored analysis data.

6. An intelligent patrol and surveillance method integrating large and small models as claimed in claim 5, wherein: The target detection step includes detecting the type and motion state of the construction equipment through a small model and generating key frames.

7. The intelligent patrol and surveillance method of integrating large and small models as claimed in claim 5, wherein: The multimodal fusion analysis step is based on key frame images and preliminary analysis results, combined with a multimodal large model to make event judgments and generate alarm information.

8. The intelligent patrol and surveillance method of integrating large and small models as claimed in claim 5, wherein: The audit report generation step includes structuring the stored data and generating a report according to the prompt word template set by the user.

Images