An AI video recognition-based abnormal outflow of a discharge port intelligent inspection method and system

Abstract

The application relates to the technical field of water environment intelligent supervision, and discloses an abnormal outflow intelligent inspection method and system for a discharge port based on AI video recognition, which comprises the following steps: first, acquiring real-time video streams and Internet of Things sensor data of a target discharge port area; performing dynamic frame extraction on the video streams to obtain key frames, performing space-time alignment on the key frames and the sensor data, and performing anti-interference enhancement on the aligned key frames according to the sensor data; inputting the enhanced key frames and the sensor data into a pre-constructed multimodal large model, extracting visual features through a visual encoder, performing semantic feature alignment on the visual features and text embedding of a large language model decoder after mapping through a projection layer, loading a preset abnormal outflow identification prompt word template to perform reasoning, and outputting whether there is outflow, a flow level and a confidence level; and generating graded early warning information according to the recognition result. The application realizes all-weather accurate monitoring and automatic early warning of outflow of a discharge port, and reduces the artificial inspection cost and the false alarm rate.

Description

Technical Field

[0001] This application relates to the field of intelligent water environment monitoring technology, and in particular to an intelligent inspection method for abnormal outflow of discharge outlets based on AI video recognition, and an intelligent inspection system for abnormal outflow of discharge outlets based on AI video recognition. Background Technology

[0002] Currently, while video surveillance facilities have been deployed on a large scale in the field of water environment supervision, their level of intelligence is generally insufficient. Most discharge outlets still rely on manual visual inspection of video footage to detect illegal discharges or abnormal outflows, which is inefficient and makes it difficult to achieve real-time, 24 / 7 early warning. Meanwhile, although IoT sensors such as rain gauges, water quality meters, and level gauges are becoming increasingly common, most only provide single-point threshold alarms, failing to deeply mine the value of the data. Furthermore, there is a lack of cross-type linkage analysis between video and sensor data, resulting in low identification accuracy and delayed response, making it difficult to meet the needs of refined and intelligent supervision. In addition, illegal discharges often occur at night or under severe weather conditions, and traditional technical solutions have significant shortcomings in terms of generalization ability, evidence chain retention, and closed-loop business processing.

[0003] In existing technologies, one approach involves deploying various specialized devices such as rain sensors, flow meters, and pH sensors to determine illegal discharge based on rules such as increased flow despite no detected rainfall, triggering emergency pool interception and audible / visual alarms. Another approach is a machine vision-based video-based sewage detection method. This method employs a TSM video processing model, dynamically adjusting supplementary lighting parameters and analyzing time-series frame sequences to determine the sewage discharge status.

[0004] However, all of the aforementioned existing technologies have significant limitations. The first type of solution fails to achieve automatic recognition of video footage; the cameras are only used for manual viewing and cannot utilize the visual characteristics of water flow for automatic judgment. Furthermore, this solution requires the deployment of numerous dedicated sensors, lacks the utilization of existing surveillance cameras, and fails to generate a structured video evidence chain. Even after an alarm, manual on-site verification is still required, lacking an automated business processing loop. The second type of solution only uses video image data and does not integrate IoT sensor data such as rainfall and water quality, thus failing to distinguish between dry-day outflow and compliant drainage during rainy days. Based on a traditional small-model architecture, it has weak generalization ability, high dataset requirements, and long iteration cycles. In addition, this solution only outputs classification results of sewage discharge status and fails to achieve tiered early warning, evidence chain retention, and an automated flow and feedback loop from alarm events to maintenance work orders. Summary of the Invention

[0005] (a) Technical problems to be solved

[0006] In view of the above-mentioned shortcomings and deficiencies of the existing technology, this application provides an intelligent inspection method and system for abnormal outflow of discharge outlets based on AI video recognition, so as to solve the following technical problems:

[0007] How to automatically identify the outflow status of discharge outlets using only ordinary surveillance cameras, including whether there is water flow, the outflow pattern and flow rate, in order to reduce reliance on dedicated physical sensors;

[0008] How to achieve spatiotemporal alignment and semantic-level fusion of video frame extraction and IoT sensor data such as rainfall and liquid level, so as to use rainfall data to help distinguish between dry outflow and compliant drainage in rainy weather, and improve the recognition accuracy through cross-validation;

[0009] How to automatically extract video clips and capture key frames before and after the anomaly occurs after determining illegal discharge, and encapsulate the identification results, sensor data, equipment information, etc. into a traceable digital evidence chain;

[0010] How to automatically push AI-generated alarms to the operation and maintenance work order system to achieve a closed-loop business process of automatic work order dispatch, mobile terminal reception, on-site handling feedback and result transmission;

[0011] How to quickly adapt to different outlet scenarios and eliminate new interference items without retraining the model, so as to reduce model iteration costs and computing power consumption;

[0012] How to output graded alarms based on outflow volume or pollution level, and retain a complete chain of event evidence including keyframes, video clips, and judgment criteria;

[0013] How to build a complete business closed loop from AI-based alarm recognition to automatic dispatch of maintenance work orders, on-site handling feedback, and model iteration and optimization.

[0014] (II) Technical Solution

[0015] To achieve the above objectives, the main technical solutions adopted in this application include:

[0016] In a first aspect, embodiments of this application provide an intelligent inspection method for abnormal outflow from discharge outlets based on AI video recognition, the specific steps of which include:

[0017] S1. Obtain real-time video stream and IoT sensor data of the target discharge outlet area, wherein the IoT sensor data includes at least rainfall data and liquid level data;

[0018] S2. Dynamically extract frames from the real-time video stream to obtain video keyframes; perform spatiotemporal alignment processing on the video keyframes and the IoT sensor data, and enhance the anti-interference of the spatiotemporally aligned video keyframes based on the IoT sensor data to obtain anti-interference enhanced video keyframes.

[0019] S3. Input the anti-interference enhanced video keyframes and the IoT sensor data into a pre-constructed multimodal large model. Extract the visual features of the anti-interference enhanced video keyframes through the visual encoder in the multimodal large model. After mapping by the projection layer, align the semantic features with the text embedding space of the large language model. Load the preset outlet abnormal outflow identification prompt word template for inference and output the identification result. The identification result includes outlet location, whether there is outflow, outflow level, judgment reason, and corresponding confidence level.

[0020] S4. Based on the identification results, generate warning information of the corresponding level, and simultaneously capture video clips and key frames before and after the occurrence of the abnormal event, associate sensor data and identification logs, and generate an immutable structured chain of evidence.

[0021] S5. Automatically convert early warning information and structured evidence chains into standardized operation and maintenance work orders, match response time limits and handling personnel according to the early warning level, and dispatch them to mobile operation and maintenance terminals through IoT protocol; receive and store on-site handling feedback data to complete the work order closed loop;

[0022] S6. Based on the on-site handling feedback data, the abnormal outflow identification prompt word template of the discharge outlet is adaptively optimized.

[0023] Optionally, in some embodiments of this application, step S2 involves dynamically extracting frames from the real-time video stream to obtain video keyframes, including:

[0024] Lightweight semantic segmentation is performed on the real-time video stream to identify the outlet area and the water surface area;

[0025] A first optical flow feature is calculated within the outlet area. The first optical flow feature includes a first optical flow velocity amplitude and a first directional consistency. The first directional consistency is used to determine whether the pixel optical flow direction within the outlet area is radial outward from the outlet.

[0026] A second optical flow feature is calculated within the water surface area. The second optical flow feature includes a second optical flow velocity amplitude and a second direction consistency. The second direction consistency is used to determine whether the pixel optical flow direction within the water surface area is a fan-shaped diffusion pattern away from the outlet.

[0027] When the amplitude of the first optical flow velocity is lower than the first threshold and the amplitude of the second optical flow velocity is lower than the second threshold, it is determined to be a static scene, and the frame extraction frequency is set to 30 minutes / frame;

[0028] When the amplitude of the first optical flow velocity is higher than the first threshold and the consistency of the first direction is higher than the first direction threshold, it is determined to be a dynamic scene, and the frame sampling frequency is set to 1 minute / frame.

[0029] Alternatively, when the amplitude of the second optical flow velocity is higher than the second threshold and the consistency of the second direction is higher than the second direction threshold, it is determined to be a dynamic scene, and the frame sampling frequency is set to 1 minute / frame.

[0030] Optionally, in some embodiments of this application, step S2, which enhances the anti-interference performance of the spatiotemporally aligned video keyframes based on the IoT sensor data, includes:

[0031] Determine whether the rainfall data in the IoT sensor data exceeds a preset rainfall threshold, and determine whether the rate of change of the liquid level data exceeds a preset liquid level threshold;

[0032] If the rainfall data exceeds a preset rainfall threshold, a preset defogging algorithm is invoked to perform image enhancement processing on the spatiotemporally aligned video keyframes in order to suppress the visual interference of raindrops on outflow recognition.

[0033] If the rainfall data does not exceed the preset rainfall threshold and the rate of change of the liquid level data exceeds the preset liquid level threshold, then the preset image sharpening and edge enhancement algorithm is invoked to process the spatiotemporally aligned video keyframes to highlight the dynamic texture features of the outlet water surface.

[0034] Otherwise, maintain the original state of the video keyframes after the spatiotemporal alignment process, or perform only regular color normalization.

[0035] Optionally, in some embodiments of this application, the abnormal outflow identification prompt word template includes seven elements: role setting and input setting, target and task boundary, identification object description, identification process description, level description, confidence and adoption strategy, and output format constraints; the abnormal outflow identification prompt word template includes an interference exclusion rule unit, which is configured with interference filtering instructions based on image light and shadow features and time series features;

[0036] The interference filtering instruction is used to identify reflective areas, raindrop spots, and fixed water stains in the image, and mark them as interference items to exclude from the outflow judgment.

[0037] The abnormal outflow identification prompt template for the outlet also includes an outflow confirmation rule unit, which is used to determine that there is outflow only when there is a moving area in the outlet area that meets the preset continuity condition and the optical flow direction is consistent and outward.

[0038] Optionally, in some embodiments of this application, the outlet abnormal outflow identification prompt word template further includes a multi-source data cross-modal verification unit;

[0039] The verification unit is configured with logical operation rules, which are used to semantically compare the outflow results identified by vision with the synchronously acquired rainfall data and liquid level data.

[0040] When the visual recognition result indicates that there is outflow and the rainfall data meets the preset no-rain conditions, a dry weather outflow judgment result is generated; when the visual recognition result indicates that there is outflow and the liquid level data change exceeds the preset threshold, an outflow risk judgment result is generated.

[0041] Optionally, in some embodiments of this application, the interference filtering instructions include:

[0042] The reflection exclusion rule is used to identify highlight areas in an image whose brightness value is higher than a preset multiple of the surrounding area, whose edges are sharp, and whose position or shape changes over time in a multi-frame image sequence, and to mark them as reflection interference.

[0043] Raindrop exclusion rules are used to identify circular or elliptical spots in an image that are within a preset pixel range, appear randomly, and persist for less than a preset frame threshold, and mark them as raindrop interference.

[0044] The pipe wall wetness exclusion rule is used to identify dark wet areas formed by water accumulation, seepage, condensation or residual moisture on the pipe wall below the outlet, the wall or the pipe wall to which the outlet belongs. The dark wet areas have no flow texture, are fixed in position in multiple frame image sequences, do not form water tongues or continuous water bodies, and are characterized as sheet-like or block-like wetness. These are marked as pipe wall wetness interference.

[0045] The shadow exclusion rule is used to identify low-brightness areas formed by light obstruction in and around the outlet. The low-brightness areas have soft or clear edges, no internal texture changes, and do not have the sense of downward extension of water flow or dynamic light flow characteristics. These areas are marked as shadow interference.

[0046] Optionally, in some embodiments of this application, the structured chain of evidence in S4 includes:

[0047] The complete video clip from 60 seconds before to 60 seconds after the abnormal event, at least 3 keyframe images extracted from the complete video clip, IoT sensor data aligned with the complete video clip in time range; multimodal large model recognition process log, device information and spatiotemporal coordinates;

[0048] The evidence chain generates a unique encrypted identifier, which has the characteristics of being tamper-proof and traceable.

[0049] Optionally, in some embodiments of this application, step S5 includes:

[0050] Analyze early warning information and structured evidence chains, and automatically populate work order fields, including outlet location, early warning level, response time limit, handling requirements, and evidence chain links;

[0051] The response time limit and candidate personnel are determined based on the warning level. Then, the target personnel are selected from the candidate personnel based on the management area, current workload and skill level of the operation and maintenance personnel. The standardized operation and maintenance work order is then dispatched to the mobile operation and maintenance terminal of the target personnel.

[0052] Send navigation routes and evidence collection instructions to the mobile maintenance terminal; receive on-site handling data uploaded by the mobile maintenance terminal, including handling photos, handling videos, sampling records, and handling results; automatically update the work order status based on the on-site handling data;

[0053] The system automatically verifies the integrity of the on-site handling data and automatically archives the work order when the verification passes. If the on-site handling data contains false alarm identifiers, the corresponding keyframes are automatically marked as negative samples for use in optimizing the abnormal outflow identification prompt word template of the discharge outlet.

[0054] Optionally, in some embodiments of this application, step S6 includes:

[0055] According to the preset iteration cycle, samples with confidence levels within the preset fuzzy range in the historical recognition results, as well as samples marked as false alarms or missed alarms, are retrieved and packaged into a high-value difficult example sample set; the high-value difficult example sample set is used to automatically optimize the abnormal outflow recognition prompt word template of the discharge outlet; after optimization, regression verification is performed through the false alarm test set, and the system is automatically put into operation after the verification is passed.

[0056] Secondly, this application proposes an intelligent inspection system for abnormal outflow from discharge outlets based on AI video recognition, comprising:

[0057] The perception layer is used to collect real-time video streams and IoT sensor data of the target discharge area, and supports compatibility with existing monitoring equipment protocols and reuse and adaptation.

[0058] The analysis layer includes edge nodes and a cloud platform; the edge nodes are used to perform dynamic frame extraction, spatiotemporal alignment, and anti-interference enhancement processing; the cloud platform deploys a multimodal large model and outlet abnormal outflow recognition prompt word templates to realize cross-modal reasoning and output recognition results.

[0059] The early warning and evidence chain layer is used to generate hierarchical early warning information based on the identification results, and automatically generate and store an immutable structured evidence chain;

[0060] The maintenance work order layer is used to automatically convert early warning information into standardized maintenance work orders, realizing automatic work order dispatch, interaction with mobile terminals, and closed-loop handling and feedback.

[0061] An iterative optimization layer is used to achieve closed-loop optimization of the abnormal outflow identification prompt word template of the discharge outlet based on the handling feedback data.

[0062] (III) Beneficial Effects

[0063] The present application discloses an intelligent inspection method and system for abnormal outflow from discharge outlets based on AI video recognition, which has the following beneficial effects:

[0064] Improve recognition accuracy and environmental adaptability: By spatiotemporal alignment and deep fusion of video visual features and multi-source sensor data, and combined with anti-interference enhancement processing driven by sensor data, environmental interference such as rain and fog is effectively suppressed, significantly improving the accuracy of abnormal outflow recognition at discharge outlets (up to 88%). The proportion of valid events verified on-site is over 95%, which is better than existing single recognition solutions.

[0065] Achieve precise outflow classification: Through inference using a pre-built multimodal large model and abnormal outflow identification prompt word templates, it can output fine-grained classification results such as no outflow, small amount of outflow, and large amount of outflow, meeting the needs of refined supervision;

[0066] Constructing a complete intelligent inspection business closed loop: from AI video recognition and sensor fusion early warning, to automatically generating maintenance work orders, dispatching them to mobile terminals, receiving handling feedback, and then to model-free self-evolution of prompt word templates based on feedback data, forming a closed-loop management system of "identification → early warning → handling → optimization", which solves the problems of manual connection after alarm, delayed response, and lack of feedback iteration in existing technologies, and significantly improves regulatory efficiency;

[0067] Reduce iteration costs and improve generalization ability: Lightweight adaptive optimization of prompt word templates based on operation and maintenance feedback data, without the need to retrain large models, reduces computing and human costs, and can quickly adapt to different outlet scenarios, with generalization ability superior to traditional solutions. Attached Figure Description

[0068] Figure 1 This is a flowchart of an intelligent inspection method for abnormal outflow from a discharge outlet based on AI video recognition, as described in this application.

[0069] Figure 2 This is an overall business process diagram of an agent-based outflow monitoring system for abnormal discharge from outlets, as described in one embodiment of this application.

[0070] Figure 3 This is a schematic diagram of the architecture of an intelligent inspection system for abnormal outflow at the discharge port in one embodiment of this application. Detailed Implementation

[0071] To better explain and facilitate understanding of this application, the following detailed description of the application is provided in conjunction with the accompanying drawings and specific embodiments.

[0072] In existing technologies, intelligent monitoring methods for abnormal outflow from discharge outlets can be mainly categorized into the following three types:

[0073] The first type is the pure IoT sensor threshold alarm method, which deploys sensors such as rain gauges, level gauges, and water quality meters at the outlet. When a certain parameter exceeds a preset threshold, an alarm is triggered. This type of method relies only on single-point thresholds, cannot utilize video visual features for cross-validation, and is susceptible to sensor drift and transient interference, resulting in false alarms. Furthermore, it cannot distinguish between outflows during dry weather and compliant drainage during rainy weather, and lacks the ability to identify outflow patterns (such as trickle or full-pipe flow).

[0074] The second category involves purely video-based manual or traditional visual methods. These methods use surveillance cameras to capture images of the outlets and rely on manual visual inspection or traditional image processing algorithms (such as frame difference and optical flow methods) to determine whether there is water flow. This type of method has a low level of intelligence, low efficiency of manual inspection, and cannot provide real-time warnings around the clock. Traditional visual algorithms are easily affected by changes in lighting, raindrops, reflections, water surface ripples, etc., have weak generalization ability, and are difficult to accurately identify small outflows or distinguish water flow from interfering objects.

[0075] The third category is single deep learning model methods, which use small models such as TSM and CNN to classify video frames from discharge outlets to determine whether pollution discharge occurs. These methods only use video data and do not integrate sensor information such as rainfall and liquid level, making it impossible to use multi-source data for joint discrimination. The models have poor generalization ability, requiring re-collection and retraining for new scenarios or interference items, resulting in long iteration cycles and high computational costs. Furthermore, they fail to achieve a closed-loop business process of tiered early warning, evidence chain retention, and alarm-to-work order conversion, making it difficult to directly translate the identification results into regulatory effectiveness.

[0076] To address this, this application provides an intelligent inspection method for abnormal outflow from discharge outlets based on AI video recognition. This method uses dynamic frame extraction and spatiotemporal alignment to synchronously fuse keyframes from videos with IoT sensor data. It then utilizes a multimodal large model combined with pre-set abnormal outflow identification prompt word templates for semantic reasoning, outputting the outflow flow level and confidence level, ultimately generating tiered early warning information. Simultaneously, through prompt word engineering, it achieves zero-sample scenario adaptation, quickly eliminating new interference items without retraining the model, and constructing a complete business loop from early warning to work order dispatch and handling feedback, thereby overcoming the shortcomings of the aforementioned prior art.

[0077] To better explain and facilitate understanding of this application, a detailed description of its embodiments is provided below in conjunction with the accompanying drawings. While exemplary embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a clearer and more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.

[0078] Example 1:

[0079] Figure 1 The flowchart below illustrates an intelligent inspection method for abnormal outflow from a discharge outlet based on AI video recognition, according to this application. Figure 1 As shown, this AI video recognition-based intelligent inspection method for abnormal outflow from discharge outlets includes:

[0080] S1. Obtain real-time video stream and IoT sensor data of the target discharge outlet area, wherein the IoT sensor data includes at least rainfall data and liquid level data;

[0081] S2. Dynamically extract frames from the real-time video stream to obtain video keyframes; perform spatiotemporal alignment processing on the video keyframes and the IoT sensor data, and enhance the anti-interference of the spatiotemporally aligned video keyframes based on the IoT sensor data to obtain anti-interference enhanced video keyframes.

[0082] S3. Input the anti-interference enhanced video keyframes and the IoT sensor data into a pre-constructed multimodal large model. Extract the visual features of the anti-interference enhanced video keyframes through the visual encoder in the multimodal large model. After mapping by the projection layer, align the semantic features with the text embedding space of the large language model. Load the preset outlet abnormal outflow identification prompt word template for inference and output the identification result. The identification result includes outlet location, whether there is outflow, outflow level, judgment reason, and corresponding confidence level.

[0083] S4. Based on the identification results, generate warning information of the corresponding level, and simultaneously capture video clips and key frames before and after the occurrence of the abnormal event, associate sensor data and identification logs, and generate an immutable structured chain of evidence.

[0084] S5. Automatically convert early warning information and structured evidence chains into standardized operation and maintenance work orders, match response time limits and handling personnel according to the early warning level, and dispatch them to mobile operation and maintenance terminals through IoT protocol; receive and store on-site handling feedback data to complete the work order closed loop;

[0085] S6. Based on the on-site handling feedback data, the abnormal outflow identification prompt word template of the discharge outlet is adaptively optimized.

[0086] This embodiment achieves deep fusion of visual information and physical environment parameters by acquiring real-time video streams of the target discharge area and IoT sensor data containing rainfall and liquid level data. On this basis, by spatiotemporally aligning video keyframes and sensor data, and using sensor data to guide anti-interference enhancement processing of video images, it effectively overcomes the limitations of traditional single video monitoring methods in complex environments such as rain, fog, and light changes, and significantly improves the clarity and feature recognition of video data.

[0087] Furthermore, this solution inputs both the anti-interference enhanced video keyframes and IoT sensor data into a pre-constructed multimodal large model. Visual features are extracted using a visual encoder, and after projection layer mapping, semantic features are aligned with the text embedding space of the large language model. Combined with a curated template of abnormal outflow identification prompts, reasoning is performed, fully leveraging the powerful semantic understanding and logical reasoning capabilities of the large language model. Cross-validation of multimodal data significantly improves the accuracy and robustness of determining whether there is outflow from the outlet and the flow level. Finally, based on the high-confidence identification results, corresponding early warning information is generated, achieving intelligent, all-weather, accurate monitoring and rapid response to abnormal outflow from the outlet, effectively reducing the cost of manual inspection and the risk of missed or false alarms. It also automatically triggers the generation and dispatch of maintenance work orders, realizing a complete closed loop from intelligent identification to business processing.

[0088] Example 2:

[0089] This embodiment provides a specific implementation of an intelligent inspection system for abnormal outflow from drainage outlets based on AI video recognition. This application is mainly applied to monitoring scenarios at the end of urban drainage networks, along rivers, or at industrial sewage outlets. Specifically, this embodiment focuses on the intelligent transformation of 130 key rainwater outlets distributed along a river in a certain city. These areas typically face complex environmental interferences such as light reflection and rain / fog obstruction.

[0090] Prior to implementation, this embodiment completed the necessary system deployment engineering. Specifically, existing high-definition surveillance cameras were upgraded in the target discharge area to obtain real-time video streams, and level gauges and rain gauges were simultaneously accessed to collect IoT sensor data. The front-end devices established a communication connection with the cloud monitoring platform via a wireless network, completing the access and basic integration of video streams and sensor data. Based on this, the cloud platform pre-configured a multimodal large model trained and debugged with a high-quality dataset, and configured a prompt word template containing seven-element rules, thereby constructing a complete monitoring environment with data acquisition, transmission, and intelligent reasoning capabilities.

[0091] 1. Acquire real-time video streams and IoT sensor data for the target discharge area.

[0092] After the system starts up, the cloud platform first acquires real-time video streams of the target discharge area via the standard RTSP protocol, and simultaneously receives synchronously collected sensor data via the Internet of Things (IoT) protocol. The IoT sensor data includes at least rainfall and liquid level data.

[0093] Video stream acquisition: The camera captures real-time images of the outlet area at a rate of 25 frames per second, and the video stream is transmitted to the cloud server via the RTSP protocol.

[0094] Sensor data acquisition: The level gauge reports the water level inside the outlet every minute, and the rain gauge reports the current rainfall intensity in real time. This data is uploaded to the cloud via IoT protocols and timestamped along with the video stream.

[0095] Preferably, to compensate for the limitations of single visual perception in extreme scenarios such as nighttime or obstructed vision, if the site environment permits, high-sensitivity microphones deployed near the outlet can be used to acquire audio spectrum data of the target outlet area. This audio spectrum data will be strictly spatiotemporally aligned with video streams and IoT sensor data, serving as a supplementary input dimension for multimodal large model inference. This audio spectrum data, as a key dimension for assisting in judging the water flow state, can effectively compensate for visual blind spots. For example, by analyzing the spectral characteristics of water flow impact sound, the system can accurately distinguish between broadband environmental white noise caused by rainfall and directional outflow sound (usually with specific low-frequency energy concentration characteristics) caused by sudden pressure changes inside the pipe.

[0096] In subsequent processing, the audio spectrum data will be strictly spatiotemporally aligned with the video stream and IoT sensor data, serving as a supplementary input dimension for multimodal large model inference, thereby constructing a comprehensive monitoring system integrating "visual-auditory-sensory".

[0097] This embodiment supports upgrading existing monitoring equipment: it is compatible with existing cameras via the RTSP protocol and uses an RS485 to 4G module to collect data from old rain gauges, allowing it to be connected to the system without replacing hardware.

[0098] 2. Perform preprocessing on the real-time video stream sequentially.

[0099] After acquiring the raw data, the system performs dynamic frame extraction, spatiotemporal alignment, and anti-interference enhancement processing on the real-time video stream in sequence.

[0100] 2.1 Dynamic frame skipping.

[0101] Specifically, dynamic frame extraction is performed on the real-time video stream to obtain video keyframes, including:

[0102] First, lightweight semantic segmentation is performed on the real-time video stream to identify the outlet area and the water surface area;

[0103] Secondly, a first optical flow feature is calculated within the outlet area. The first optical flow feature includes a first optical flow velocity amplitude and a first directional consistency. The first directional consistency is used to determine whether the pixel optical flow direction within the outlet area is radial from the outlet outward.

[0104] Simultaneously, a second optical flow feature is calculated within the water surface area. The second optical flow feature includes a second optical flow velocity amplitude and a second direction consistency. The second direction consistency is used to determine whether the pixel optical flow direction within the water surface area is a fan-shaped diffusion pattern away from the outlet.

[0105] Finally, based on the optical flow characteristics of the discharge area and the water surface area, the frame skipping strategy is adaptively and dynamically switched:

[0106] When the amplitude of the first optical flow velocity in the outlet area is lower than the first threshold (0.5 pixels / frame) and the amplitude of the second optical flow velocity in the water surface area is lower than the second threshold (0.5 pixels / frame), it is determined to be a static scene without water flow disturbance. Low-frequency frame extraction is adopted, and the frame extraction frequency is set to 30 minutes / frame to reduce data transmission and computing power consumption.

[0107] When the amplitude of the first optical flow velocity in the outlet area is higher than the first threshold (0.5 pixels / frame) and the consistency of the first direction is higher than the first direction threshold (e.g., the standard deviation of the direction angle is less than 15°), it is determined to be a dynamic scene of active outflow from the outlet. High-frequency frame sampling is adopted, and the frame sampling frequency is set to 1 minute / frame to ensure that the abnormal outflow characteristics are not lost.

[0108] In addition, when the amplitude of the second optical flow velocity in the water surface area is higher than the second threshold (0.5 pixels / frame) and the consistency of the second direction is higher than the second direction threshold (e.g., the standard deviation of the direction angle is less than 20°), it is determined to be a dynamic scene in which the water surface is driven by outflow to generate diffusion disturbance. High-frequency frame dropping is also used, with the frame dropping frequency set to 1 minute / frame to ensure that the abnormal state is completely captured.

[0109] For each video keyframe obtained by frame extraction, a sliding window mechanism is used to time-match it with IoT sensor data. The sliding window width is preset to ±2 seconds. If multiple sensor data exist within the window, the one with the smallest absolute difference from the video keyframe's timestamp is selected for binding; if there is no valid data within the window, the frame is marked as "sensor missing," and this frame will not participate in subsequent multimodal fusion determination to avoid false associations.

[0110] To ensure the reliability of video frame and sensor data timestamps during long-term operation, edge nodes monitor NTP synchronization status in real time. If NTP synchronization fails for more than 30 seconds, the edge node automatically switches to its local RTC clock to continue generating timestamps, avoiding data interruption due to network jitter. After the NTP connection is restored, the system performs retrospective correction on the evidence chain timestamps generated during the switchover: the correction is uniformly applied based on the clock offset between the local RTC and NTP, and the correction results (including the offset and the affected data range) are recorded in the log for future reference. The corrected evidence chain maintains temporal continuity, meeting the time consistency requirements for law enforcement tracing.

[0111] This step uses a dual-region optical flow judgment, combining the outlet area and the water surface area, rather than a single-region optical flow judgment. This allows for accurate differentiation between genuine outflow and false disturbances such as wind, raindrops, and reflections, reducing invalid frame drops and false triggers at the source and solving the technical problems of bandwidth waste and abnormal missed captures in traditional fixed frame drop strategies.

[0112] Lightweight semantic segmentation and optical flow feature computation have a significantly lower computational complexity than multimodal large-model inference. In practical deployments, the above steps can be executed in real-time by the edge device's central processing unit with extremely low power consumption, typically taking only milliseconds per process. In contrast, multimodal large-model inference requires the use of a graphics processing unit (GPU) or neural network processor, with each inference process taking tens to hundreds of times longer and consuming hundreds of times more energy than step 2. Therefore, by pre-filtering the real-time video stream and triggering high-computation inference only when the scene is determined to be dynamic, the overall average power consumption and computational resource usage of the system can be significantly reduced. This solves the problems of edge device overheating, response latency, and high hardware costs caused by continuous operation of large models in traditional solutions.

[0113] 2.2 Enhanced anti-interference capabilities.

[0114] After obtaining video keyframes and completing spatiotemporal alignment with sensor data, anti-interference enhancement is performed on the spatiotemporally aligned video keyframes based on the IoT sensor data, specifically including:

[0115] Determine whether the rainfall data in the IoT sensor data exceeds a preset rainfall threshold, and determine whether the rate of change of the liquid level data exceeds a preset liquid level threshold;

[0116] If the rainfall data exceeds a preset rainfall threshold, a preset defogging algorithm is invoked to perform image enhancement processing on the spatiotemporally aligned video keyframes in order to suppress the visual interference of raindrops on outflow recognition.

[0117] If the rainfall data does not exceed the preset rainfall threshold and the rate of change of the liquid level data exceeds the preset liquid level threshold, then the preset image sharpening and edge enhancement algorithm is invoked to process the spatiotemporally aligned video keyframes to highlight the dynamic texture features of the outlet water surface.

[0118] Otherwise, the original state of the video keyframes after the spatiotemporal alignment process is maintained, or only conventional color normalization is performed. That is, when there is no rainfall and the liquid level is stable, no enhancement processing is performed to preserve the fine-grained texture features in the original image and avoid loss of water flow details or introduction of artifacts due to over-enhancement.

[0119] As a further preferred option, after calling the rain and fog removal algorithm when the rainfall exceeds the threshold, if the liquid level change rate is also detected to exceed the threshold (indicating strong outflow on rainy days), image sharpening and edge enhancement algorithms can be further called to highlight the water flow texture, making it easier for the model to identify abnormal discharge behavior on rainy days.

[0120] The rain and fog removal algorithm is a single-image rain removal algorithm based on a convolutional neural network. This algorithm is pre-trained, and the training process includes: constructing a synthetic rain and fog dataset containing different rainfall intensities and backgrounds, and collecting monitoring images of drainage outlets in real rainy weather scenarios as training samples; training the model using a generative adversarial network architecture, where the generator is responsible for separating the clear background layer from the rainy image, and the discriminator is responsible for distinguishing the generated image from the real rainless image.

[0121] The rain and fog removal algorithm in this embodiment differs from general rain removal models by introducing regularization constraints based on prior knowledge of fluid motion physics. Specifically, leveraging the fact that real water flow conforms to the laws of fluid mechanics, a regularization term based on the consistency of optical flow direction is constructed in the training loss function. This regularization term mandates that the detected motion vector field must spatially conform to the physical distribution characteristics of radiation outward from the center of the outlet, thus mathematically eliminating false motions that do not conform to physical laws (such as the random motion of leaves blowing in the wind, light flickering, etc.). This design ensures that the algorithm can accurately preserve the dynamic texture features of the water flow while removing rain streaks, effectively preventing the smoothing of subtle water flow fluctuations as raindrop noise, thereby providing a reliable guarantee for the accurate identification of "outflow" features in subsequent large-scale models.

[0122] It should be noted that the anti-interference enhancement in this embodiment is selectively performed only under specific conditions (rainfall exceeding a threshold or a sudden rise in liquid level), and does not change the image resolution, cropping, or scaling, which is fundamentally different from conventional image quality enhancement preprocessing. For normal scenes without interference, the system still retains the original image directly input into the large model to fully preserve fine-grained texture features.

[0123] 3. Multimodal large model inference and feature alignment.

[0124] The enhanced video keyframes, along with corresponding IoT sensor data, are input into a pre-built multimodal large model (such as GPT-4o, Qwen-VL, etc.) for inference. Before inputting into the large model, this embodiment also introduces color space conversion and water quality anomaly feature extraction steps to enhance the model's sensitivity to water pollution characteristics. The specific processing flow is as follows:

[0125] 3.1 Color space conversion and extraction of water quality anomaly features.

[0126] First, the enhanced anti-interference video keyframes are converted from the conventional RGB color space to the LAB color space. The sensitivity of the L channel (luminance channel) in the LAB color space to changes in illumination is utilized to detect areas of abrupt brightness changes in the image; simultaneously, the A and B channels (color channels) are used to accurately identify color shifts in the water.

[0127] Secondly, based on a pre-set pollution feature database, targeted optical characteristic identification is performed on the image:

[0128] Oil film identification: Focus on capturing the rainbow-colored interference fringes and irregular halo patterns produced under illumination;

[0129] Foam recognition: Focuses on capturing the bright reflective areas and specific aggregated textures on the water surface.

[0130] Finally, the identified chromaticity shift features, oil film interference fringe features, and foam aggregation texture features are transformed into visual feature semantic description text (e.g., "Rainbow-colored interference fringes and local chromaticity shifts were detected on the surface of the discharge outlet water"), and dynamically integrated into subsequent prompt word templates to guide the multimodal large model to focus on these visual anomalies.

[0131] It should be noted that the above conversion to the LAB color space is only used to assist feature analysis and does not change the image data input to the multimodal large model; the model still actually receives the original RGB video keyframes. The semantic description text obtained from the analysis is incorporated into the cue word template as additional visual feature cues, while the image itself is still input into the model for inference in the original RGB format.

[0132] 3.2 Constructing prompt word templates and aligning multimodal features.

[0133] In this embodiment, the multimodal large model does not directly receive images for open-ended question-and-answer sessions. Instead, it loads a pre-constructed, dedicated prompt word template for abnormal outflow scenarios at discharge outlets. This prompt word template employs a structured seven-element composition method, with each part described below:

[0134] a) Role setting and input setting: Assign the model a specific professional identity and thinking background, the recognition ability it possesses, and clarify the types of input it receives;

[0135] b) Objectives and task boundaries: Clearly define the core objectives and absolute insurmountable boundaries of this task, and specify the area to be recognized by the model and the minimum number of pixels to be recognized;

[0136] c) Description of the object to be identified: Define the specific visual features of the object to be identified, including shape, color, texture, depth, spatial location, etc. At the same time, the visual features of the interfering objects should also be defined.

[0137] d) Recognition process description: guide the model to perform visual reasoning in a specific order, forcing it to "first locate, then eliminate interference, and then confirm features";

[0138] e) Level Description: Define threshold information such as the quantity, proportion, and quantitative description of each target category;

[0139] f) Confidence and Adoption Strategy: Each decision of the model is accompanied by a confidence quantification, and the model decision is only adopted when its confidence level is greater than a certain specific value;

[0140] g) Output format constraints: Force the model to output in a predefined data structure (such as JSON), and prohibit free text.

[0141] It also includes an interference exclusion rule unit, an outflow confirmation rule unit, a multi-source data cross-modal verification unit, and a dynamic few-sample example unit. The specific processing flow is as follows:

[0142] Visual Feature Extraction and Dynamic Few-Shot Augmentation: A visual encoder (such as VisionTransformer, ViT) in a multimodal large model is used to perform deep feature extraction on the enhanced video keyframes. During this process, the dynamic few-shot example unit first acquires real-time environmental state data of the current discharge outlet area (e.g., heavy rain or direct sunlight), and retrieves several typical interference images with the highest matching degree from a pre-constructed interference sample library (e.g., "raindrop spot samples under heavy rain" or "water surface reflection samples under strong afternoon sunlight"). The pre-constructed interference sample library is a set of negative samples of the discharge outlet environment that has been pre-collected and labeled, covering different lighting angles, rainfall intensities, and wall aging degrees.

[0143] This unit dynamically inserts these typical interference images and their corresponding "non-outflow" labels into the context window of the prompt word template as a reference benchmark for few-shot inference, thereby guiding the model to perform negative sample filtering on similar texture features in the current keyframe.

[0144] Text embedding and semantic alignment: IoT sensor data (rainfall values, liquid level values ​​and their rate of change) are converted into natural language descriptions and encoded through a text embedding layer. Subsequently, a pre-trained projection layer maps visual feature vectors to the text embedding space, achieving cross-modal alignment between visual features and text semantic features.

[0145] 3.3 Logical reasoning based on prompt word templates.

[0146] The aligned multimodal features are input into the large language model, and the model performs the following inference steps based on each rule unit in the prompt word template:

[0147] 3.3.1 Interference Removal (Negative Filtering): The interference removal rule unit executes interference filtering instructions based on image lighting features and time series features. Specifically:

[0148] Reflection exclusion rules: Identify bright areas with brightness values ​​significantly higher than the surrounding areas (such as brightness differences exceeding a preset threshold). If the edges are sharp and move with the viewing angle, they are marked as reflective interference.

[0149] Raindrop exclusion rules: Spots that are small, round or elliptical (with a diameter of 2-10 pixels), appear randomly, and last for a very short period of time (less than 3 frames) are identified as raindrop interference.

[0150] The pipe wall wetness exclusion rule is used to identify dark wet areas formed by water accumulation, seepage, condensation or residual moisture on the pipe wall below the outlet, the wall or the pipe wall to which the outlet belongs. The dark wet areas have no flow texture, are fixed in position in multiple frame image sequences, do not form water tongues or continuous water bodies, and are characterized as sheet-like or block-like wetness. These are marked as pipe wall wetness interference.

[0151] The shadow exclusion rule is used to identify low-brightness areas formed by light obstruction in and around the outlet. The low-brightness areas have soft or clear edges, no internal texture changes, and do not have the sense of downward extension of water flow or dynamic light flow characteristics. These areas are marked as shadow interference.

[0152] 3.3.2 Outflow Determination: The outflow confirmation rule unit only determines outflow when there is a moving area in the outlet region that meets the preset continuity conditions (e.g., more than 3 consecutive frames) and the optical flow direction is consistently outward (radial or fan-shaped diffusion). Simultaneously, the model uses a chain-of-thought approach to comprehensively calculate and output intermediate quantitative indicators such as pixel ratio, water flow width ratio, water flow velocity, and pipe wall condition, based on which the outflow level is classified as follows:

[0153] No outflow: The water surface is calm, and the disturbance amplitude is less than 5% of the outlet diameter; the pipe wall is dry, damp, or has old water stains (dark color, clear edges, no direction of extension); there may be stagnant water at the bottom of the pipe (calm water surface, no flow patterns, and no outward extension); there may be reflections or shadows near the pipe opening, but no signs of directional flow; the downstream water surface is calm or has only natural ripples (such as wind blowing or fallen leaves), and has no spatial connection with the outlet.

[0154] Small outflow (water flow but small volume): The water flow is in the form of dripping, linear flow or wall-following flow, with a slow velocity (visually estimated <0.2m / s); the water flow does not fill the pipe, and obvious gaps are visible inside the pipe opening, and the water body is thin and weak; the downstream water surface has only slight ripples or local color changes, the disturbance range is small, and there is no obvious diffuse plume.

[0155] Large outflow: The water flow is continuous, stable and has a large volume, which is manifested as: the water flow is obviously columnar, waterfall-like or jet-like, with a relatively fast flow velocity (visually estimated ≥0.2m / s), splashing white water droplets; the downstream water surface is obviously disturbed, the water flow spreads over a wide range, and may be accompanied by scum, colored bands or continuous turbid plumes.

[0156] 3.3.3 Cross-modal verification: The multi-source data cross-modal verification unit performs semantic comparison between visual recognition results and sensor data.

[0157] When rainfall data shows that the cumulative rainfall in the past 24 hours exceeds a preset threshold (e.g., 10 mm), it is preferentially marked as "normal rainfall runoff";

[0158] When rainfall data shows no rainfall in the past 72 hours (dry weather), the data is further combined with the change threshold of liquid level data and video recognition results to determine "large outflow in dry weather / small outflow in dry weather";

[0159] If the rainfall data shows no rainfall in the past 72 hours (dry weather) and the visual recognition result is no outflow, the output will be "No outflow".

[0160] 3.3.4 Confidence Threshold Adoption Strategy

[0161] According to the preset confidence adoption strategy in the prompt word template, when the multimodal large model outputs recognition results, a confidence quantification value (ranging from 0 to 1, representing the model's degree of certainty in its judgment) must be added to each judgment. The system only considers the recognition result reliable and uses it for subsequent warning triggering and business processing when the confidence of the model output is higher than the preset adoption threshold (e.g., 0.85). If the confidence is lower than the threshold, the recognition result is automatically marked as "awaiting manual review" and is not directly used to generate a warning push. Instead, it is stored in the review queue for maintenance personnel to view periodically or as a candidate sample for subsequent prompt word template optimization.

[0162] 3.4 Spatial consistency verification based on attention mechanism:

[0163] To further ensure the reliability of multimodal large model inference results and prevent the model from experiencing "illusions" or shifting its focus due to complex background interference, this embodiment introduces a spatial attention verification mechanism during model inference. The specific steps are as follows:

[0164] First, during the inference process of the multimodal large model, the attention weight matrix output by the visual encoder is extracted in real time. Based on this weight matrix, it is back-mapped to the original image space to generate an attention heatmap representing the visual attention regions of the multimodal large model. This heatmap can intuitively reflect which pixel regions in the image the model mainly focuses on when making judgments.

[0165] Secondly, the system obtains the preset spatial coordinate range (i.e., the bounding box of the actual physical location of the outlet) of the outlet area in the video keyframe after anti-interference enhancement. The system automatically determines whether the core pixels with weight values ​​higher than the preset weight threshold (such as the top 20% of high-weight areas) in the attention heatmap mainly fall within the preset spatial coordinate range.

[0166] Finally, perform differential output based on the spatial matching results:

[0167] If the high-weighted pixels fall within the preset spatial coordinate range, it indicates that the model is correctly focusing on the outlet area, and the system outputs the recognition result normally.

[0168] If a high-weight pixel does not fall within the preset spatial coordinate range (e.g., the model is overly focused on the swaying of trees in the background or distant vehicles), the current inference result is determined to have "spatial attention deviation". At this time, the system will output a recognition result with a "spatial attention deviation identifier", automatically generate spatial deviation feedback information, record it in the log, and prompt manual review.

[0169] 3.5 Hierarchical Structured Result Output and Application

[0170] Through the above process, the output of the multimodal large model adopts a hierarchical structured form.

[0171] The first level is the basic status identification result, including whether outflow was detected, the flow rate level (e.g., trickle, thin stream, gush), and the confidence level. This level of result is used to achieve real-time basic monitoring of the outlet status.

[0172] The second level is the refined analysis result (preferred output), which is based on the confirmation of outflow in the first level. It further outputs the outflow type (e.g., large outflow in dry weather, small outflow in dry weather, normal rainfall runoff) and abnormal water quality characteristics (e.g., oil film, foam, abnormal color).

[0173] This hierarchical output mechanism enables the system to quickly filter out invalid information based on the basic status identification results, and trigger refined analysis only when outflow is detected, thereby guiding maintenance personnel to make hierarchical responses: routine records are made for runoff with only flow and no water quality abnormalities, while high-level early warning work orders are generated for illegal discharge behavior accompanied by water quality abnormalities such as oil film and foam, so as to achieve precise scheduling of maintenance work.

[0174] 3.6 Example of a prompt word template:

[0175] The prompt word template in this embodiment uses structured natural language instructions, and its specific content is shown in the following example:

[0176] You are a seasoned expert in inspecting urban drainage systems. Based on the provided monitoring footage and sensor data, please analyze the status of the drainage outlets following these steps:

[0177] Step 1: Pay attention to abnormal water quality characteristics;

[0178] Based on the visual feature descriptions provided by the system, please pay close attention to whether there are rainbow-colored interference fringes (oil film features) or bright aggregated textures (foam features) in the image, and whether there is a significant color shift in the water body.

[0179] Step 2: Eliminate visual distractions;

[0180] Please observe the image carefully. If you find any of the following features, please mark them as interference and ignore them:

[0181] Reflective interference: High-brightness, sharp-edged, and irregularly fluctuating areas in the image depending on the viewing angle or time; often appearing on the upper half, side, or edge of the pipe's inner wall, characterized by concentrated brightness, specular reflection (bright spots), clear or blurred boundaries, but without flowing textures, and the position of the reflection is consistent with the direction of the light source (which can be inferred from the direction of the shadows of other objects in the image);

[0182] Raindrop interference: Blurry spots that appear randomly in the image, are circular or elliptical in shape, and last for a very short number of frames;

[0183] Water stain interference: dark-colored, non-flowing, and fixed damp areas on the pipe wall or wall at the outlet;

[0184] Pipe wall wetting interference: The outlet and the pipe wall are wetted in patches due to condensation or residual moisture. The color is darkened, there is no water flow pattern, and no water tongue or continuous water body is formed.

[0185] Shadow interference: Low-brightness areas formed by the edge of the outlet and surrounding objects blocking light. The color is dark, the edges are soft or clear, there is no texture variation, it blends into the surrounding environment, and it is related to the shape of the obstruction. It may appear as an arc or strip, but it does not have the sense of downward extension of water flow.

[0186] Step 3: Identify outflow characteristics;

[0187] After eliminating the above interferences, please determine whether there is a real outflow from the outlet. A real outflow must simultaneously meet the following conditions: the water flow is continuous in time, and the direction of the light flow shows a tendency to radiate outward from the center of the outlet or to spread in a fan shape.

[0188] The outflow situation must be judged by comprehensively considering the flow pattern, pipe wall characteristics, and water surface conditions.

[0189] Water flow status: whether it extends continuously: the water flow extends outward from the pipe opening, forming a distinct water tongue or ribbon-like water body, connecting with the downstream water area, and the direction of extension is affected by gravity; transparency and color: the water flow may be clear or turbid, but usually has a significant color difference from the pipe wall material, and the flow produces changes in light and shadow; the water flow always flows in the direction of gravity (downward, outward), while reflections and shadows are not constrained by gravity.

[0190] Water surface condition: Surface texture: The surface of flowing water has ripples, waves, bubbles or white foam (splashes), and these textures have a dynamic feel; when water flows into downstream water, it will cause directional disturbance, diffusion or turbid plume, the disturbance area directly corresponds to the outlet location, and the disturbance shape is consistent with the direction of water flow impact.

[0191] Step 4: Based on the outflow characteristics, a comprehensive judgment is made in conjunction with data from sensors such as rainfall and liquid level.

[0192] If there is no outflow, it is directly judged as "no outflow";

[0193] If there is outflow (large / small outflow) and there is no rainfall, further combine the change threshold of liquid level data to determine "large outflow in dry weather / small outflow in dry weather";

[0194] If there is outflow (large / small outflow) and rainfall occurs, it is classified as "normal rainfall runoff".

[0195] Output requirements:

[0196] Please strictly follow the following output format:

[0197] Interference detected: [Yes / No, please specify the type of interference if detected];

[0198] Is there any water quality abnormality: [Yes / No, if so, please specify oil film / foam / color shift];

[0199] Is there an outflow: [Yes / No];

[0200] Final assessment result: [Large outflow during drought / Small outflow during drought / Normal rainfall runoff / No outflow];

[0201] Confidence level: [0-1].

[0202] 4. Early warning information generation and processing.

[0203] Based on the recognition results output by the multimodal large model and the aforementioned confidence threshold adoption strategy, the system automatically executes subsequent business logic:

[0204] Result parsing and filtering: The system parses the JSON data output by the model and first determines whether the confidence level is higher than a preset threshold (e.g., 0.85). If it is lower than the threshold, the frame is marked as awaiting manual review and no warning is triggered directly, thereby reducing the false alarm rate.

[0205] Warning generation: If the confidence level meets the standard and the identification result is "outflow", the system matches the corresponding warning level (important, urgent) according to the outflow flow level. Specifically, a small amount of outflow corresponds to an important warning, which is pushed to the area grid member, requiring on-site verification within 24 hours; a large amount of outflow corresponds to an urgent warning, which is pushed to the operation and maintenance team leader and environmental law enforcement department, requiring a response within 2 hours.

[0206] Simultaneously, the system automatically generates a structured chain of evidence: extracting a complete video clip from 60 seconds before to 60 seconds after the abnormal event, extracting at least 3 keyframe images from this clip, and associating it with IoT sensor data for the corresponding time period, multimodal large-scale model recognition process logs, device information, and spatiotemporal coordinates. The system calculates a unique encrypted verification value (Hash value) for the above evidence chain data packet using the SHA-256 hash algorithm. This hash value is stored together with the evidence chain data packet in a cloud-based tamper-proof log. Any subsequent modification to the evidence chain data packet will result in a hash value mismatch, thus ensuring the immutability and traceability of the evidence chain. After the evidence chain is generated, it is automatically associated with the early warning work order.

[0207] Information push: Generate early warning information including outlet location, real-time captured images (keyframes after anti-interference enhancement), sensor data snapshots and traffic level assessment, and push it to relevant operation and maintenance personnel in real time via mobile APP or SMS to guide them to carry out accurate on-site handling.

[0208] If the outflow rate reaches the preset sampling trigger level (such as large outflow or small outflow), the system will automatically generate a sampling reminder instruction, which includes the outlet location, suspected pollutant type and suggested sampling time window, and push it to the mobile terminal and synchronize it to the control interface of the physical sampling device.

[0209] Upon arrival at the scene, the maintenance personnel conducted verification and took appropriate action.

[0210] If illegal discharge or abnormal outflow is confirmed, upload photos and videos of the on-site handling, rapid water quality test results (if any), sampling bottle number, and other information via mobile device, and update the work order status to "handled".

[0211] If the on-site verification confirms it's a false alarm (e.g., mistaking a reflection for water flow), select the "False Alarm" tag and upload a description and a photo of the scene. The system will automatically add that frame and its tag ("False Alarm - Reflection") to the false alarm test set.

[0212] 5. Closed-loop self-evolution of multimodal large models.

[0213] After step S4 is executed, this embodiment also includes a closed-loop self-evolution step for a multimodal large model, which aims to optimize the model's judgment capabilities by continuously learning from historical data. The specific process is as follows:

[0214] According to a preset iteration cycle (e.g., weekly or monthly), the system automatically triggers the update process of the prompt word template. In the evolutionary process of the multimodal large model, the system first retrieves samples from historical recognition results with confidence levels within a preset ambiguity range (e.g., confidence levels between 40% and 60%), as well as alarm data that has been manually reviewed and corrected (e.g., samples falsely reported as illegal discharge by the model but manually corrected as reflective). The system encapsulates these two types of data into a high-value difficult example sample set. Using this difficult example sample set, the prompt word template is lightweightly optimized: for example, by adding new interference descriptions (e.g., "water marks on the wall after rain"), refining the judgment threshold (e.g., adjusting the water flow pixel ratio threshold from 2% to 1.5%), and adjusting the recognition process order. Simultaneously, screenshots confirmed as false alarms by manual review are added to a pre-built false alarm test set for subsequent iterative testing of the prompt word template, continuously reducing the false alarm rate. The optimized prompt word template is automatically deployed to the cloud, achieving continuous improvement in recognition performance without retraining the model.

[0215] Through the aforementioned business loop, this embodiment directly transforms AI recognition results into executable operation and maintenance tasks, forming a complete intelligent inspection chain of "identification → early warning → work order → handling → feedback → optimization".

[0216] This embodiment synchronizes video keyframes with sensor data such as rainfall and liquid level in time through dynamic frame extraction and spatiotemporal alignment, providing a unified data foundation for subsequent joint inference;

[0217] By calling the rain and fog removal algorithm based on rainfall data and the image sharpening algorithm based on the liquid level change rate, targeted anti-interference enhancement of video key frames was achieved, improving image quality under rainy or rapidly changing water level conditions and reducing the interference of environmental noise on recognition.

[0218] By using a multimodal large-scale model visual encoder and language decoder to align semantic features of image and sensor data, it is possible to make joint judgments by combining visual features and numerical data, thereby improving the accuracy of outflow status recognition.

[0219] By loading preset prompt word templates, large models can output whether there is outflow, flow level and confidence level without retraining for the outlet recognition task, thus reducing model deployment and computing power overhead.

[0220] An attention space verification mechanism is added to the inference process. By comparing the attention heatmap with the preset coordinate range of the outlet, it can detect whether the model is paying attention to the correct area and output a labeled result when it deviates, thus enhancing the reliability of the inference results.

[0221] Based on the identification results, a graded early warning information (important / urgent warning) is generated, and video clips and key frames before and after the anomaly are automatically extracted to generate a unique evidence chain ID, providing structured evidence for subsequent handling and tracing.

[0222] Through a closed-loop self-evolutionary process, the system adaptively updates the judgment threshold and interference filtering rules in the prompt word template by using samples with confidence levels in the fuzzy range and data after manual verification. This enables the system to continuously optimize over time and reduce false alarm and false negative rates.

[0223] Example 3:

[0224] This embodiment provides an intelligent inspection system for abnormal outflow from discharge outlets based on AI video recognition. Figure 3 This is a schematic diagram of the intelligent inspection system architecture for abnormal outflow at the discharge outlet in this embodiment. Figure 3 As shown, this embodiment proposes a layered, decoupled, and highly collaborative system core architecture. This architecture is divided into a perception layer, an analysis layer, and a processing and feedback layer from bottom to top, aiming to solve the problems of difficult identification and slow response to illegal discharge behavior in complex environments. Specifically, it includes:

[0225] 1. Perception layer: Holographic acquisition of multidimensional data;

[0226] The perception layer is responsible for the digital mapping of the physical world.

[0227] Hardware deployment: Deploy intelligent video surveillance equipment (such as high-definition PTZ cameras supporting the RTSP protocol), IoT sensors (including rain gauges, level gauges, and optional water quality sensors) and high-sensitivity microphones (to capture water flow impact sounds and environmental noise) in the target discharge area.

[0228] In addition to conventional video stream and sensor data acquisition, this layer introduces a mechanism for acquiring audio spectrum data. Ambient sound is captured via microphones, compensating for the limitations of single-vision perception in low-light conditions or when vision is obstructed. All front-end devices are responsible for initial formatting and caching of the raw data, providing a standardized data source for subsequent processing.

[0229] 2. Analysis Layer: Spatiotemporal alignment and multimodal intelligent reasoning;

[0230] The analysis layer adopts a hybrid architecture that combines edge computing and cloud computing, and is the core part of intelligent recognition.

[0231] Edge-side preprocessing: Edge nodes (such as industrial-grade edge computing gateways deployed near discharge outlets) execute dynamic frame extraction strategies in real time, adaptively adjusting the frame extraction frequency based on the optical flow characteristics of the discharge outlet area and the water surface area (30 minutes / frame for static scenes, 1 minute / frame for dynamic scenes). Simultaneously, based on IoT sensor data (whether rainfall exceeds thresholds, whether the liquid level change rate is abnormal), de-fogging algorithms or image sharpening algorithms are selectively invoked for anti-interference enhancement. This lightweight edge-side processing significantly reduces cloud bandwidth pressure and ensures that key frames for abnormal events are not lost.

[0232] Deep cloud analytics: The cloud platform receives video keyframes and aligned sensor data uploaded from edge nodes and performs the following analysis:

[0233] Spatiotemporal alignment: Strict alignment between video keyframes and rainfall and liquid level data is achieved through a timestamp sliding window (window width ±2 seconds); if NTP synchronization fails, it automatically switches to the local RTC clock and backtracks to correct after recovery to ensure the temporal continuity of the evidence chain.

[0234] Multimodal large-scale model inference: Aligned data is input into a pre-built multimodal large-scale model (such as GPT-4o, Qwen-VL). The model combines a pre-built interference sample library (containing negative samples from different lighting, rainfall, wall aging, etc.) for dynamic few-shot learning. That is, based on the current real-time environmental state (such as heavy rain, strong light), it retrieves the most similar typical interference image and uses it as a reference benchmark for "non-outflow" into the context window of the prompt word template. The model comprehensively analyzes visual features (water flow direction, texture, oil film / foam optical properties), IoT sensor values ​​(whether the rainfall is 0, liquid level change rate), and auxiliary audio spectrum features (optional) to accurately determine whether there is abnormal outflow and outputs the outlet location, whether there is outflow, flow level (none / small amount / large amount), reason for judgment, and confidence level.

[0235] Tiered early warning: Based on the confidence level of the inference results (adoption threshold 0.85) and the traffic level, a tiered early warning strategy is automatically executed: a small outflow during dry weather triggers an "important warning" (checked within 24 hours), and a large outflow during dry weather triggers an "emergency warning" (responded within 2 hours), and high-risk events are pushed to the handling and feedback layer in real time.

[0236] 3. Feedback and Handling Layer: Business Closed Loop and Model Iteration;

[0237] The feedback layer is responsible for translating the results of intelligent analysis into actual management actions and forming a data-driven continuous optimization loop.

[0238] Work Order Flow: The cloud-based work order management module automatically converts early warning information and a structured evidence chain (which generates a unique encrypted verification value using the SHA-256 hash algorithm to ensure immutability) into standardized maintenance work orders. Work order fields include outlet location, early warning level, response time limit, handling requirements, and evidence chain link. Based on the early warning level, the response time limit and candidate personnel are determined. Then, considering the maintenance personnel's jurisdiction, current workload, and skill level, target personnel are selected from the candidates. The work order is accurately dispatched to their mobile maintenance terminal via the MQTT protocol or by calling the backend HTTP interface. Specifically, the cloud can proactively push work orders to mobile terminals, or the mobile maintenance terminal can proactively retrieve work orders by calling the backend HTTP interface; both methods ensure reliable work order dispatch. After arriving on-site, maintenance personnel receive navigation and evidence collection guidance via their mobile terminals, and upload on-site handling photos, videos, sampling records, and handling results. The cloud automatically verifies the integrity of the feedback data and completes work order archiving.

[0239] Data feedback drives iteration: If the processing results contain false alarm markers, the corresponding keyframes are automatically marked as negative samples and added to the false alarm test set. According to a preset iteration cycle (e.g., weekly), the system automatically retrieves samples from historical recognition results with confidence levels in the ambiguous range (e.g., 40%~60%), as well as samples marked as false alarms / missed alarms, and encapsulates them into a high-value difficult example sample set. Lightweight adaptive optimization is then performed only on the interference filtering rules, outflow judgment thresholds, and grading standards in the structured prompt word template, without retraining the entire multimodal large model. The optimized template is automatically deployed after regression validation through the false alarm test set. Through this "data feedback - prompt word evolution" mechanism, the system can continuously adapt to new outflow scenarios and environmental changes, achieving self-iteration of recognition accuracy and long-term stable operation.

[0240] The intelligent inspection system architecture for abnormal outflow at discharge outlets provided in this embodiment has the following beneficial effects: 1. Layered decoupling and efficient collaboration: By dividing the system into a bottom-up perception layer, analysis layer, and handling feedback layer, the responsibilities of each layer are clearly defined and the interfaces are clear, reducing the coupling between modules. The perception layer focuses on multi-source data collection, the analysis layer utilizes edge-cloud collaboration to achieve real-time inference, and the handling feedback layer completes the business loop. The overall architecture supports independent upgrades and horizontal expansion.

[0241] 2. Enhanced Multi-Dimensional Perception for Complex Environments: In addition to video and IoT sensors, the perception layer can also incorporate audio spectrum acquisition, effectively compensating for blind spots in nighttime or visually obstructed scenarios. It also supports the reuse and adaptation of existing monitoring and sensing equipment, reducing system deployment and upgrade costs.

[0242] 3. Edge-cloud collaboration, balancing real-time performance and cost-effectiveness: Edge nodes perform lightweight dynamic frame extraction and anti-interference enhancement, filtering redundant data and reducing cloud pressure; the cloud deploys multimodal large models for high-precision inference, balancing real-time response and deep analysis capabilities.

[0243] 4. Dynamic few-sample inference with strong scene adaptability: The analysis layer introduces a dynamic retrieval and injection mechanism of the interference sample library in the multimodal large model inference. It matches typical interference images as negative sample references according to the real-time environmental state, and can quickly adapt to different discharge port scenarios without retraining the model.

[0244] 5. Tiered early warning and automated work order closed loop: Based on the tiered early warning strategy of confidence and traffic level, combined with the intelligent dispatching of work orders according to the region, load and skill level of the operation and maintenance personnel, the entire process from early warning to work order dispatch, on-site handling and feedback archiving is automated and closed loop, which significantly improves the efficiency of supervision and response.

[0245] 6. Data feedback-driven self-evolution of prompt words: The interference filtering rules, judgment thresholds and grading standards in the prompt word template are optimized periodically by using false positive negative samples and fuzzy interval samples. After regression verification, they are automatically put into operation, achieving continuous evolution of "the more you use, the more accurate it becomes". There is no need to retrain a multimodal large model, and the iteration cost is low.

[0246] Example 4:

[0247] Finally, this application also proposes a computing device, which includes a processor and a memory. The memory stores a computer program, and the processor executes the instructions stored in the memory so that the computer device performs the intelligent inspection method for abnormal outflow of discharge outlets based on AI video recognition described in the above embodiments.

[0248] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make modifications, alterations, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A method for intelligent inspection of abnormal outflow from discharge outlets based on AI video recognition, characterized in that, include: S1. Obtain real-time video stream and IoT sensor data of the target discharge outlet area, wherein the IoT sensor data includes at least rainfall data and liquid level data; S2. Dynamically extract frames from the real-time video stream to obtain video keyframes; The video keyframes are spatiotemporally aligned with the IoT sensor data, and the spatiotemporally aligned video keyframes are enhanced against interference based on the IoT sensor data to obtain the enhanced video keyframes. S3. Input the anti-interference enhanced video keyframes and the IoT sensor data into a pre-constructed multimodal large model. Extract the visual features of the anti-interference enhanced video keyframes through the visual encoder in the multimodal large model. After mapping by the projection layer, align the semantic features with the text embedding space of the large language model. Load the preset outlet abnormal outflow identification prompt word template for inference and output the identification result. The identification result includes outlet location, whether there is outflow, outflow level, judgment reason, and corresponding confidence level. S4. Based on the identification results, generate warning information of the corresponding level, and simultaneously capture video clips and key frames before and after the occurrence of the abnormal event, associate sensor data and identification logs, and generate an immutable structured chain of evidence. S5. Automatically convert early warning information and structured evidence chains into standardized operation and maintenance work orders, match response time limits and handling personnel according to the early warning level, and dispatch them to mobile operation and maintenance terminals through IoT protocol; receive and store on-site handling feedback data to complete the work order closed loop; S6. Based on the on-site handling feedback data, the abnormal outflow identification prompt word template of the discharge outlet is adaptively optimized.

2. The method according to claim 1, characterized in that, In step S2, dynamic frame extraction is performed on the real-time video stream to obtain video keyframes, including: Lightweight semantic segmentation is performed on the real-time video stream to identify the outlet area and the water surface area; A first optical flow feature is calculated within the outlet area. The first optical flow feature includes a first optical flow velocity amplitude and a first directional consistency. The first directional consistency is used to determine whether the pixel optical flow direction within the outlet area is radial outward from the outlet. A second optical flow feature is calculated within the water surface area. The second optical flow feature includes a second optical flow velocity amplitude and a second direction consistency. The second direction consistency is used to determine whether the pixel optical flow direction within the water surface area is a fan-shaped diffusion pattern away from the outlet. When the amplitude of the first optical flow velocity is lower than the first threshold and the amplitude of the second optical flow velocity is lower than the second threshold, it is determined to be a static scene, and the frame extraction frequency is set to 30 minutes / frame; When the amplitude of the first optical flow velocity is higher than the first threshold and the consistency of the first direction is higher than the first direction threshold, it is determined to be a dynamic scene, and the frame sampling frequency is set to 1 minute / frame. Alternatively, when the amplitude of the second optical flow velocity is higher than the second threshold and the consistency of the second direction is higher than the second direction threshold, it is determined to be a dynamic scene, and the frame sampling frequency is set to 1 minute / frame.

3. The method according to claim 1, characterized in that, In step S2, anti-interference enhancement is performed on the spatiotemporally aligned video keyframes based on the IoT sensor data, including: Determine whether the rainfall data in the IoT sensor data exceeds a preset rainfall threshold, and determine whether the rate of change of the liquid level data exceeds a preset liquid level threshold; If the rainfall data exceeds a preset rainfall threshold, a preset defogging algorithm is invoked to perform image enhancement processing on the spatiotemporally aligned video keyframes in order to suppress the visual interference of raindrops on outflow recognition. If the rainfall data does not exceed the preset rainfall threshold and the rate of change of the liquid level data exceeds the preset liquid level threshold, then the preset image sharpening and edge enhancement algorithm is invoked to process the spatiotemporally aligned video keyframes to highlight the dynamic texture features of the outlet water surface. Otherwise, maintain the original state of the video keyframes after the spatiotemporal alignment process, or perform only regular color normalization.

4. The method according to claim 1, characterized in that, The abnormal outflow identification prompt word template for the discharge outlet includes seven elements: role setting and input setting, target and task boundary, description of the identification object, description of the identification process, level description, confidence and adoption strategy, and output format constraints; the abnormal outflow identification prompt word template for the discharge outlet includes an interference exclusion rule unit, which is configured with interference filtering instructions based on image light and shadow features and time series features; The interference filtering instruction is used to identify reflective areas, raindrop spots, and fixed water stains in the image, and mark them as interference items to exclude from the outflow judgment. The abnormal outflow identification prompt template for the outlet also includes an outflow confirmation rule unit, which is used to determine that there is outflow only when there is a moving area in the outlet area that meets the preset continuity condition and the optical flow direction is consistent and outward.

5. The method according to claim 4, characterized in that, The abnormal outflow identification prompt word template for the discharge outlet also includes a multi-source data cross-modal verification unit; The verification unit is configured with logical operation rules, which are used to semantically compare the outflow results identified by vision with the synchronously acquired rainfall data and liquid level data. When the visual recognition result indicates that there is outflow and the rainfall data meets the preset no-rain conditions, a dry weather outflow (large / small) judgment result is generated; when the visual recognition result indicates that there is outflow and the liquid level data change exceeds the preset threshold, an outflow risk judgment result is generated.

6. The method according to claim 4, characterized in that, The interference filtering instructions include: The reflection exclusion rule is used to identify highlight areas in an image whose brightness value is higher than a preset multiple of the surrounding area, whose edges are sharp, and whose position or shape changes over time in a multi-frame image sequence, and to mark them as reflection interference. Raindrop exclusion rules are used to identify circular or elliptical spots in an image that are within a preset pixel range, appear randomly, and persist for less than a preset frame threshold, and mark them as raindrop interference. The pipe wall wetness exclusion rule is used to identify dark wet areas formed by water accumulation, seepage, condensation or residual moisture on the pipe wall below the outlet, the wall or the pipe wall to which the outlet belongs. The dark wet areas have no flow texture, are fixed in position in multiple frame image sequences, do not form water tongues or continuous water bodies, and are characterized as sheet-like or block-like wetness. These are marked as pipe wall wetness interference. The shadow exclusion rule is used to identify low-brightness areas formed by light obstruction in and around the outlet. The low-brightness areas have soft or clear edges, no internal texture changes, and do not have the sense of downward extension of water flow or dynamic light flow characteristics. These areas are marked as shadow interference.

7. The method according to claim 1, characterized in that, The structured chain of evidence in S4 includes: The complete video clip from 60 seconds before to 60 seconds after the abnormal event, at least 3 keyframe images extracted from the complete video clip, IoT sensor data aligned with the complete video clip in time range; multimodal large model recognition process log, device information and spatiotemporal coordinates; The evidence chain generates a unique encrypted identifier, which has the characteristics of being tamper-proof and traceable.

8. The method according to claim 1, characterized in that, Step S5 includes: Analyze early warning information and structured evidence chains, and automatically populate work order fields, including outlet location, early warning level, response time limit, handling requirements, and evidence chain links; The response time limit and candidate personnel are determined based on the warning level. Then, the target personnel are selected from the candidate personnel based on the management area, current workload and skill level of the operation and maintenance personnel. The standardized operation and maintenance work order is then dispatched to the mobile operation and maintenance terminal of the target personnel. Send navigation routes and evidence collection instructions to the mobile maintenance terminal; receive on-site handling data uploaded by the mobile maintenance terminal, including handling photos, handling videos, sampling records, and handling results; automatically update the work order status based on the on-site handling data; The system automatically verifies the integrity of the on-site handling data and automatically archives the work order when the verification passes. If the on-site handling data contains false alarm identifiers, the corresponding keyframes are automatically marked as negative samples for use in optimizing the abnormal outflow identification prompt word template of the discharge outlet.

9. The method according to claim 1, characterized in that, Step S6 includes: According to the preset iteration cycle, samples with confidence levels within the preset fuzzy range in the historical recognition results, as well as samples marked as false alarms or missed alarms, are retrieved and packaged into a high-value difficult example sample set; the high-value difficult example sample set is used to automatically optimize the abnormal outflow recognition prompt word template of the discharge outlet; after optimization, regression verification is performed through the false alarm test set, and the system is automatically put into operation after the verification is passed.

10. An intelligent inspection system for abnormal outflow from discharge outlets based on AI video recognition, characterized in that, include: The perception layer is used to collect real-time video streams and IoT sensor data of the target discharge area, and supports compatibility with existing monitoring equipment protocols and reuse of existing equipment. The analysis layer includes edge nodes and a cloud platform; the edge nodes are used to perform dynamic frame extraction, spatiotemporal alignment, and anti-interference enhancement processing. The cloud platform deploys a multimodal large model and anomaly outflow identification prompt word templates to achieve cross-modal reasoning and output identification results; The early warning and evidence chain layer is used to generate hierarchical early warning information based on the identification results, and automatically generate and store an immutable structured evidence chain; The maintenance work order layer is used to automatically convert early warning information into standardized maintenance work orders, realizing automatic work order dispatch, interaction with mobile terminals, and closed-loop handling and feedback. An iterative optimization layer is used to achieve closed-loop optimization of the abnormal outflow identification prompt word template of the discharge outlet based on the handling feedback data.

Images