Visual recognition-based underground pipe gallery fault early warning method and system, and storage medium

Abstract

This invention proposes a method, system, and storage medium for early warning of underground utility tunnel faults based on visual recognition. The method includes: acquiring real-time monitoring video streams of the underground utility tunnel; preprocessing the real-time monitoring video streams to obtain a target image sequence; inputting the target image sequence into a multi-task visual recognition model to obtain multiple visual recognition results; performing three-dimensional spatiotemporal alignment and fusion analysis on the multiple visual recognition results to determine multiple comprehensive risk levels; and generating a fault report and outputting early warning information based on the multiple comprehensive risk levels. The fault early warning method provided by this invention synchronously outputs the recognition results of water leakage, cracks, equipment status, and moving objects through multi-task visual recognition model inference; and combines spatiotemporal alignment with a risk rule base for fusion analysis, transforming a single recognition result into a comprehensive risk assessment of the underground utility tunnel, thereby rapidly generating fault reports, timely achieving fault early warning, and improving the early warning accuracy and comprehensive perception capability of utility tunnel monitoring.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method, system and storage medium for early warning of faults in underground utility tunnels based on visual recognition. Background Technology

[0002] As a critical infrastructure for the centralized laying of various municipal pipelines, the safe and stable operation of urban underground utility tunnels is essential for maintaining the normal functioning of the city. The internal environment of these tunnels is typically enclosed and complex, operating under conditions of high humidity and low light year-round. This presents various potential safety risks, including water leakage, structural deformation and cracks, abnormal operation of critical equipment, and intrusion by foreign objects or unauthorized personnel. Traditional manual inspection methods are not only inefficient and costly in terms of manpower, but also fail to achieve comprehensive, round-the-clock safety coverage.

[0003] In practical applications, there are two main drawbacks: "single-mode" and "system redundancy," making it difficult to balance algorithmic intelligence and engineering economy. Firstly, visual detection solutions are highly specialized, limited to single-risk identification. For example, deep learning models for leak detection only optimize the edge and color features of specific defects, lacking a multi-task parallel processing mechanism. This makes them unsuitable for diverse and heterogeneous risk scenarios within utility tunnels, such as crack extraction, equipment malfunctions, and object intrusion. Achieving comprehensive monitoring requires the sequential deployment of multiple independent models, resulting in computational redundancy and disrupting the synergy of comprehensive risk perception. Secondly, multi-sensor fusion solutions adopted to compensate for visual limitations heavily rely on the stacking of numerous physical hardware components such as fiber optic gratings and lidar. This not only leads to complex system topologies and difficulties in synchronous calibration but also incurs high procurement and maintenance costs. More importantly, for the vast number of existing old utility tunnels, rewiring and adding such networks is almost impossible. Summary of the Invention

[0004] This invention provides a method, system, and storage medium for early warning of underground utility tunnel faults based on visual recognition, in order to solve technical problems such as limited functionality, system redundancy, and difficulty in hardware deployment when conducting safety inspections of underground utility tunnels.

[0005] In a first aspect, the present invention provides a method for early warning of underground utility tunnel faults based on visual recognition, comprising: Obtain real-time monitoring video streams of underground utility tunnels; The real-time monitoring video stream is preprocessed to obtain the target image sequence; The target image sequence is input into a pre-built multi-task visual recognition model to obtain multiple visual recognition results, wherein the multi-task visual recognition model is used to perform water leakage area recognition, structural crack recognition, key equipment status detection, and moving object tracking and detection. The multiple visual recognition results are aligned in three dimensions in a spatiotemporal manner, and a preset risk rule base is called to perform a fusion analysis on the spatiotemporally aligned multiple visual recognition results to determine multiple comprehensive risk levels. The preset risk rule base includes association judgment rules for different visual recognition results in a preset spatial domain and within a continuous time window. A fault diagnosis report for the underground utility tunnel is generated based on multiple comprehensive risk levels, and early warning information is generated based on the fault diagnosis report.

[0006] In one embodiment of the present invention, preprocessing the real-time monitoring video stream to obtain a target image sequence includes: decoding and frame-segmenting the real-time monitoring video stream to obtain an initial image sequence; The target image sequence is obtained by extracting key regions, correcting illumination, and filtering noise from each frame of the initial image sequence.

[0007] In one embodiment of the present invention, key region extraction, illumination correction, and noise filtering are performed on each frame of the initial image sequence to obtain the target image sequence. This includes: extracting a region of interest (ROI) corresponding to a preset key region from each frame of the image based on camera calibration parameters, and adjusting the ROI to a uniform size; selecting an illumination reference region within the ROI, and adaptively adjusting the ROI based on the pixel features of the illumination reference region to obtain a corrected ROI; and performing a hybrid filtering process on the corrected ROI to obtain the target image sequence.

[0008] In one embodiment of the present invention, the multi-task visual recognition model includes: a shared backbone network for extracting multi-scale shared feature maps from the target image sequence; a first task branch for decoding and segmenting the multi-scale shared feature maps to output a leakage segmentation result; a second task branch for extracting linear features from the multi-scale shared feature maps to output a crack detection result; a third task branch for detecting key equipment anchor points from the multi-scale shared feature maps to output an equipment status result; and a fourth task branch for tracking moving object anchor points from the multi-scale shared feature maps to output a moving object trajectory result; wherein the multiple visual recognition results include the leakage segmentation result, the crack detection result, the equipment status result, and the moving object trajectory result.

[0009] In one embodiment of the present invention, the construction steps of the multi-task visual recognition model include: obtaining a labeled training sample set, the labeled training sample set including leakage labeled samples, crack labeled samples, equipment status labeled samples, and moving object labeled samples; using the labeled training sample set to perform multi-task joint training on the shared backbone network and each task branch to obtain the prediction results of each task branch; determining the loss value between the prediction results of each task branch and the corresponding labeled ground truth value based on the loss function; adjusting the parameters of the shared backbone network and each task branch using the loss value until the loss value converges to obtain the multi-task visual recognition model.

[0010] In one embodiment of the present invention, multiple visual recognition results are 3D spatiotemporally aligned, and a preset risk rule base is invoked to perform fusion analysis on the spatiotemporally aligned multiple visual recognition results to determine multiple comprehensive risk levels. This includes: extracting the two-dimensional image coordinates and timestamps of multiple visual recognition results from the target image sequence; mapping the two-dimensional image coordinates of each visual recognition result to the three-dimensional spatial coordinates of the underground utility tunnel based on the calibration parameters, and synchronizing the three-dimensional spatial coordinates according to the timestamps; a single visual recognition result and its corresponding three-dimensional spatial coordinates and timestamp constitute a risk event; multiple risk events form a risk event sequence according to the timestamp order; and invoking the preset risk rule base to perform coexistence analysis in a preset spatial domain and change analysis within a continuous time period on the risk event sequence to obtain multiple comprehensive risk levels.

[0011] In one embodiment of the present invention, the preset risk rule base is invoked to perform coexistence analysis in a preset spatial domain and change analysis within a continuous time window on the risk event sequence to obtain multiple comprehensive risk levels, including: determining risk type entities in the risk event sequence based on multiple visual recognition results; if multiple risk type entities have a coexistence relationship in the preset spatial neighborhood, then determining an initial risk probability based on the confidence level of the visual recognition results and a predefined spatial weight; if the risk type entities with a coexistence relationship continue to exist or deteriorate within the continuous time window, then increasing the initial risk probability to obtain a fused risk probability; and determining the corresponding comprehensive risk level based on the multiple fused risk probabilities in the risk event sequence.

[0012] In one embodiment of the present invention, a fault diagnosis report for the underground utility tunnel is generated based on multiple comprehensive risk levels, and an early warning message is generated based on the fault diagnosis report. This includes: extracting key attribute information of each risk event in the risk event sequence; filling the key attribute information into a preset structure template to form multiple structured risk entries; sorting the multiple structured risk entries based on the multiple comprehensive risk levels to generate a fault diagnosis report; and generating the early warning message if the comprehensive risk level of any structured risk entry reaches or exceeds a preset risk level threshold.

[0013] Secondly, the present invention also provides a visual recognition-based underground utility tunnel fault early warning system, the system comprising: The acquisition module is used to acquire real-time monitoring video streams of underground utility tunnels; The data processing module is used to preprocess the real-time monitoring video stream to obtain the target image sequence; The visual recognition module is used to input the target image sequence into a pre-built multi-task visual recognition model to obtain multiple visual recognition results, wherein the multi-task visual recognition model is used to perform water leakage area recognition, structural crack recognition, key equipment status detection, and moving object tracking and detection. The risk determination module is used to perform three-dimensional spatiotemporal alignment on multiple visual recognition results, and call a preset risk rule library to perform fusion analysis on the spatiotemporally aligned multiple visual recognition results to determine multiple comprehensive risk levels. The preset risk rule library includes association judgment rules for different visual recognition results in a preset spatial domain and within a continuous time window. The early warning module is used to generate a fault diagnosis report for the underground utility tunnel based on multiple comprehensive risk levels, and to generate early warning information based on the fault diagnosis report.

[0014] Thirdly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, the computer program being used to cause a computer to execute the visual recognition-based underground utility tunnel fault early warning method as described in any of the preceding claims.

[0015] The beneficial effects of this invention are as follows: This invention proposes a method, system, and storage medium for early warning of underground utility tunnel faults based on visual recognition. The method includes: acquiring real-time monitoring video streams of the underground utility tunnel; preprocessing the real-time monitoring video streams to obtain a target image sequence; inputting the target image sequence into a multi-task visual recognition model to obtain multiple visual recognition results, including water leakage, structural cracks, equipment status, and moving objects; performing three-dimensional spatiotemporal alignment and fusion analysis on the multiple visual recognition results to determine multiple comprehensive risk levels; and generating a fault report and outputting early warning information based on the multiple comprehensive risk levels. The fault early warning method provided by this invention simultaneously outputs the recognition results of water leakage, cracks, equipment status, and moving objects through a single inference by a multi-task visual recognition model, resulting in fast computation speed and strong comprehensive judgment ability. By combining spatiotemporal alignment and risk rule base fusion analysis, a single recognition result is transformed into a comprehensive risk assessment of the underground utility tunnel to quickly generate a fault report and achieve fault early warning. Without the need for additional sensors such as fiber optic gratings throughout the process, the system complexity and hardware cost are reduced, while improving the early warning accuracy and comprehensive perception capability of the utility tunnel monitoring. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0017] In the attached diagram: Figure 1 This is a flowchart illustrating the visual recognition-based early warning method for underground utility tunnel faults provided in this embodiment of the invention. Figure 2 This is a block diagram of a visual recognition-based underground utility tunnel fault early warning system provided in an embodiment of the present invention. Figure 3 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application. Detailed Implementation

[0018] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0019] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. The drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0020] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0021] The technical solution of this invention is mainly applied to the safety monitoring scenario of urban underground utility tunnels. Underground utility tunnels (hereinafter referred to as underground tunnels) are public tunnels that centrally house municipal pipelines such as electricity, communications, broadcasting, water supply, drainage, heating, and gas. Their internal environment has extremely unique characteristics. Specifically, tunnels are usually buried deep underground, with narrow and relatively enclosed spaces, and are constantly exposed to high humidity, low light, and poor ventilation. Under these complex conditions, the main structure and internal facilities of the tunnel face multi-dimensional safety threats: on the one hand, groundwater pressure and soil settlement can easily cause structural cracks in the concrete sidewalls or joints of the tunnel, leading to water leakage; on the other hand, critical equipment such as high-voltage cables, distribution cabinets, and fire extinguishers laid inside the tunnel may become damaged or missing due to dampness or aging; furthermore, as a lifeline for the city, the tunnel must also be strictly protected against unauthorized personnel or foreign objects illegally entering.

[0022] To address the aforementioned multi-dimensional security risks, traditional monitoring methods primarily rely on regular manual inspections. However, manual inspections are not only inefficient and costly in terms of manpower, but also limited by the frequency of inspections, making it difficult to achieve real-time, all-weather, and comprehensive security coverage. With the evolution of computer vision and artificial intelligence technologies, vision-based automated inspection solutions are gradually entering this field.

[0023] The first conventional technical approach is to develop a single visual recognition model for a specific risk. For example, an improved semantic segmentation network might be used to identify water leaks, or an object detection network might be used to identify personnel intrusion. The fundamental flaw of this approach lies in "functional silos" and "computing power disasters." Risks within utility tunnels are often concurrent. To achieve comprehensive monitoring, multiple independent deep learning models must be loaded and run simultaneously on edge computing devices. Since these models each have their own independent feature extraction network (Backbone), when processing the same frame of high-definition video stream, a large number of redundant low-level convolutional operations are repeatedly performed, causing the edge device's GPU memory to overflow rapidly and inference latency to increase dramatically, making it impossible to meet the needs of real-time early warning.

[0024] The second conventional technical approach is a multi-sensor fusion solution, which involves deploying fiber optic strain sensors on a large scale within the utility tunnel to monitor cracks, temperature and humidity sensors to monitor water seepage, and infrared or lidar to monitor intrusion. The challenge with this approach lies in its extremely high system complexity and exorbitant retrofitting costs. Multi-source heterogeneous sensors not only require laying massive amounts of communication and power cables but also face extremely difficult clock synchronization and spatial coordinate calibration problems. Especially in many existing, older utility tunnels, re-excavating and re-laying cables to add such complex sensor networks is almost impossible to implement.

[0025] Based on a deep understanding of the aforementioned technical shortcomings such as "single functionality, system redundancy, and deployment difficulties," this application proposes a technical solution for visual recognition of underground utility tunnels. Its core lies in constructing a multi-task visual recognition model capable of simultaneously performing tasks such as identifying leakage areas, structural cracks, detecting the status of key equipment, and tracking moving objects. This eliminates the redundant architecture of traditional solutions that deploy independent models for single risks or stack multiple heterogeneous sensors. Furthermore, this application introduces a three-dimensional spatiotemporal alignment mechanism and a pre-defined risk rule base, mapping the results of multi-task visual recognition to a sequence of risk events with temporal logic in three-dimensional physical space, and performing fusion analysis to determine the comprehensive risk level. Finally, a fault diagnosis report for the underground utility tunnel is generated based on the comprehensive risk level, and early warning information is generated based on the fault diagnosis report when needed. Without adding additional physical sensors, relying solely on the existing video surveillance resources of the utility tunnel, structured comprehensive perception and accurate early warning of complex environmental risks can be achieved, revealing the fault status of the underground utility tunnel.

[0026] To make the description of this application clearer and more accurate, and to avoid ambiguity, the following provides a centralized definition and explanation of some key custom terms and professional concepts involved in the specific embodiments of this application.

[0027] Region of Structural Interest (RoSI) refers to specific physical structural areas with a high probability of occurrence, pre-marked manually or filtered by algorithms in the 3D BIM (Building Information Modeling) design drawings or physical space of underground utility tunnels. Examples include structural expansion joints, arch shoulders prone to stress concentration, the area around sump pits, and pre-installed mounting bases for standard equipment. These areas have clearly defined 3D spatial coordinate boundaries.

[0028] Homography mapping, in computer vision, refers to the projection transformation that maps points on one plane to another. In this invention, it specifically refers to establishing a mathematical transformation relationship between a specific structural plane (such as a sidewall plane) in the three-dimensional physical space of the pipe gallery and the plane of a fixed camera's two-dimensional image sensor.

[0029] F1 score: A statistical metric used to measure the accuracy of binary or multi-class classification models. It considers both precision and recall of the classification model, and can be viewed as a harmonic average of precision and recall, with a maximum value of 1 and a minimum value of 0. In this application, it is used to objectively quantify the recognition performance of each task branch on the validation set.

[0030] To solve the above problems, such as Figure 1 As shown, this application provides a method for early warning of faults in underground utility tunnels based on visual recognition. This method includes at least steps S110 to S150: Step S110: Obtain the real-time monitoring video stream of the underground utility tunnel.

[0031] Specifically, in the actual operating environment of underground utility tunnels, fixed network high-definition cameras are typically deployed at regular intervals (e.g., 20 to 50 meters) along the top or side walls of the tunnel. Real-time monitoring video streams (usually in H.264 or H.265 encoding format) output by these cameras are synchronously acquired via industrial Ethernet or fiber optic ring networks.

[0032] The acquired real-time monitoring video streams often contain a large amount of background information unrelated to risk identification (such as deep corridor views and normal pipeline surfaces), and are highly susceptible to contamination from poor lighting (such as light attenuation and local shadows) and complex noise (such as water vapor flares and electromagnetic interference stripes). Directly inputting this unprocessed "dirty data" into a pre-defined multi-task visual recognition model would not only consume significant computing power but also severely interfere with the model's extraction of features such as minute cracks or water stains. Therefore, preprocessing of the acquired real-time monitoring video streams is necessary to output standardized, high signal-to-noise ratio images, laying a solid data foundation for subsequent fault reasoning.

[0033] S120. Preprocess the real-time monitoring video stream to obtain the target image sequence.

[0034] In one embodiment, preprocessing the real-time monitoring video stream to obtain a target image sequence includes: decoding and framing the real-time monitoring video stream to obtain an initial image sequence; and performing key region extraction, illumination correction, and noise filtering on each frame of the initial image sequence to obtain the target image sequence.

[0035] For example, preprocessing the real-time monitoring video stream to obtain the target image sequence includes: parallel decoding of each real-time monitoring video stream using a hardware decoder to restore it to an uncompressed raw image data stream; extracting each raw image data stream frame by frame at a preset fixed sampling frequency to obtain the initial image of each video stream; summarizing the initial images extracted from all cameras at the same timestamp and establishing an index based on the camera number and timestamp to generate a structured initial image sequence; and performing key region cropping, illumination normalization, and noise filtering on each frame of the initial image sequence to obtain the target image sequence.

[0036] In one embodiment, key region extraction, illumination correction, and noise filtering are performed on each frame of the initial image sequence to obtain the target image sequence. This includes: extracting a region of interest (ROI) corresponding to a preset key region from each frame of the image based on camera calibration parameters, and adjusting the ROI to a uniform size; selecting an illumination reference region within the ROI, and adaptively adjusting the ROI based on the pixel features of the illumination reference region to obtain a corrected ROI; and performing a hybrid filtering process on the corrected ROI to obtain the target image sequence.

[0037] For example, multiple fixed Regions of Structural Interest (RoSIs) are defined on the design drawings of each standard cross-section type (e.g., single-compartment, double-compartment). These RoSIs include, but are not limited to: the left and right side walls (prone to cracking and leakage), the top joints and arched shoulder areas (prone to leakage and deformation), the area near the bottom drainage ditch (prone to water accumulation and foreign objects), and the pre-set installation areas for standard equipment (e.g., fire extinguisher boxes, electrical distribution cabinets). Using the calibration intrinsic parameters (e.g., focal length, principal point) and extrinsic parameters (e.g., position and orientation relative to the reference axis of the utility tunnel) of each camera during deployment, a homography mapping matrix is ​​established from the two-dimensional image coordinates of the camera to the three-dimensional spatial coordinates of the underground utility tunnel. Based on the homography mapping matrix, the three-dimensional spatial coordinates of the RoSIs (i.e., key regions) defined in the three-dimensional spatial coordinate system are dynamically mapped to the two-dimensional image coordinate system of the corresponding camera, thereby obtaining the Region of Interest (ROI). The region of interest (ROI) in each original frame of the target image sequence is cropped, and all cropped ROIs are then reduced or enlarged to a uniform standard size (e.g., 256×256 pixels) using bilinear interpolation. In this embodiment, the acquisition of key structural ROIs for the utility tunnel ensures that, regardless of the monitoring perspective, the cropped area is always aligned with the critical parts of the tunnel's physical structure, rather than simply the image center or random locations. This embodiment also standardizes image dimensions, fundamentally eliminating a large number of background pixels irrelevant to structural risks (such as empty spaces in the middle of the tunnel), focusing on risk-prone areas, and effectively improving the computational efficiency and feature extraction targeting of subsequent models.

[0038] Within each region of interest (ROI) in an image, a surface sub-region with uniform material, stable texture, and in a dry and normal state is selected as the illumination reference region (e.g., a smooth concrete surface). The average pixel color of the illumination reference region within each ROI is calculated. Based on the average pixel color, a target white value point under standard illumination is set. The pixel value within each ROI is adjusted based on the target white value point to achieve adaptive adjustment of pixel values, thus obtaining a corrected ROI. This dynamically compensates for uneven color and intensity of illumination in different areas of the same pipe gallery and at different times due to light attenuation, distance attenuation, or local shadows. This makes the image data more stable in color and brightness, reducing the interference of illumination changes on texture recognition (e.g., cracks) and color recognition (e.g., water seepage).

[0039] Underground utility tunnel environments exhibit unique image noise, such as localized highlight flares caused by water vapor or condensation, speckled noise from dust adhesion, and periodic electromagnetic interference stripes that may be caused by electrical equipment. This study detects high-saturation, high-brightness regions in the color space. For suspected flare areas within the region of interest (ROI) of the corrected image, a guided-filter-based edge-preserving smoothing algorithm is used to process the ROI. This process suppresses localized highlights while maintaining the sharpness of true edges (including crack edges), preventing over-smoothing.

[0040] Subsequently, the region of interest in the corrected image is converted into a three-primary-color space, and an adaptive median filter is applied to the region of interest. The filter window size is dynamically adjusted according to the local pixel intensity variance. A larger filter window is used in flat areas (such as walls) to effectively remove spots and coarse stripes, while the filter window is automatically reduced in high-texture or edge areas to avoid loss of detail. Finally, a slight Gaussian low-pass filter is applied to remove residual high-frequency random noise, resulting in the target image. Multiple target images constitute the target image sequence.

[0041] S130. Input the target image sequence into the pre-built multi-task visual recognition model to obtain multiple visual recognition results. The multi-task visual recognition model is used to perform water leakage area recognition, structural crack recognition, key equipment status detection, and moving object tracking and detection.

[0042] For example, the target image sequence is input into a pre-constructed multi-task visual recognition model. The multi-task visual recognition model is a deep learning network based on a shared backbone network and four parallel task branches. The four parallel task branches are used to simultaneously execute the following recognition tasks. Specifically, the first task branch performs pixel-level semantic segmentation of the water leakage area, the second task branch performs pixel-level semantic segmentation and linear feature extraction of structural cracks, the third task branch performs target detection and classification of the critical equipment status, and the fourth task branch performs target detection and tracking of moving objects, which can be people or foreign objects.

[0043] In one embodiment, the multi-task visual recognition model includes: a shared backbone network for extracting multi-scale shared feature maps from the target image sequence; a first task branch for decoding and segmenting the multi-scale shared feature maps to output a leakage segmentation result; a second task branch for extracting linear features from the multi-scale shared feature maps to output a crack detection result; a third task branch for detecting key equipment anchor points from the multi-scale shared feature maps to output an equipment status result; and a fourth task branch for tracking moving object anchor points from the multi-scale shared feature maps to output a moving object trajectory result; wherein the multiple visual recognition results include the leakage segmentation result, the crack detection result, the equipment status result, and the moving object trajectory result.

[0044] For example, the multi-task visual recognition model includes a shared backbone network, a first task branch, a second task branch, a third task branch, and a fourth task branch. The shared backbone network employs a lightweight convolutional neural network architecture, which consists of multiple cascaded depthwise separable convolutional modules, each with residual connections. These depthwise separable convolutional modules can be used to perform depthwise convolution and pointwise convolution. The shared backbone network is used to extract multi-scale shared feature maps from the target image sequence. These multi-scale shared feature maps are feature maps from multiple different levels.

[0045] Specifically, the first task branch is used to segment the water leakage area from the multi-scale shared feature map. It adopts an encoder-decoder structure. The encoder shares parameters with the second task branch, and the third and fourth task branches also share parameters. The decoder performs progressive upsampling through transposed convolutional layers and outputs a pixel-level segmentation mask of the water leakage area through a 1×1 convolutional layer, which is the leakage segmentation result.

[0046] Specifically, the second task branch is the structural crack and feature extraction branch, which adopts the encoder-decoder structure corresponding to the first task branch, and connects a linear feature extraction module to the output of the decoder. The linear feature extraction module extracts the linear skeleton feature map of the crack through differentiable morphological thinning operation, thus obtaining the crack detection result.

[0047] The generation process of the linear skeleton feature map is based on an iterative thinning algorithm, as shown in the following equation:

[0048] in, For the first The linear skeleton feature map after the next iteration, i.e., the result after one refinement; For the first The linear skeleton feature map of the next iteration is a binary image (matrix) representing the crack skeleton obtained in the current iteration; For the first The structural elements in each direction are preset binary templates, with a total of 8 different directions, used to depict local pixel patterns; This represents the morphological hit-or-miss transform, used to detect specific patterns in an image that match structural elements; This indicates element-wise multiplication; This represents logical NOT, which inverts the value of each pixel in a binary image (0 becomes 1, 1 becomes 0). Indicates to from arrive Perform a union operation on the 8 results.

[0049] Specifically, the third task branch is the key equipment detection branch, which adopts an anchor-based target detection network. The target detection network includes a region proposal module and a classification regression head. The region proposal module forms candidate boxes on a multi-scale shared feature map. The classification regression head outputs the bounding box coordinates and state classification probability vector of the key equipment based on the candidate boxes, i.e., the equipment state result.

[0050] Specifically, the fourth task branch is the moving object tracking and detection branch. It adopts the target detection network corresponding to the third task branch and integrates an online real-time tracking algorithm. The target detection network is used to lock the anchor point of the moving object. The state of the moving object is predicted by Kalman filtering and the Hungarian algorithm is used for inter-frame detection and correlation. The bounding box coordinates, tracking identifier and motion trajectory of the moving object are output, thus obtaining the trajectory result of the moving object.

[0051] It should be emphasized that the leakage segmentation results include the leakage segmentation confidence level, the crack detection results include the crack confidence level, the equipment status results include the equipment status confidence level, and the moving object trajectory results include the moving object trajectory confidence level.

[0052] In one embodiment, the construction steps of the multi-task visual recognition model include: acquiring a labeled training sample set, the labeled training sample set including leakage labeled samples, crack labeled samples, equipment status labeled samples, and moving object labeled samples; using the labeled training samples to perform multi-task joint training on the shared backbone network and each task branch to obtain the prediction results of each task branch; determining the loss value between the prediction results of each task branch and the corresponding labeled ground truth value based on a loss function; adjusting the parameters of the shared backbone network and each task branch using the loss value until the loss value converges to obtain the multi-task visual recognition model.

[0053] For example, a labeled training sample set is obtained, which includes leakage labeled samples, crack labeled samples, equipment status labeled samples, and moving object labeled samples. The leakage labeled samples contain ground truth values ​​for the leakage area; the crack labeled samples contain ground truth values ​​for structural cracks; the equipment status labeled samples contain ground truth values ​​for the status of key equipment; and the moving object labeled samples contain ground truth values ​​for moving objects. Using the labeled training sample set, multi-task joint training is performed on the shared backbone network, the first task branch, the second task branch, the third task branch, and the fourth task branch, achieving end-to-end training. The training uses the Adam optimizer (Adaptive Moment Estimation), and the learning rate decays exponentially with each training epoch. The update formula is shown below:

[0054] in, For the first The learning rate of the round; The initial learning rate; The attenuation rate; This refers to the total number of training rounds.

[0055] During training, the following results are obtained: leakage segmentation prediction from the first task branch, crack prediction from the second task branch, equipment status prediction from the third task branch, and movement trajectory prediction from the fourth task branch.

[0056] The total loss of a multi-task visual recognition model is a weighted sum of the loss values ​​of each task branch. The expression for the loss function is shown below:

[0057] in, This represents the total loss value. This represents the binary cross-entropy loss value for the first branch task; This represents the binary cross-entropy loss value for the second task branch; The mean squared error loss value is extracted from the linear skeleton. The sum of the cross-entropy classification loss and the smoothed L1 localization loss for the third task branch; The sum of the cross-entropy classification loss, smoothed L1 localization loss, and tracking association loss for the fourth task branch; , , and These are dynamically adjusted weighting coefficients.

[0058] It should be noted that the smoothed L1 localization loss value is the standard loss value corresponding to the anchor-based target detection network in the third and fourth task branches.

[0059] The weight coefficients are adjusted every M training epochs based on the F1 score of each task branch on the labeled training sample set. , , and M is an integer greater than or equal to 2. The adjustment formula is shown below:

[0060] in, For the first The F1 score of each task branch on the labeled training sample set; and Let i and j be the indices of the corresponding tasks, where i and j are positive integers, 1 ≤ i ≤ 4, and 1 ≤ j ≤ 4.

[0061] The loss function is used to calculate the loss value between each prediction result and the labeled ground truth value in the labeled training sample set. Based on the loss value, the parameters of the shared backbone network, the first task branch, the second task branch, the third task branch and the fourth task branch are adjusted until the loss value meets the preset loss threshold, and the trained multi-task visual recognition model is obtained.

[0062] S140. Perform three-dimensional spatiotemporal alignment on multiple visual recognition results, and call the preset risk rule base to perform fusion analysis on the spatiotemporally aligned multiple visual recognition results to determine multiple comprehensive risk levels. The preset risk rule base includes association judgment rules for different visual recognition results in a preset spatial domain and within a continuous time window.

[0063] In one implementation, two-dimensional image coordinates and timestamps of multiple visual recognition results in the target image sequence are extracted; based on the calibration parameters, the two-dimensional image coordinates of each visual recognition result are mapped to the three-dimensional spatial coordinates of the underground utility tunnel, and the three-dimensional spatial coordinates are synchronized in time according to the timestamps. A single visual recognition result and its corresponding three-dimensional spatial coordinates and timestamp constitute a risk event, and multiple risk events form a risk event sequence in the order of the timestamps; the preset risk rule base is called to perform coexistence analysis in a preset spatial domain and change analysis within a continuous time period on the risk event sequence to obtain multiple comprehensive risk levels.

[0064] For example, the initial image sequence includes multiple initial images. A target image containing visual recognition results is acquired from the target image sequence. Two-dimensional image coordinates of the visual recognition results in the corresponding initial images are extracted from the target images, and the timestamp of the visual recognition results is determined based on the capture time of the initial images. Based on the pre-calibrated intrinsic and extrinsic parameters of the cameras, the two-dimensional image coordinates of the visual recognition results output by each task branch are uniformly mapped to three-dimensional spatial coordinates based on the design axis of the underground utility tunnel, resulting in three-dimensional spatial coordinates for leakage segmentation, crack detection, equipment status, and moving object trajectory. Time synchronization is performed based on the timestamps parsed from the real-time monitoring video stream and synchronized via a network time protocol. Recognition results from all cameras within the same time period are categorized and aligned according to time windows. A single visual recognition result and its corresponding three-dimensional spatial coordinates and timestamp constitute a risk event. Multiple risk events are sorted according to their timestamps to obtain a risk event sequence. A preset risk rule base is used to perform coexistence analysis in a preset spatial domain and change analysis within a continuous time period on the risk event sequence, resulting in multiple comprehensive risk levels.

[0065] In one implementation, a preset risk rule base is invoked to perform coexistence analysis and change analysis within a preset spatial domain on a risk event sequence to obtain multiple comprehensive risk levels. This includes: determining risk type entities in the risk event sequence based on multiple visual recognition results; if multiple risk type entities have a coexistence relationship within the preset spatial neighborhood, determining an initial risk probability based on the confidence level of the visual recognition results and a predefined spatial weight; if the risk type entities with a coexistence relationship continue to exist or deteriorate within a continuous time window, increasing the initial risk probability to obtain a fused risk probability; and determining the corresponding comprehensive risk level based on the multiple fused risk probabilities in the risk event sequence.

[0066] For example, risk type entities in a risk event sequence are identified through multiple visual recognition results. These risk type entities include leakage entities corresponding to leakage segmentation results, crack entities corresponding to crack detection results, key equipment entities corresponding to equipment status results, and moving object entities corresponding to moving object trajectory results. In a three-dimensional spatial coordinate system, it is determined whether two or more risk type entities exist within a preset spatial neighborhood. If two or more risk type entities exist within the preset spatial neighborhood, i.e., these risk type entities coexist, the initial risk probability is calculated based on the confidence level of the visual recognition results and predefined spatial weights. For example, in a three-dimensional spatial coordinate system, taking a certain spatial point as a reference point, if leakage entities and crack entities exist within a spatial range of less than 1 meter, a coexistence relationship is determined. The predefined spatial weights are then applied: crack confidence level × 0.6 + leakage segmentation confidence level × 0.4 = initial risk probability. 0.6 is the spatial weight for structural cracks occurring in the underground utility tunnel, and 0.4 is the spatial weight for areas with water leakage in the underground utility tunnel. Within a continuous time window, a temporal evolution analysis is performed on risk type entities with coexisting relationships. If a phenomenon of continued existence or deterioration occurs, the initial risk probability is increased to obtain the fused risk probability. Based on multiple fused risk probabilities in the risk event sequence and the preset mapping relationship, multiple corresponding comprehensive risk levels are determined. The mapping relationship is the mapping relationship between fused risk probabilities and comprehensive risk levels.

[0067] S150. Generate fault diagnosis reports for underground utility tunnels based on multiple comprehensive risk levels, and generate early warning information based on the fault diagnosis reports.

[0068] In one implementation, key attribute information of each risk event in the risk event sequence is extracted, and the key attribute information is filled into a preset structure template to form multiple structured risk entries; the multiple structured risk entries are sorted based on multiple comprehensive risk levels to generate a fault diagnosis report; if the comprehensive risk level of any structured risk entry reaches or exceeds a preset risk level threshold, the warning information is generated.

[0069] For example, key attribute information such as event number, time range, risk type, three-dimensional spatial coordinates, comprehensive risk level, confidence level of visual recognition results, and associated original image frame index field are extracted from each risk event in the risk event sequence. The key attribute information of each risk event is then filled into a preset structure template to obtain multiple structured risk entries. Based on the comprehensive risk level of each structured risk entry, these entries are sorted from high to low to generate a fault diagnosis report. Preset risk level thresholds corresponding to different comprehensive risk levels are set. The comprehensive risk level of each structured risk entry in the fault diagnosis report is compared with the preset risk level threshold. If the comprehensive risk level reaches or exceeds the corresponding preset risk level threshold, an early warning message is generated. The early warning message includes at least a trigger event marker, risk location, comprehensive risk level, and suggested handling measures. Based on the comprehensive risk level of the early warning message, and according to a preset communication protocol and distribution strategy, the early warning message is simultaneously sent to the corresponding monitoring terminal and the responsible person's terminal.

[0070] like Figure 2 As shown, the present invention also provides a visual recognition-based underground utility tunnel fault early warning system, which includes: Acquisition module 210 is used to acquire real-time monitoring video streams of underground utility tunnels; Data processing module 220 is used to preprocess the real-time monitoring video stream to obtain a target image sequence; The visual recognition module 230 is used to input the target image sequence into a pre-built multi-task visual recognition model to obtain multiple visual recognition results, wherein the multi-task visual recognition model is used to perform water leakage area recognition, structural crack recognition, key equipment status detection and moving object tracking detection. The risk determination module 240 is used to perform three-dimensional spatiotemporal alignment on multiple visual recognition results, and call a preset risk rule library to perform fusion analysis on the spatiotemporally aligned multiple visual recognition results to determine multiple comprehensive risk levels. The preset risk rule library includes association judgment rules for different visual recognition results in a preset spatial domain and within a continuous time window. The early warning module 250 is used to generate a fault diagnosis report of the underground utility tunnel based on multiple comprehensive risk levels, and to generate early warning information based on the fault diagnosis report.

[0071] It should be noted that the underground utility tunnel fault early warning system based on visual recognition provided in the above embodiments and the underground utility tunnel fault early warning method based on visual recognition provided in the above embodiments belong to the same concept. The specific way of performing each step has been described in detail in the system embodiments, and will not be repeated here.

[0072] This invention proposes a method and system for early warning of underground utility tunnel faults based on visual recognition. The method includes: acquiring real-time monitoring video streams of the underground utility tunnel; preprocessing the video streams to obtain a target image sequence; inputting the target image sequence into a multi-task visual recognition model to obtain multiple visual recognition results, including water leakage, structural cracks, equipment status, and moving objects; performing three-dimensional spatiotemporal alignment and fusion analysis on the multiple visual recognition results to determine the comprehensive risk level; and generating a fault report and outputting early warning information based on the comprehensive risk level. The fault early warning method provided by this invention simultaneously outputs the recognition results of water leakage, cracks, equipment status, and moving objects through a single inference by the multi-task visual recognition model, resulting in fast computation speed and strong comprehensive judgment. By combining spatiotemporal alignment with a risk rule base for fusion analysis, a single recognition result is transformed into a comprehensive risk assessment of the underground utility tunnel, enabling rapid generation of fault reports and timely fault early warning. Without requiring additional sensors such as fiber optic gratings throughout the process, the system complexity and hardware cost are reduced, while improving the early warning accuracy and comprehensive perception capability of the utility tunnel monitoring.

[0073] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3 As shown, the electronic device may include: a processor 310, a communications interface 320, a memory 330, and a communications bus 340, wherein the processor 310, the communications interface 320, and the memory 330 communicate with each other through the communications bus 340. The processor 310 can call logic instructions in the memory 330 to execute a visual recognition-based underground utility tunnel fault early warning method. This method includes: acquiring a real-time monitoring video stream of the underground utility tunnel; preprocessing the real-time monitoring video stream to obtain a target image sequence; inputting the target image sequence into a pre-built multi-task visual recognition model to obtain multiple visual recognition results, wherein the multi-task visual recognition model is used to perform leakage area identification, structural crack identification, key equipment status detection, and moving object tracking detection; performing three-dimensional spatiotemporal alignment of the multiple visual recognition results, and calling a preset risk rule base to perform fusion analysis on the spatiotemporally aligned multiple visual recognition results to determine multiple comprehensive risk levels, wherein the preset risk rule base includes association judgment rules for different visual recognition results within a preset spatial domain and continuous time window; generating a fault diagnosis report for the underground utility tunnel based on the multiple comprehensive risk levels, and generating early warning information based on the fault diagnosis report.

[0074] Furthermore, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0075] This application also provides a non-volatile computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the visual recognition-based underground utility tunnel fault early warning method provided by the above methods. The method includes: acquiring a real-time monitoring video stream of the underground utility tunnel; preprocessing the real-time monitoring video stream to obtain a target image sequence; inputting the target image sequence into a pre-constructed multi-task visual recognition model to obtain multiple visual recognition results, wherein the multi-task visual recognition model is used to perform leakage area identification, structural crack identification, key equipment status detection, and moving object tracking detection; performing three-dimensional spatiotemporal alignment of the multiple visual recognition results, and calling a preset risk rule base to perform fusion analysis on the spatiotemporally aligned multiple visual recognition results to determine multiple comprehensive risk levels, wherein the preset risk rule base includes association judgment rules for different visual recognition results in a preset spatial domain and within a continuous time window; generating a fault diagnosis report of the underground utility tunnel based on the multiple comprehensive risk levels, and generating early warning information based on the fault diagnosis report.

[0076] The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0077] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0078] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A visual recognition-based underground pipe gallery fault early warning method, characterized in that, include: Obtain real-time monitoring video streams of underground utility tunnels; The real-time monitoring video stream is preprocessed to obtain the target image sequence; The target image sequence is input into a pre-built multi-task visual recognition model to obtain multiple visual recognition results, wherein the multi-task visual recognition model is used to perform water leakage area recognition, structural crack recognition, key equipment status detection, and moving object tracking and detection. The multiple visual recognition results are aligned in three dimensions in a spatiotemporal manner, and a preset risk rule base is called to perform a fusion analysis on the spatiotemporally aligned multiple visual recognition results to determine multiple comprehensive risk levels. The preset risk rule base includes association judgment rules for different visual recognition results in a preset spatial domain and within a continuous time window. A fault diagnosis report for the underground utility tunnel is generated based on multiple comprehensive risk levels, and early warning information is generated based on the fault diagnosis report. 2.The visual recognition based underground pipe gallery fault early warning method according to claim 1, characterized in that, The real-time monitoring video stream is preprocessed to obtain a target image sequence, including: The real-time monitoring video stream is decoded and segmented into frames to obtain an initial image sequence; The target image sequence is obtained by extracting key regions, correcting illumination, and filtering noise from each frame of the initial image sequence. 3.The visual recognition based underground pipe gallery fault early warning method according to claim 2, characterized in that, The target image sequence is obtained by performing key region extraction, illumination correction, and noise filtering on each frame of the initial image sequence, including: Based on the camera's calibration parameters, the region of interest corresponding to the preset key region is extracted from each frame of the image, and the region of interest is adjusted to a uniform size. A lighting reference region is selected within the region of interest in the image, and adaptive pixel adjustment is performed on the region of interest based on the pixel features of the lighting reference region to obtain the corrected region of interest. The region of interest in the corrected image is subjected to a hybrid filtering process to obtain the target image sequence. 4.The visual recognition based underground pipe gallery fault early warning method according to claim 1, characterized in that, The multi-task visual recognition model includes: A shared backbone network is used to extract multi-scale shared feature maps from the target image sequence; The first task branch is used to decode and segment the multi-scale shared feature map and output the leakage segmentation result. The second task branch is used to extract linear features from the multi-scale shared feature map and output crack detection results. The third task branch is used to perform key device anchor point detection on the multi-scale shared feature map and output the device status result. The fourth task branch is used to perform moving object anchor point tracking on the multi-scale shared feature map and output the moving object trajectory result. The visual recognition results include the leakage segmentation results, the crack detection results, the equipment status results, and the moving object trajectory results. 5.The visual recognition-based underground pipe gallery fault early warning method according to claim 4, characterized in that, The construction steps of the multi-task visual recognition model include: Obtain a set of labeled training samples, which includes leakage labeled samples, crack labeled samples, equipment status labeled samples, and moving object labeled samples. The shared backbone network and each task branch are jointly trained using the labeled training sample set to obtain the prediction results of each task branch. The loss value between the prediction result of each task branch and the corresponding labeled ground truth value is determined based on the loss function. The parameters of the shared backbone network and each task branch are adjusted using the loss value until the loss value converges, thus obtaining the multi-task visual recognition model. 6.The visual recognition based underground pipe gallery fault early warning method according to claim 3, characterized in that, The multiple visual recognition results are spatiotemporally aligned in three dimensions, and a preset risk rule base is invoked to perform a fusion analysis on the spatiotemporally aligned visual recognition results to determine multiple comprehensive risk levels, including: Extract the two-dimensional image coordinates and timestamps of multiple visual recognition results from the target image sequence; Based on the calibration parameters, the two-dimensional image coordinates of each visual recognition result are mapped to the three-dimensional spatial coordinates of the underground utility tunnel, and the three-dimensional spatial coordinates are synchronized in time according to the timestamp. A single visual recognition result and its corresponding three-dimensional spatial coordinates and timestamp constitute a risk event, and multiple risk events form a risk event sequence according to the timestamp order. The preset risk rule base is invoked to perform coexistence analysis in a preset spatial domain and change analysis within a continuous time period on the risk event sequence, thereby obtaining multiple comprehensive risk levels.

7. The method for early warning of underground utility tunnel faults based on visual recognition according to claim 6, characterized in that, The preset risk rule base is invoked to perform coexistence analysis in a preset spatial domain and change analysis within a continuous time window on the risk event sequence, resulting in multiple comprehensive risk levels, including: Based on multiple visual recognition results, the risk type entities in the risk event sequence are determined; If multiple entities of the aforementioned risk type coexist within the preset spatial neighborhood, the initial risk probability is determined based on the confidence level of the visual recognition result and the predefined spatial weight. If entities of the risk type that have a coexistence relationship continue to exist or deteriorate within a continuous time window, the initial risk probability is increased to obtain the fused risk probability. The corresponding comprehensive risk level is determined based on the multiple fusion risk probabilities in the risk event sequence.

8. The method for early warning of underground utility tunnel faults based on visual recognition according to claim 7, characterized in that, A fault diagnosis report for the underground utility tunnel is generated based on multiple comprehensive risk levels, and early warning information is generated based on the fault diagnosis report, including: Extract key attribute information of each risk event in the risk event sequence, and fill the key attribute information into a preset structure template to form multiple structured risk entries; Based on the multiple comprehensive risk levels, the multiple structured risk items are sorted to generate a fault diagnosis report; If the overall risk level of any of the structured risk items reaches or exceeds the preset risk level threshold, the warning information is generated.

9. A visual recognition-based early warning system for underground utility tunnel faults, characterized in that, The system includes: The acquisition module is used to acquire real-time monitoring video streams of underground utility tunnels; The data processing module is used to preprocess the real-time monitoring video stream to obtain the target image sequence; The visual recognition module is used to input the target image sequence into a pre-built multi-task visual recognition model to obtain multiple visual recognition results, wherein the multi-task visual recognition model is used to perform water leakage area recognition, structural crack recognition, key equipment status detection, and moving object tracking and detection. The risk determination module is used to perform three-dimensional spatiotemporal alignment on multiple visual recognition results, and call a preset risk rule library to perform fusion analysis on the spatiotemporally aligned multiple visual recognition results to determine multiple comprehensive risk levels. The preset risk rule library includes association judgment rules for different visual recognition results in a preset spatial domain and within a continuous time window. The early warning module is used to generate a fault diagnosis report for the underground utility tunnel based on multiple comprehensive risk levels, and to generate early warning information based on the fault diagnosis report.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the visual recognition-based underground utility tunnel fault early warning method according to any one of claims 1 to 8.

Images