Unmanned inspection multi-mode AI identification and alarm linkage monitoring system for heat supply pipe network

Abstract

The invention relates to the technical field of intelligent operation and maintenance of a heat supply pipe network, and discloses a heat supply pipe network unmanned inspection multi-mode AI identification and alarm linkage monitoring system, which comprises a three-dimensional reference construction module for establishing a pipe network model and a global coordinate system; the unmanned inspection collection module collects multi-modal data and constructs a voxel map; the space calculation analysis module determines a candidate abnormal curved surface through view cone back projection and Boolean operation, and utilizes ray stepping detection shielding and thermal and geometric conformal mapping analysis verification; the air-ground cooperative mutual verification module dispatches a ground robot for re-checking based on the confidence coefficient, and calculates the heterogeneous viewpoint volume overlapping degree; and the alarm linkage control module locks a target curved surface, calculates an abnormal center, drives digital twin display and generates a work order. According to the invention, through three-dimensional view cone mapping and air-ground cooperation mutual identification, environment interference false alarms are effectively eliminated, and high-precision positioning and closed-loop linkage monitoring of heat supply pipe network abnormity are realized.

Description

Technical Field

[0001] This invention relates to the field of intelligent operation and maintenance technology for heating pipeline networks, specifically a multimodal AI recognition and alarm linkage monitoring system for unmanned inspection of heating pipeline networks. Background Technology

[0002] Heating networks are the core infrastructure of urban centralized heating systems, responsible for transporting heat energy generated by heat sources to various heat user terminals through long-distance pipelines. Artificial intelligence (AI) recognition technology is a technique that uses algorithms such as deep learning to extract and analyze features from image data in order to automatically identify targets or abnormal states.

[0003] Currently, the inspection of heating pipelines is gradually introducing drones equipped with infrared and visible light dual-light pods. Multimodal AI recognition technology is used to process the collected two-dimensional images. By analyzing the morphological characteristics of the temperature difference distribution in the infrared thermal image, automated monitoring of pipeline leaks, insulation layer damage, and abnormalities in the surrounding environment can be achieved.

[0004] However, existing technologies are mostly limited to feature analysis of two-dimensional image planes, lacking the fusion of three-dimensional spatial depth information. This makes it difficult for the system to effectively distinguish between the actual thermal anomalies of the pipeline network itself and environmental interference caused by the heating of foreground obstructions such as trees and fences, easily leading to false alarms in complex scenarios. At the same time, the single-view inspection mode lacks a logical verification mechanism based on physical laws, and the identified abnormal data is difficult to directly convert into accurate three-dimensional physical coordinates to drive subsequent maintenance processes. This results in a gap in the information flow from fault discovery to alarm handling, making it difficult to meet the intelligent monitoring requirements of high-precision, closed-loop management.

[0005] Therefore, this invention proposes a multimodal AI recognition and alarm linkage monitoring system for unmanned inspection of heating pipelines to address the shortcomings of existing technologies. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a multimodal AI recognition and alarm linkage monitoring system for unmanned inspection of heating pipelines. This system solves the technical problems of existing technologies in heating pipeline inspection, such as the lack of three-dimensional spatial logic leading to susceptibility to environmental interference and false alarms, the lack of heterogeneous perspective collaborative verification mechanisms, and the difficulty in providing accurate three-dimensional positioning and alarm closed-loop linkage.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a multimodal AI recognition and alarm linkage monitoring system for unmanned inspection of heating pipelines, comprising:

[0008] The 3D benchmark construction module is used to import geographic information data of heating pipeline network and convert it into a 3D triangular mesh model to establish a global world coordinate system as a unified reference system.

[0009] The unmanned inspection and data acquisition module is used to control the sensor group on the heterogeneous mobile inspection terminal to acquire multimodal image data, calculate the six-degree-of-freedom pose matrix of the heterogeneous mobile inspection terminal relative to the global world coordinate system, and construct an environmental voxel map using depth perception sensor data.

[0010] The spatial computation and analysis module is used to extract two-dimensional anomaly features from the multimodal image data, back-project the two-dimensional anomaly features with the camera intrinsic parameter matrix to generate an aerial three-dimensional view frustum spatial envelope, determine candidate anomaly surface patches by calculating the Boolean intersection of the aerial three-dimensional view frustum spatial envelope and the three-dimensional triangular mesh model, detect whether the ray path of the aerial three-dimensional view frustum spatial envelope in the environmental voxel map is truncated by obstacle voxels, and further use the infrared thermal imaging data to perform thermal and geometric surface conformal mapping analysis on the candidate anomaly surface patches to generate anomaly verification results. The anomaly verification results include anomaly classification labels, physical consistency confidence scores, and path occlusion status markers.

[0011] The air-ground collaborative verification module is used to dispatch a ground inspection robot to collect verification data when the abnormal verification result indicates that there is occlusion or insufficient confidence, and to calculate the volume overlap between the air three-dimensional view frustum spatial envelope and the ground three-dimensional view frustum spatial envelope to generate a final confirmation result.

[0012] The alarm linkage control module is used to respond to the abnormal verification result or the final confirmation result to lock the target surface, calculate the coordinates of the abnormal center, drive the digital twin terminal to position and display, and output the maintenance work order.

[0013] Preferably, the three-dimensional benchmark construction module converts the geographic information data of the heating pipeline network into the three-dimensional triangular mesh model, specifically including:

[0014] The geographic information data of the heating pipeline network is analyzed to obtain the three-dimensional coordinate point sequence of the pipe segment centerline and the nominal diameter value of the pipe segment. For any pipe segment, the axial unit vector from the starting point to the ending point is calculated.

[0015] Select an auxiliary vector that is not parallel to the axial unit vector, and construct a local orthogonal basis vector perpendicular to the axial unit vector through the vector cross product operation;

[0016] The pipe segment radius is determined based on the nominal diameter value. Combined with the discrete segment angles in the circumferential direction, the three-dimensional spatial coordinates of the discrete vertices on the pipe segment surface are calculated using the local orthogonal basis vectors and the three-dimensional coordinate point sequence of the pipe segment centerline.

[0017] Adjacent discrete vertices are connected in a preset index order to generate triangular facets, which are then combined to form the three-dimensional triangular mesh model. A bounding volume hierarchical spatial index tree is then constructed based on the triangular facets.

[0018] Preferably, the multimodal image data in the unmanned inspection and acquisition module includes infrared thermal imaging data acquired by the infrared thermal imaging sensor in the sensor group, and visible light image data acquired by the visible light image sensor in the sensor group.

[0019] Preferably, the heterogeneous mobile inspection terminal in the unmanned inspection and data collection module includes an aerial drone and a ground inspection robot;

[0020] The unmanned inspection and acquisition module fuses the coordinate data output by the positioning sensor with the angular velocity and acceleration data output by the inertial measurement unit through an extended Kalman filter algorithm or a graph optimization algorithm, and calculates the aerial six-degree-of-freedom pose matrix of the aerial UAV or the ground six-degree-of-freedom pose matrix of the ground inspection robot in real time.

[0021] The environmental voxel map is obtained by the unmanned inspection and acquisition module using a spatial rasterization algorithm to divide the continuous three-dimensional space containing the three-dimensional point cloud set output by the depth perception sensor into multiple voxel units. Among the multiple voxel units, the voxel unit containing at least one of the three-dimensional point cloud data is marked as an obstacle occupancy state.

[0022] Preferably, the spatial calculation and analysis module extracts two-dimensional anomaly features from the multimodal image data, and combines the camera intrinsic parameter matrix to back-project the two-dimensional anomaly features to generate an aerial three-dimensional view frustum spatial envelope, specifically including:

[0023] The infrared thermal imaging data and the visible light image data are processed using a deep learning semantic segmentation model to extract the pixel mask set of the two-dimensional anomaly features and obtain the outer edge contour pixels of the pixel mask set.

[0024] Call the pre-stored camera intrinsic parameter matrix corresponding to the sensor group, and extract the rotation matrix and translation vector from the six-degree-of-freedom pose matrix in the air, and use the translation vector as the camera optical center;

[0025] Based on the geometric inverse transformation principle of the pinhole imaging model, the three-dimensional ray unit direction vector of the outer edge contour pixel point relative to the optical center of the camera in the global world coordinate system is calculated using the inverse matrix of the camera intrinsic parameter matrix and the rotation matrix.

[0026] Construct a polygonal pyramidal geometric region with the camera's optical center as its vertex and its side edges composed of all the three-dimensional ray unit direction vectors, as the spatial envelope of the three-dimensional aerial view cone.

[0027] Preferably, the scheduling of the ground inspection robot by the air-ground collaborative verification module specifically includes:

[0028] When the path occlusion status marker indicates that there is occlusion or the physical consistency confidence value is lower than the preset confidence threshold, the optimal observation line vector is planned based on the normal vector of the candidate abnormal surface patch and the environmental voxel map.

[0029] Traverse the ground passable area voxels in the environmental voxel map and calculate the observation evaluation function of the candidate location. The observation evaluation function represents the cosine similarity between the observation line of sight and the opposite direction of the surface normal vector.

[0030] The optimal observation position is selected as the position that maximizes the observation evaluation function and satisfies the distance constraint, visibility constraint, and accessibility constraint. The optimal observation position is then sent to the ground inspection robot to collect the verification data.

[0031] Preferably, the calculation of the volume overlap between the air-to-ground collaborative verification module and the ground-based three-dimensional view frustum spatial envelope specifically includes:

[0032] Based on the ground six-degree-of-freedom pose matrix of the ground inspection robot and the intrinsic parameter matrix of the ground camera recorded during ground verification, a ground three-dimensional view frustum spatial envelope is constructed from the ground perspective.

[0033] A local voxel mesh is constructed within the three-dimensional bounding box of the candidate anomalous surface patch, and each voxel unit in the local voxel mesh is traversed.

[0034] The volume of voxel units that are simultaneously located inside the spatial envelope of the three-dimensional view frustum in the air and the spatial envelope of the three-dimensional view frustum on the ground, and that include the surface of the three-dimensional triangular mesh model, is counted as the intersection volume.

[0035] The volume of voxel units located inside the spatial envelope of the three-dimensional view frustum in the air or the spatial envelope of the three-dimensional view frustum on the ground, and including the surface of the three-dimensional triangular mesh model, is counted and used as the union volume.

[0036] The ratio of the intersection volume to the union volume is calculated to obtain the heterogeneous viewpoint 3D intersection-union ratio coefficient, and the heterogeneous viewpoint 3D intersection-union ratio coefficient is used as the volume overlap.

[0037] Preferably, the alarm linkage control module responding to the abnormal verification result or the final confirmation result to lock the target surface specifically includes:

[0038] When the path occlusion status flag indicates no occlusion and the physical consistency confidence value is higher than the preset confidence threshold, the candidate abnormal surface patch is directly locked as the target surface.

[0039] When the volume overlap is greater than the preset verification threshold, in response to the final confirmation result, the subset of triangular facets located inside the intersection volume in the three-dimensional triangular mesh model is determined to be a refined anomaly region, and the refined anomaly region is locked as the target surface.

[0040] Preferably, the calculation of the anomaly center coordinates by the alarm linkage control module specifically includes:

[0041] Obtain all vertices of the set of discrete triangular facets contained in the locked target surface;

[0042] The arithmetic mean of the global coordinate vectors of all vertices is calculated to obtain the anomaly center coordinate vector, and the anomaly center coordinate vector is used as the anomaly center coordinates.

[0043] Preferably, the alarm linkage control module drives the digital twin terminal to locate, display, and output maintenance work orders, specifically including:

[0044] Obtain the unit normal vector of the discrete triangular facet set contained in the target surface, and calculate the average value of the unit normal vectors to obtain the average unit normal vector;

[0045] The viewpoint transformation parameters of the virtual camera are constructed using the anomaly center coordinate vector and the average unit normal vector, so as to drive the digital twin terminal to focus on and display the target surface;

[0046] The maintenance work order is generated by packaging evidence data based on the trigger source of the locked target surface.

[0047] This invention provides a multimodal AI recognition and alarm linkage monitoring system for unmanned inspection of heating pipe networks. It has the following beneficial effects:

[0048] 1. This invention utilizes a spatial computing module to construct an aerial 3D view frustum envelope during unmanned inspections, accurately mapping 2D anomaly features extracted by multimodal AI recognition onto the physical pipeline surface. Combined with environmental voxel mapping and a ray-stepping algorithm to detect the line-of-sight path, it effectively eliminates false alarms caused by foreground occlusion. This mechanism solves the problem of traditional technologies being unable to distinguish between pipeline leaks and environmental interference, significantly improving detection accuracy and 3D positioning accuracy in complex heating pipeline scenarios.

[0049] 2. This invention establishes an air-ground collaborative verification mechanism based on the volume overlap of heterogeneous viewpoints. For areas with insufficient confidence in multimodal AI identification, a ground robot is automatically dispatched for verification. The physical laws of thermal characteristics are verified using surface geodesic gradients, and the consistency of observations is quantified by calculating the intersection-union ratio of the viewpoint cone envelope. This collaborative unmanned inspection mode achieves logical verification of anomalies such as fires and violations, significantly reducing the false alarm rate and constructing a highly reliable intelligent diagnostic system.

[0050] 3. This invention constructs an alarm linkage monitoring and response system based on three-dimensional geometric features. It calculates the anomaly center based on the geometric properties of the target surface and automatically generates virtual viewpoint parameters to drive the digital twin terminal to focus on the fault area. The system automatically packages multimodal AI-recognized evidence to generate standardized maintenance work orders based on single-channel rapid response or dual-channel collaborative confirmation of the source, achieving a closed-loop process from image acquisition to maintenance and repair, effectively improving the efficiency of pipeline emergency repairs. Attached Figure Description

[0051] Figure 1 This is an architecture diagram of the multimodal AI recognition and alarm linkage monitoring system for unmanned inspection of heating pipelines according to the present invention;

[0052] Figure 2 This is a flowchart of the two-dimensional to three-dimensional anomaly localization analysis based on frustum backprojection of the present invention;

[0053] Figure 3 This is a flowchart illustrating the logical process of air-ground collaborative verification and three-dimensional volume overlap confirmation in this invention.

[0054] Among them, 10 is the 3D benchmark construction module; 20 is the unmanned inspection and data acquisition module; 30 is the spatial calculation and analysis module; 40 is the air-ground collaborative verification module; and 50 is the alarm linkage control module. Detailed Implementation

[0055] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] See attached document Figure 1 This invention provides a multimodal AI recognition and alarm linkage monitoring system for unmanned inspection of heating pipelines. The system includes a three-dimensional benchmark construction module 10, an unmanned inspection acquisition module 20, a spatial calculation and analysis module 30, an air-ground collaborative verification module 40, and an alarm linkage control module 50.

[0057] The 3D benchmark construction module 10 imports geographic information data of the heating pipeline network, converts it into a 3D triangular mesh model, and establishes a global world coordinate system. This global world coordinate system serves as a unified reference system for all sensor data, spatial location calculations, and alarm positioning within the system.

[0058] The unmanned inspection and data acquisition module 20 is configured on a heterogeneous mobile inspection terminal, which specifically includes an aerial drone and a ground inspection robot. The unmanned inspection and data acquisition module 20 is used to acquire environmental data and equipment status data during the inspection process. The module 20 controls the infrared thermal imaging sensor, visible light image sensor, and depth sensing sensor mounted on the heterogeneous mobile inspection terminal to simultaneously acquire data, outputting multimodal image data including infrared thermal imaging data and visible light image data, and using the data acquired by the depth sensing sensor to construct an environmental voxel map in real time.

[0059] Meanwhile, the unmanned inspection and acquisition module 20 pre-stores the camera intrinsic parameter matrices of the infrared thermal imaging sensor and the visible light image sensor, and calculates the pose of the heterogeneous mobile inspection terminal relative to the global world coordinate system in real time at the acquisition time; when the heterogeneous mobile inspection terminal is an aerial UAV, it calculates and outputs an aerial six-degree-of-freedom pose matrix; when the heterogeneous mobile inspection terminal is a ground inspection robot, it calculates and outputs a ground six-degree-of-freedom pose matrix.

[0060] The spatial computation and analysis module 30 is connected to the 3D benchmark construction module 10 and the unmanned inspection and acquisition module 20. The spatial computation and analysis module 30 receives a 3D triangular mesh model, infrared thermal imaging data, visible light image data, camera intrinsic parameter matrix, aerial six-degree-of-freedom pose matrix, and environmental voxel map. The spatial computation and analysis module 30 uses image processing algorithms to process the infrared thermal imaging data and visible light image data to extract 2D anomaly features. Combining the camera intrinsic parameter matrix, the spatial computation and analysis module 30 back-projects the 2D anomaly features to generate an aerial 3D view frustum spatial envelope, and calculates the Boolean intersection of this aerial 3D view frustum spatial envelope and the 3D triangular mesh model provided by the 3D benchmark construction module 10 to determine candidate anomaly surface patches, and detects whether the ray path of the aerial 3D view frustum spatial envelope in the environmental voxel map is truncated by obstacle voxels. The spatial computation and analysis module 30 further uses infrared thermal imaging data to perform thermal and geometric surface conformal mapping analysis on the candidate anomaly surface patches. Based on the above analysis, the spatial computation and analysis module 30 generates anomaly verification results. The anomaly verification results include anomaly classification labels, physical consistency confidence scores, and path occlusion status markers.

[0061] The air-to-ground collaborative verification module 40 is connected to the spatial calculation and analysis module 30. The air-to-ground collaborative verification module 40 receives and parses the anomaly verification results. When the path occlusion status marker in the anomaly verification result indicates occlusion, or the physical consistency confidence value is lower than a preset confidence threshold, the air-to-ground collaborative verification module 40 plans the optimal observation line-of-sight vector for the ground inspection robot based on the normal vectors of the candidate anomaly surface patches and the environmental voxel map. The air-to-ground collaborative verification module 40 schedules the ground inspection robot to collect verification data and calculates the volume overlap between the airborne 3D view frustum spatial envelope and the ground 3D view frustum spatial envelope in 3D space, generating the final confirmation result.

[0062] The alarm linkage control module 50 is connected to both the spatial calculation and analysis module 30 and the air-to-ground collaborative verification module 40. When the physical consistency confidence value in the final confirmation result is higher than the preset confidence threshold and there is no obstruction, the alarm linkage control module 50 directly responds to the output of the spatial calculation and analysis module 30; when air-to-ground collaborative verification is initiated, the alarm linkage control module 50 responds to the final confirmation result output by the air-to-ground collaborative verification module 40. The alarm linkage control module 50 extracts the three-dimensional coordinates of the anomaly center in the global world coordinate system, drives the positioning and display of the pipeline network model on the digital twin management terminal, and outputs a maintenance work order.

[0063] See attached document Figure 1 The 3D benchmark construction module 10 is used to construct and store a digital 3D spatial benchmark for the heating pipeline network area. This module 10 establishes a unified mathematical measurement environment by parsing the input geospatial data, providing geometrically dependent objects for subsequent frustum calculations.

[0064] The 3D benchmark construction module 10 establishes a global world coordinate system, denoted as . The global coordinate system is defined using the ENU (East-Northeast) Cartesian rectangular coordinate system. The system selects a fixed geographic anchor point within the heating network area as the coordinate origin. ,definition The axis points due east. The axis points due north. The axis is perpendicular to the tangent plane of the reference ellipsoid and points upwards. The 3D reference construction module 10 receives geodetic coordinates (including longitude) in WGS-84 format output from the positioning sensor. ,latitude and ellipsoid height Transform it into a global world coordinate system using the Gaussian projection formula. Three-dimensional Cartesian coordinates This global coordinate system serves as a unified reference for all sensor data, spatial location calculations, and alarm positioning within the system, ensuring the numerical consistency of infrared thermal imaging data, visible light image data, and the geometric location of the pipeline network in physical space.

[0065] The 3D benchmark construction module 10 imports and parses the geographic information data of the heating pipeline network. This geographic information data is vector data containing the network topology and attributes, specifically including the 3D coordinate sequence of pipe segment centerlines, the nominal diameter of the pipe segments, the spatial location of valve nodes, and the direction of medium flow. Since the original geographic information data of the heating pipeline network is a linear skeleton without volume, the 3D benchmark construction module 10, based on the pipe segment centerlines and nominal diameters, transforms it into a 3D triangular mesh model using a geometric expansion algorithm, denoted as... .

[0066] 3D triangular mesh model From the set of vertices and triangular facet set Composition. For any straight pipe segment in the geographic information data of the heating pipeline network, let the three-dimensional coordinates of its starting point be... The three-dimensional coordinates of the endpoint are The pipe section radius is The three-dimensional benchmark construction module 10 first calculates the axial unit vector of the pipe segment from the starting point to the ending point. :

[0067] ;

[0068] To construct the cylindrical geometry on the pipe segment surface, the 3D datum construction module 10 constructs a unit vector perpendicular to the axis. Local orthogonal basis vectors and The 3D benchmark construction module 10 selects unit vectors that are not parallel to the axis. auxiliary vector For example, when Not parallel to the global world coordinate system of When the axis is used, take the auxiliary vector. If they are parallel, then select Axis. Then, local orthogonal basis vectors are constructed using the vector cross product operation:

[0069] ;

[0070] Using local orthogonal basis vectors, the three-dimensional datum construction module 10 determines the pipe segment radius based on the nominal diameter value. The vertex coordinates are then calculated using discrete segmented angles along the circumference. Let the number of discrete segments along the circumference be... For the first Each segment ( ), and its corresponding circumferential angle Using local orthogonal basis vectors and the sequence of three-dimensional coordinate points along the pipe segment centerline, the three-dimensional spatial coordinates of discrete vertices on the pipe segment surface are calculated. Generate using the following formula:

[0071] ;

[0072] in, These correspond to the starting and ending faces of the pipe segment, respectively. The 3D benchmark construction module 10 stores the generated discrete vertices into a vertex set. Connect adjacent discrete vertices according to a preset index order to generate triangular facets (for example, connect them in a counter-clockwise order to form two triangles to cover a rectangular area), and store the generated triangular facets in a triangular facet set. , set of vertices With triangular facet set Combined to form a three-dimensional triangular mesh model .

[0073] The 3D benchmark building module 10 further calculates the 3D triangular mesh model. The unit normal vector of each triangular facet in the middle And construct a spatial index tree of the bounding volume hierarchy (BVH).

[0074] See attached document Figure 1 The unmanned inspection and data acquisition module 20 serves as the system's sensing interface connecting the physical world and the digital space. It is configured on the heterogeneous mobile inspection terminal that performs the actual inspection tasks, responsible for data acquisition and preprocessing. The heterogeneous mobile inspection terminal specifically includes aerial drones and ground inspection robots. The aerial drones utilize a multi-rotor flight platform to collect macroscopic data on the distribution and top status of the heating pipe network from an aerial perspective; the ground inspection robots use wheeled or tracked mobile chassis to collect local details and bottom status of the heating pipe network from a side or upward perspective. Based on the aforementioned hardware, the unmanned inspection and data acquisition module 20 acquires environmental data and equipment status data during the inspection process. Environmental data refers to three-dimensional data reflecting the distribution of obstacles and spatial geometry around the pipe network; equipment status data refers to image data reflecting the temperature distribution and appearance of the pipe network itself.

[0075] The unmanned inspection and data acquisition module 20 controls multiple sensor groups mounted on a heterogeneous mobile inspection terminal to collect data synchronously. The sensor groups not only include infrared thermal imaging sensors, visible light image sensors, and depth sensing sensors, but also physically integrate positioning sensors and inertial measurement units (IMUs).

[0076] Infrared thermal imaging sensors are long-wave infrared uncooled focal plane detectors used to acquire radiative thermal images reflecting the temperature distribution on the surface of an object; visible light image sensors are CMOS or CCD image sensors used to acquire RGB images reflecting the texture and color of an object; depth sensing sensors are multi-line lidar (LiDAR) or RGB-D depth cameras used to acquire point cloud data reflecting spatial geometric location.

[0077] The positioning sensor is a GNSS receiver that supports RTK carrier phase differential, used to acquire high-precision geographic location information; the inertial measurement unit is a microelectromechanical system containing a three-axis gyroscope and a three-axis accelerometer, used to acquire angular velocity and acceleration data of the heterogeneous mobile inspection terminal.

[0078] The unmanned inspection and data acquisition module 20 ensures that all sensors capture data simultaneously through hardware trigger signals or a unified timestamp protocol, and outputs multimodal image data, including infrared thermal imaging data and visible light image data. Specifically, the infrared thermal imaging data is a two-dimensional temperature matrix recording the radiation temperature values ​​of object surfaces within the field of view. The two-dimensional temperature matrix Each element value in the data corresponds to a physical temperature value (Celsius or Kelvin); the visible light image data is specifically a three-channel RGB pixel matrix that records the intensity of light reflected from objects within the field of view. It is used to characterize the texture and color features of an object.

[0079] Meanwhile, the unmanned inspection and data acquisition module 20 pre-stores the camera intrinsic parameter matrices of the infrared thermal imaging sensor and the visible light image sensor. This camera intrinsic parameter matrix is ​​an inherent parameter matrix calibrated based on the pinhole imaging model, denoted as... For any given sensor, its intrinsic parameter matrix... Includes focal length ( ) and principal point coordinates ( The format is as follows:

[0080] ;

[0081] The camera intrinsic parameter matrix Used to establish the mapping relationship between the pixel coordinate system of a two-dimensional image and the three-dimensional camera coordinate system of the sensor.

[0082] The unmanned inspection data acquisition module 20 utilizes measurement data from positioning sensors (such as RTK-GNSS) and inertial measurement units to calculate in real time the heterogeneous mobile inspection terminal's position relative to the global world coordinate system at the time of data acquisition. The six-degree-of-freedom pose matrix in the air (i.e., the coordinate system established in the three-dimensional reference construction module 10). Or ground six-degree-of-freedom pose matrix .

[0083] The specific calculation process is as follows: The unmanned inspection and data acquisition module 20 first receives the WGS-84 geodetic coordinates output by the positioning sensor, and constructs the coordinate origin defined in the three-dimensional reference module 10 based on the coordinate origin. The low-frequency positioning coordinates are converted to local Cartesian coordinates using Gauss-Kruger projection. Simultaneously, the unmanned inspection and acquisition module 20 receives the high-frequency angular velocity and acceleration output from the inertial measurement unit (IMU). The unmanned inspection and acquisition module 20 employs an extended Kalman filter (EKF) algorithm or a graph optimization algorithm to loosely or tightly couple the low-frequency positioning coordinates with the high-frequency inertial data, thereby eliminating IMU integral drift and filling GNSS signal gaps.

[0084] When the heterogeneous mobile inspection terminal is an aerial drone, the matrix output after fusion calculation is labeled as an aerial six-degree-of-freedom pose matrix. When the heterogeneous mobile inspection terminal is a ground inspection robot, the matrix output after fusion calculation is labeled as a ground six-degree-of-freedom pose matrix. .

[0085] The pose matrices mentioned above all describe rigid body space transformations. Homogeneous transformation matrices are collectively referred to as :

[0086] ;

[0087] in, for The rotation matrix represents the position of the heterogeneous mobile inspection terminal relative to the global world coordinate system at the time of data acquisition. The orientation and direction; for Translation vector (here) The transpose operator (converts row vectors to column vectors) indicates that the heterogeneous mobile inspection terminal is located in the global world coordinate system at the time of data acquisition. Spatial coordinates in the equation.

[0088] The unmanned inspection and data acquisition module 20 uses data collected by a depth sensing sensor to construct an environmental voxel map in real time. The raw data output by the depth sensing sensor is a dense set of 3D point clouds. The unmanned inspection and data acquisition module 20 employs a spatial rasterization algorithm to divide a continuous three-dimensional space into fixed-size cubic units (voxels). For point cloud datasets... any point in Calculate the voxel grid index where it is located:

[0089] ;

[0090] in, The origin of the inspection area is the boundary point; The resolution is the side length of the voxel. The unmanned inspection and acquisition module 20 marks voxel units containing at least one point cloud data as "obstacle occupied". The set of all voxels marked as occupied constitutes the environmental voxel map. This environmental voxel map describes the spatial distribution of non-pipeline obstacles such as trees and buildings around the heating pipeline network in real time, and is output as environmental data to subsequent modules for occlusion detection.

[0091] See attached document Figure 1 and Figure 2 The spatial calculation and analysis module 30, as the core processing unit of the system, is connected to the 3D benchmark construction module 10 and the unmanned inspection and acquisition module 20. The spatial calculation and analysis module 30 receives the 3D triangular mesh model. Infrared thermal imaging data, visible light image data, camera intrinsic parameter matrix Aerial six-degree-of-freedom pose matrix And environmental voxel maps.

[0092] The spatial computing and analysis module 30 utilizes image processing algorithms to process infrared thermal imaging data and visible light image data to extract two-dimensional anomaly features. Specifically, the image processing algorithm integrates a deep learning semantic segmentation model. This deep learning semantic segmentation model is built upon an encoder-decoder network architecture such as a fully convolutional neural network (FCN) or U-Net, and its parameters have been pre-optimized and trained on a multimodal sample dataset of heating pipe networks containing scenarios such as leaks, fires, flooding, and violations.

[0093] The deep learning semantic segmentation model is configured to use a multimodal image matrix formed by concatenating infrared thermal imaging data and visible light image data channels as the input tensor. High-dimensional features are extracted through convolution and pooling operations in the encoder structure within the deep learning semantic segmentation model, and the feature map size is restored through deconvolution and upsampling operations in the decoder structure. The final output is a pixel-level classification probability map with the same size as the input image. The spatial computation and analysis module 30 performs thresholding and connected component analysis on the output pixel-level classification probability map to obtain two-dimensional anomaly features.

[0094] Specifically, the spatial computing and analysis module 30 uses a deep learning semantic segmentation model to process infrared thermal imaging data and visible light image data, and extracts a set of pixel masks for two-dimensional anomaly features. And obtain the outer edge contour pixels of the pixel mask set.

[0095] The spatial calculation and analysis module 30 performs threshold segmentation and connected component analysis on the output pixel-level classification probability map to obtain two-dimensional anomaly features.

[0096] Two-dimensional anomaly features are specifically represented as a set of binary pixel masks on the image plane. and their corresponding anomaly classification labels Anomaly classification tags The definition is as follows:

[0097] When there are pixel areas in infrared thermal imaging data where the local temperature is significantly higher than the background and exhibits an irregular diffusion pattern, they are marked as "pipeline leaks".

[0098] When infrared data exhibits extremely high temperature saturation and visible light data When accompanied by smoke texture features, it is marked as an "environmental fire";

[0099] When visible light data identifies a high-reflectivity water surface or liquid accumulation in the trench area, it is marked as "regional flooding".

[0100] When a target outline of a person or engineering vehicle is identified in the visible light data, it is marked as "personnel violation".

[0101] The spatial computation and analysis module 30, in conjunction with the camera intrinsic parameter matrix, back-projects two-dimensional anomaly features to generate a three-dimensional aerial view frustum spatial envelope. Back projection generates the spatial envelope of a three-dimensional view frustum in the air. The specific process is as follows:

[0102] The spatial computation and analysis module 30 combines the camera intrinsic parameter matrix to back-project two-dimensional anomaly features to generate a three-dimensional aerial view frustum spatial envelope. The specific process is as follows:

[0103] First, the spatial calculation and analysis module 30 calls the pre-stored camera intrinsic parameter matrix corresponding to the sensor group. And from the six-degree-of-freedom pose matrix in the air Extracting the rotation matrix With translation vector Translate vector As the optical core of the camera .

[0104] Next, let the set of extracted outer edge contour pixels be... ,in The spatial computation and analysis module 30 is based on the geometric inverse transformation principle of the pinhole imaging model, utilizing the inverse matrix of the camera intrinsic parameter matrix. and rotation matrix Calculate the outer edge contour pixels in the global world coordinate system relative to the camera optical center 3D ray unit direction vector :

[0105] ;

[0106] Finally, the spatial calculation and analysis module 30 is constructed based on the camera optical center. For vertices, lateral edges, and all three-dimensional ray unit direction vectors The resulting multi-faceted pyramidal geometric region serves as the spatial envelope of the three-dimensional view cone in the air. This pyramidal geometric region mathematically describes the set of all possible physical locations of two-dimensional anomalous pixels in three-dimensional space.

[0107] Spatial calculation and analysis module 30 calculates the spatial envelope of the three-dimensional view frustum in the air. The Boolean intersection of the surface with the 3D triangular mesh model provided by the 3D benchmark construction module 10 is used to determine candidate anomalous surface patches. The Boolean intersection operation refers to filtering out surface patches whose spatial location is completely contained within the spatial envelope of a 3D view frustum in 3D Euclidean space. Internal 3D triangular mesh model A subset of triangular facets. This subset of triangular facets constitutes the candidate anomalous surface patches. This represents the projection mapping region of an abnormal region in a two-dimensional image onto the surface of a three-dimensional physical network.

[0108] For anomaly classification tags For the scenario of "personnel violation", the spatial computing and analysis module 30 further calculates the spatial envelope of the three-dimensional view frustum in the air. Inner target centroid and candidate anomalous surface patch The minimum Euclidean distance between them. If this minimum Euclidean distance is less than the preset safety restricted area threshold (i.e., the minimum allowable physical distance extending outward from the outer wall of the pipeline network as set according to the heating pipeline network safety operation management specifications, for example, 2.0 meters), then a violation is determined to have occurred.

[0109] Spatial computation and analysis module 30 detects the spatial envelope of a three-dimensional aerial view frustum. The detection process checks whether the ray path in the environmental voxel map is truncated by obstacle voxels. The ray-marching algorithm is used. First, the spatial calculation and analysis module 30 analyzes the candidate anomalous surface patches. Select any three-dimensional surface point as the target point, denoted as . Subsequently, regarding the optical center of the camera... Point to the target point The detection ray is processed by the spatial calculation and analysis module 30, which traverses the voxel elements traversed by the ray path. If a voxel with the status marked "obstacle occupied" exists on the path, and the center of that voxel is within a certain distance from the camera's optical center... The distance is less than the target point To the camera optical center If the distance is less than or equal to the specified distance, the ray path is determined to be truncated, and a path occlusion status marker is generated. A value of 1 (or a Boolean value of True, indicating occlusion) indicates that the anomalous features in the image originate from foreground occlusions (such as trees or fences) rather than the network itself.

[0110] The spatial computation and analysis module 30 further utilizes infrared thermal imaging data to perform thermal and geometric conformal mapping analysis on candidate anomalous surface patches, generating anomaly verification results. Thermal and geometric conformal mapping analysis refers to using projective texture mapping technology to read the temperature matrix from the infrared thermal imaging data. Map it to a 3D candidate anomaly surface patch At each vertex, a temperature scalar field is formed on the surface of the three-dimensional manifold. The spatial calculation and analysis module 30 calculates the temperature scalar field. Geodesic gradient on the tangent plane of a curved surface :

[0111] ;

[0112] in, is the unit normal vector of the corresponding point on the candidate anomalous surface patch; This is the Euclidean gradient. The spatial calculation and analysis module 30 uses the geodesic gradient... Calculate the physical consistency confidence value based on the directional distribution characteristics. If the geodesic gradient If the direction exhibits isotropic decay characteristics centered on a certain point outwards (consistent with the heat conduction equation), then the calculated physical consistency confidence score is... It is a high value; if the geodesic gradient If the distribution of the data is disordered, discontinuous, or unrelated to the geometric features of the surface, then the calculated confidence score of physical consistency is... The value is low. The anomaly verification result specifically includes anomaly classification labels. Physical consistency confidence score and path occlusion status markers .

[0113] See attached document Figure 1 and Figure 3 The air-to-ground collaborative verification module 40 is connected to the spatial calculation and analysis module 30. The air-to-ground collaborative verification module 40 receives and parses the candidate anomaly surface patches output by the spatial calculation and analysis module 30. And the anomaly verification results. The anomaly verification results specifically include anomaly classification labels. Physical consistency confidence score and path occlusion status markers .

[0114] The air-to-ground collaborative verification module 40 determines whether to initiate the ground verification process based on the anomaly verification results. The air-to-ground collaborative verification module 40 reads the path obstruction status markers. Confidence level of physical consistency When the path is obstructed. A value of true indicates that the line of sight from an aerial perspective is interrupted by an obstruction; or when the physical consistency confidence score is true. If the confidence level is lower than a preset confidence threshold (e.g., 0.75, which represents the minimum probability limit required for the system to recognize a single-viewpoint thermal feature as a genuine anomaly), it indicates that the thermal feature does not conform to typical heat conduction laws. When any of the above conditions are met, the air-ground collaborative verification module 40 generates a verification instruction and executes the planning of the optimal observation line-of-sight vector.

[0115] The air-ground collaborative verification module 40 plans the optimal observation line-of-sight vector for the ground inspection robot based on the normal vectors of the candidate anomaly surface patches and the environmental voxel map. The air-ground collaborative verification module 40 first calculates the candidate anomaly surface patches... The geometric center is defined as the target point. And obtain the unit normal vector at the target point. Optimal line-of-sight vector This refers to the movement of a ground inspection robot from its planned station point to the target point. The three-dimensional vector. The planning process is structured as a constrained optimization problem, aiming to find the optimal ground observation location in the environmental voxel map that satisfies specific geometric constraints. .

[0116] The air-ground collaborative verification module iterates through 40 ground-accessible voxels in the environmental voxel map and calculates each candidate location. Observation evaluation function :

[0117] ;

[0118] The evaluation function The physical meaning of is the cosine similarity between the direction of the observation line and the opposite direction of the surface normal vector. The closer the value is to 1, the closer the observation angle is to the perpendicular surface, and the clearer the image texture.

[0119] Air-Ground Collaborative Verification Module 40 Selection The optimal observation position is the position that maximizes the value of the position while simultaneously satisfying the following constraints. :

[0120] Distance constraints: ,in and These represent the minimum imaging distance and the maximum effective resolution distance of the visible light image sensor mounted on the ground robot.

[0121] Visibility constraint: Connectivity and The straight path does not pass through any voxel cells marked as "obstacle occupied" in the environment voxel map;

[0122] Reachability constraints: Candidate locations The voxel it is located in is marked as "ground" or "accessible area" on the environment voxel map.

[0123] The optimal observation position was calculated. Subsequently, the air-ground collaborative verification module 40 generates a certificate from... point to The unit vector is used as the optimal observation line-of-sight vector. .

[0124] The air-ground collaborative verification module 40 schedules a ground inspection robot to collect and verify data. The air-ground collaborative verification module 40 will determine the optimal observation location. Coordinates and optimal line-of-sight vector The data is sent to the ground inspection robot. The ground inspection robot moves to the location and adjusts the gimbal attitude, collecting infrared thermal imaging data and visible light image data for verification, and recording the ground six-degree-of-freedom pose matrix at the time of acquisition. The ground inspection robot will collect ground infrared thermal imaging data, ground visible light image data, and ground six-degree-of-freedom pose matrix. The data is transmitted back to the air-ground collaborative verification module 40 in real time, serving as the data basis for constructing the ground three-dimensional view cone spatial envelope and generating the final confirmation result.

[0125] The air-to-ground collaborative verification module 40 calculates the volumetric overlap between the airborne 3D view frustum spatial envelope and the ground-based 3D view frustum spatial envelope in 3D space, generating the final confirmation result. Specifically, the volumetric overlap is the 3D Intersection over Union (IoU) ratio of the heterogeneous viewpoints, used to quantify the consistency between airborne and ground-based observation results in 3D physical space. The calculation process is as follows:

[0126] First, the air-ground collaborative verification module 40 calls the spatial envelope of the aerial three-dimensional view frustum already generated in the spatial calculation and analysis module 30. Meanwhile, the air-ground collaborative verification module 40 adopts the same back projection method as the spatial calculation and analysis module 30, but it is based on the ground six-degree-of-freedom pose matrix recorded during ground verification. In addition to the intrinsic parameter matrix of the ground camera, a three-dimensional view frustum spatial envelope is constructed from the ground perspective, denoted as the ground three-dimensional view frustum spatial envelope. .

[0127] Secondly, the air-ground collaborative verification module 40 is used in candidate abnormal surface patches. A high-resolution local voxel mesh is constructed within the 3D bounding box. The air-ground collaborative verification module traverses each voxel cell in this local mesh 40 times. And make the following judgment:

[0128] If voxel unit At the same time located Internal and Internally, and voxel units Includes a 3D triangular mesh model On the surface, the air-ground collaborative verification module 40 performs two operations: first, it includes the volume of the surface in the intersection volume. Secondly, the three-dimensional triangular mesh model contained within the voxel. The triangular patch index is added to the refined anomaly region set. ;

[0129] If voxel unit lie in Internal or Internally, and voxel units Includes a 3D triangular mesh model The surface volume is then included in the union volume. .

[0130] Finally, the air-ground collaborative verification module 40 calculates the intersection volume. Union volume The ratio value is used to obtain the 3D intersection-union ratio coefficient of heterogeneous viewpoints ( The heterogeneous viewpoint 3D intersection-union ratio coefficient is used as the volume overlap. This coefficient is a dimensionless value within the closed interval [0,1], used to quantitatively characterize the degree of geometric overlap between the abnormal spatial region determined by the aerial UAV's line of sight and the abnormal spatial region determined by the ground inspection robot's line of sight in 3D physical space.

[0131] The logic for generating the final confirmation result is as follows: if the calculated... If the value exceeds a preset verification threshold (e.g., 0.5), it indicates that both the aerial and ground-based perspectives point to the same physical location on the pipeline, confirming the anomaly and generating a confirmation alarm; if... If the value is less than the verification threshold, it indicates that the abnormal feature is a viewpoint-related false signal (such as sunlight reflection or background perspective error), and the system will automatically filter out the abnormality.

[0132] The final confirmation result is specifically defined as a structured data packet containing multi-dimensional verification information, which specifically includes:

[0133] Verification status flag : A Boolean status bit used to indicate whether the current abnormal event has been double-confirmed, the value of which is determined by the verification threshold determination logic mentioned above;

[0134] Geometric consistency metric: i.e., the calculated value This serves as the confidence weight for subsequent assessments of the severity level of the fault;

[0135] Ground-based verification evidence data: including ground inspection robots at optimal observation positions. Ground infrared thermal imaging data and ground visible light image data were collected at the location;

[0136] Refine the anomaly region Defined as a three-dimensional triangular mesh model The middle is located in the intersection volume The set of internal triangular facets.

[0137] See attached document Figure 1 The alarm linkage control module 50 is connected to the spatial calculation and analysis module 30 and the air-ground collaborative verification module 40, respectively. The alarm linkage control module 50 is configured as the final decision-making unit of the system, used to parse the verification data from the preceding modules, execute the alarm judgment logic, and drive the synchronous response of the digital twin terminal and the physical operation and maintenance process.

[0138] The alarm linkage control module 50 receives the anomaly verification result output by the spatial calculation and analysis module 30 and the final confirmation result output by the air-ground collaborative verification module 40. The anomaly verification result includes a physical consistency confidence score. Path occlusion status marker The final confirmed results include the 3D intersection-union coefficients of heterogeneous viewpoints. Ground-based verification of evidence data and detailed analysis of anomaly areas The alarm linkage control module 50 has the following built-in dual-channel alarm triggering logic:

[0139] Channel 1 (Fast Response Channel): Alarm linkage control module 50 reads the anomaly verification result. If the path is obstructed, the status flag is displayed. The value is false (i.e., the logical state is False or the value is 0, indicating that the line of sight is unobstructed), and the physical consistency confidence score is... If the confidence level is higher than the preset confidence threshold (e.g., 0.75, which does not meet the activation conditions of the air-to-ground collaborative verification module 40), then the abnormal feature is determined to have high physical credibility. The alarm linkage control module 50 directly responds to the output of the spatial calculation and analysis module 30 and locks the candidate abnormal surface patch. As the target surface, a confirmation alarm signal is generated.

[0140] Channel Two (Collaborative Confirmation Channel): When the system initiates the air-to-ground collaborative verification process, the alarm linkage control module 50 reads the final confirmation result. If the heterogeneous viewpoint 3D intersection-over-interference ratio coefficient... If the result exceeds a preset verification threshold (e.g., 0.5), the consistency between the aerial and ground observation results in geometric space is determined to be valid. The alarm linkage control module 50 responds to the final confirmation result output by the air-ground collaborative verification module 40, and locks down the refined anomaly area. As the target surface, a confirmation alarm signal is generated.

[0141] After generating a confirmed alarm signal, the alarm linkage control module 50 extracts the anomaly center in the global world coordinate system. The three-dimensional coordinates in the target surface. (i.e., the locked target surface) or The alarm linkage control module 50 determines the precise location by calculating the geometric centroid of the discrete triangular facet set. Assume the target surface contains... There are vertices, and the global coordinate vector of each vertex is... The alarm linkage control module 50 calculates the coordinate vector of the abnormal center. And the anomaly center coordinate vector is used as the anomaly center coordinate:

[0142] ;

[0143] Meanwhile, in order to obtain the best viewing angle in the digital twin scenario, the alarm linkage control module 50 calculates the average unit normal vector of all vertices to obtain the average unit normal vector. :

[0144] ;

[0145] in, For the first The unit normal vector of each vertex.

[0146] The alarm linkage control module 50 drives the positioning and display of the pipeline network model on the digital twin management terminal. The digital twin management terminal renders a 3D scene through a virtual camera, and the alarm linkage control module 50 utilizes the anomaly center coordinate vector. With average unit normal vector The viewpoint transformation parameters of the virtual camera are constructed to drive the digital twin terminal to focus on and display the target surface. Specifically, the alarm linkage control module 50 controls the spatial position of the virtual camera. The anomaly center is set to be offset by a fixed observation distance along the direction of the average normal vector. The position, that is The virtual camera's gaze focus is set to the coordinate vector of the anomaly center. By updating the view matrix of the rendering engine, the display interface of the digital twin management terminal automatically and smoothly moves and focuses on the abnormal area on the pipeline model, and renders the corresponding target surface with a highlight color.

[0147] The alarm linkage control module 50 outputs a maintenance work order. The alarm linkage control module 50 packages evidence data to generate a maintenance work order based on the trigger source of the locked target surface (i.e., channel one or channel two). When triggered via channel one, the work order includes aerial infrared thermal imaging data and visible light image data; when triggered via channel two, the work order additionally includes ground verification evidence data. The alarm linkage control module 50 categorizes the anomalies with tags. Anomaly center coordinate vector The converted geographical latitude and longitude coordinates and the aforementioned evidence data are packaged into standardized maintenance work order files and sent to the operation and maintenance management system via a communication interface.

Claims

1. A multimodal AI recognition and alarm linkage monitoring system for unmanned inspection of heating pipeline networks, characterized in that: include: The 3D benchmark construction module is used to import geographic information data of heating pipeline network and convert it into a 3D triangular mesh model to establish a global world coordinate system as a unified reference system. The unmanned inspection and data acquisition module is used to control the sensor group on the heterogeneous mobile inspection terminal to acquire multimodal image data, calculate the six-degree-of-freedom pose matrix of the heterogeneous mobile inspection terminal relative to the global world coordinate system, and construct an environmental voxel map using depth perception sensor data. The spatial computation and analysis module is used to extract two-dimensional anomaly features from the multimodal image data, back-project the two-dimensional anomaly features with the camera intrinsic parameter matrix to generate an aerial three-dimensional view frustum spatial envelope, determine candidate anomaly surface patches by calculating the Boolean intersection of the aerial three-dimensional view frustum spatial envelope and the three-dimensional triangular mesh model, detect whether the ray path of the aerial three-dimensional view frustum spatial envelope in the environmental voxel map is truncated by obstacle voxels, and further use infrared thermal imaging data to perform thermal and geometric surface conformal mapping analysis on the candidate anomaly surface patches to generate anomaly verification results. The anomaly verification results include anomaly classification labels, physical consistency confidence scores, and path occlusion status markers. The air-ground collaborative verification module is used to dispatch a ground inspection robot to collect verification data when the abnormal verification result indicates that there is occlusion or insufficient confidence, and to calculate the volume overlap between the air three-dimensional view frustum spatial envelope and the ground three-dimensional view frustum spatial envelope to generate a final confirmation result. The alarm linkage control module is used to respond to the abnormal verification result or the final confirmation result to lock the target surface, calculate the coordinates of the abnormal center, drive the digital twin terminal to position and display, and output the maintenance work order.

2. The multimodal AI recognition and alarm linkage monitoring system for unmanned inspection of heating pipelines according to claim 1, characterized in that, The three-dimensional benchmark construction module converts the geographic information data of the heating pipeline network into the three-dimensional triangular mesh model, specifically including: The geographic information data of the heating pipeline network is analyzed to obtain the three-dimensional coordinate point sequence of the pipe segment centerline and the nominal diameter value of the pipe segment. For any pipe segment, the axial unit vector from the starting point to the ending point is calculated. Select an auxiliary vector that is not parallel to the axial unit vector, and construct a local orthogonal basis vector perpendicular to the axial unit vector through the vector cross product operation; The pipe segment radius is determined based on the nominal diameter value. Combined with the discrete segment angles in the circumferential direction, the three-dimensional spatial coordinates of the discrete vertices on the pipe segment surface are calculated using the local orthogonal basis vectors and the three-dimensional coordinate point sequence of the pipe segment centerline. Adjacent discrete vertices are connected in a preset index order to generate triangular facets, which are then combined to form the three-dimensional triangular mesh model. A bounding volume hierarchical spatial index tree is then constructed based on the triangular facets.

3. The multimodal AI recognition and alarm linkage monitoring system for unmanned inspection of heating pipelines according to claim 1, characterized in that, The multimodal image data in the unmanned inspection and acquisition module includes infrared thermal imaging data acquired by the infrared thermal imaging sensor in the sensor group, and visible light image data acquired by the visible light image sensor in the sensor group.

4. The multimodal AI recognition and alarm linkage monitoring system for unmanned inspection of heating pipelines according to claim 1, characterized in that, The heterogeneous mobile inspection terminal in the unmanned inspection and data collection module includes an aerial drone and a ground inspection robot. The unmanned inspection and acquisition module fuses the coordinate data output by the positioning sensor with the angular velocity and acceleration data output by the inertial measurement unit through an extended Kalman filter algorithm or a graph optimization algorithm, and calculates the aerial six-degree-of-freedom pose matrix of the aerial UAV or the ground six-degree-of-freedom pose matrix of the ground inspection robot in real time. The environmental voxel map is obtained by the unmanned inspection and acquisition module using a spatial rasterization algorithm to divide the continuous three-dimensional space where the three-dimensional point cloud set output by the depth perception sensor is located into multiple voxel units. Among the multiple voxel units, the voxel unit containing at least one of the three-dimensional point cloud data is marked as an obstacle occupancy state.

5. The multimodal AI recognition and alarm linkage monitoring system for unmanned inspection of heating pipelines according to claim 1, characterized in that, The spatial computation and analysis module extracts two-dimensional anomaly features from the multimodal image data, and combines these features with the camera intrinsic parameter matrix to back-project the two-dimensional anomaly features to generate an aerial three-dimensional view frustum spatial envelope. Specifically, this includes: The infrared thermal imaging data and the visible light image data are processed using a deep learning semantic segmentation model to extract the pixel mask set of the two-dimensional anomaly features and obtain the outer edge contour pixels of the pixel mask set. Call the pre-stored camera intrinsic parameter matrix corresponding to the sensor group, and extract the rotation matrix and translation vector from the six-degree-of-freedom pose matrix in the air, and use the translation vector as the camera optical center; Based on the geometric inverse transformation principle of the pinhole imaging model, the three-dimensional ray unit direction vector of the outer edge contour pixel point relative to the optical center of the camera in the global world coordinate system is calculated using the inverse matrix of the camera intrinsic parameter matrix and the rotation matrix. Construct a polygonal pyramidal geometric region with the camera's optical center as its vertex and its side edges composed of all the three-dimensional ray unit direction vectors, as the spatial envelope of the three-dimensional aerial view cone.

6. The multimodal AI recognition and alarm linkage monitoring system for unmanned inspection of heating pipelines according to claim 5, characterized in that, The air-ground collaborative verification module schedules the ground inspection robot specifically including: When the path occlusion status marker indicates that there is occlusion or the physical consistency confidence value is lower than the preset confidence threshold, the optimal observation line vector is planned based on the normal vector of the candidate abnormal surface patch and the environmental voxel map. Traverse the ground passable area voxels in the environmental voxel map and calculate the observation evaluation function of the candidate location. The observation evaluation function represents the cosine similarity between the observation line of sight and the opposite direction of the surface normal vector. The optimal observation position is selected as the position that maximizes the observation evaluation function and satisfies the distance constraint, visibility constraint, and accessibility constraint. The optimal observation position is then sent to the ground inspection robot to collect the verification data.

7. The multimodal AI recognition and alarm linkage monitoring system for unmanned inspection of heating pipelines according to claim 1, characterized in that, The air-to-ground collaborative verification module calculates the volume overlap between the airborne three-dimensional view frustum spatial envelope and the ground-based three-dimensional view frustum spatial envelope, specifically including: Based on the ground six-degree-of-freedom pose matrix of the ground inspection robot and the intrinsic parameter matrix of the ground camera recorded during ground verification, a ground three-dimensional view frustum spatial envelope is constructed from the ground perspective. A local voxel mesh is constructed within the three-dimensional bounding box of the candidate anomalous surface patch, and each voxel unit in the local voxel mesh is traversed. The volume of voxel units that are simultaneously located inside the spatial envelope of the three-dimensional view frustum in the air and the spatial envelope of the three-dimensional view frustum on the ground, and that include the surface of the three-dimensional triangular mesh model, is counted as the intersection volume. The volume of voxel units located inside the spatial envelope of the three-dimensional view frustum in the air or the spatial envelope of the three-dimensional view frustum on the ground, and including the surface of the three-dimensional triangular mesh model, is counted and used as the union volume. The ratio of the intersection volume to the union volume is calculated to obtain the heterogeneous viewpoint 3D intersection-union ratio coefficient, and the heterogeneous viewpoint 3D intersection-union ratio coefficient is used as the volume overlap.

8. The multimodal AI recognition and alarm linkage monitoring system for unmanned inspection of heating pipelines according to claim 1, characterized in that, The alarm linkage control module responds to the abnormal verification result or the final confirmation result to lock the target surface, specifically including: When the path occlusion status flag indicates no occlusion and the physical consistency confidence value is higher than the preset confidence threshold, the candidate abnormal surface patch is directly locked as the target surface. When the volume overlap is greater than the preset verification threshold, in response to the final confirmation result, the subset of triangular facets located inside the intersection volume in the three-dimensional triangular mesh model is determined to be a refined anomaly region, and the refined anomaly region is locked as the target surface.

9. The multimodal AI recognition and alarm linkage monitoring system for unmanned inspection of heating pipelines according to claim 1, characterized in that, The calculation of the anomaly center coordinates by the alarm linkage control module specifically includes: Obtain all vertices of the set of discrete triangular facets contained in the locked target surface; The arithmetic mean of the global coordinate vectors of all vertices is calculated to obtain the anomaly center coordinate vector, and the anomaly center coordinate vector is used as the anomaly center coordinates.

10. The multimodal AI recognition and alarm linkage monitoring system for unmanned inspection of heating pipelines according to claim 9, characterized in that, The alarm linkage control module drives the digital twin terminal to locate, display, and output maintenance work orders, specifically including: Obtain the unit normal vector of the discrete triangular facet set contained in the target surface, and calculate the average value of the unit normal vectors to obtain the average unit normal vector; The viewpoint transformation parameters of the virtual camera are constructed using the anomaly center coordinate vector and the average unit normal vector, so as to drive the digital twin terminal to focus on and display the target surface; The maintenance work order is generated by packaging evidence data based on the trigger source of the locked target surface.

Images