Method and system for monitoring and processing blockage condition of tunnel drainage pipeline

Abstract

The invention discloses a tunnel drainage pipeline blockage condition monitoring and processing method and system, and the method comprises the steps: collecting data, such as optical images, three-dimensional point cloud, ultrasonic reflection, gas components, infrared imaging, sound wave images, electromagnetic induction and flow velocity, through a multi-mode sensor array, carrying out the denoising and normalization preprocessing, and extracting multi-scale features; fusing and analyzing the feature data by using a self-attention mechanism and a graph neural network to generate optimized multi-modal comprehensive feature data; based on the data, a full-connection neural network discriminates the type, the number, the size, the blockage degree and the cleaning difficulty of blockages, and dynamic correction is carried out through real-time data feedback and a self-learning mechanism; and finally, cleaning scheme data is generated, and a cleaning task is executed through the mechanical arm. According to the method, the comprehensiveness of data acquisition, the depth of feature extraction, the intelligentization of feature fusion and the accuracy and the real-time performance of a discrimination model are improved, and comprehensive monitoring and efficient processing of the blockage condition of the tunnel drainage pipeline are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of tunnel drainage pipeline monitoring and processing, and in particular to a tunnel drainage pipeline blockage monitoring and processing method and system. Background Art

[0002] Monitoring and handling of blockage in tunnel drainage pipes is of great significance to the normal operation of tunnel drainage systems. Blockage in drainage pipes can lead to failure of the drainage system, causing water accumulation in the tunnel, seriously affecting traffic safety and the stability of the tunnel structure. Therefore, timely and accurate monitoring and handling of blockage in tunnel drainage pipes is the key to ensuring safe operation of tunnels. However, there are some problems with current drainage pipe monitoring, such as single data collection, inaccurate feature extraction, simple discrimination model, and lack of dynamic correction mechanism. These problems are mainly caused by the fact that existing monitoring technologies rely on a single data source and lack advanced data processing and analysis technologies, which makes it difficult to obtain comprehensive and accurate information on blockages in complex environments.

[0003] The shortcomings or defects of the existing technology (Chinese invention patent, publication number: CN114998827A, name: A method and system for monitoring and processing blockages in municipal drainage pipes) in the face of these problems are mainly reflected in the following aspects: First, the data collection is single and mainly relies on optical images, which makes it difficult to fully capture blockage information in complex environments. Secondly, the data preprocessing is insufficient, and the data noise and normalization processing are not fully removed, which affects the accuracy of feature extraction. Third, the feature extraction method is simple and cannot fully capture the deep-level features of multimodal data. Fourth, the discriminant model is relatively simple and it is difficult to handle the complexity of multimodal data, which can easily lead to misjudgment or missed judgment. Finally, there is a lack of dynamic adaptive correction mechanism, and the model cannot be optimized according to real-time feedback data, which affects the accuracy and real-time performance of the discrimination results. Summary of the invention

[0004] In view of the many problems existing in the above-mentioned prior art, the present invention provides a method and system for monitoring and processing the blockage of a tunnel drainage pipe. The present invention collects comprehensive internal data of the pipe through a multimodal sensor array, and after denoising and normalization preprocessing, uses a convolutional neural network, a three-dimensional convolutional neural network and a long short-term memory network to extract multi-scale features; dynamically adjusts the weights of each modal feature through a self-attention mechanism, uses a graph neural network to extract high-level correlation features, and generates optimized multimodal comprehensive feature data; based on these feature data, a fully connected neural network identifies the type, quantity, size, degree of blockage and difficulty of cleaning of the blockage, and performs dynamic correction through real-time data feedback and a self-learning mechanism, and finally generates cleaning solution data and executes the cleaning task through a robotic arm; the scheme realizes comprehensive monitoring and efficient processing of the blockage of the tunnel drainage pipe.

[0005] likeFigure 1-Figure 2 As shown, a method for monitoring and handling blockage of a tunnel drainage pipe comprises the following steps:

[0006] Use a multimodal sensor array to collect optical image data, three-dimensional point cloud data, ultrasonic reflection data, gas composition data, infrared imaging data, acoustic image data, electromagnetic induction data and flow rate data inside the drainage pipe, denoise and normalize the collected raw data to generate a preprocessed raw feature data set;

[0007] The preprocessed original feature data set is processed by convolutional neural network, three-dimensional convolutional neural network and long short-term memory network to extract multi-scale features and generate a pooled feature data set;

[0008] Use the self-attention mechanism to dynamically adjust the weights of each modal feature, perform graph neural network processing on the pooled feature data set, extract high-level correlation features, and generate optimized multi-modal comprehensive feature data;

[0009] Based on the optimization of multi-modal comprehensive feature data, the discriminant model is used to discriminate the type, quantity, size, degree of blockage and difficulty of cleaning of the blockage, and the discriminant result data is generated. The discriminant model parameters are dynamically and adaptively corrected through real-time data feedback and self-learning mechanism to generate corrected discriminant result data;

[0010] Perform preliminary judgment, secondary judgment and final decision on the correction judgment result data, use the rule engine and machine learning model to evaluate the necessity of cleaning, and generate the final cleaning plan data;

[0011] According to the final cleaning plan data, a robotic arm is used to perform the cleaning task, and multimodal sensors are used for real-time monitoring to collect feedback data, optimize the cleaning strategy and discrimination model, and generate the final cleaning feedback data.

[0012] Preferably, the multimodal sensor array includes a high-resolution optical camera for capturing optical image data inside the drainage pipe; a lidar for acquiring three-dimensional point cloud data; an ultrasonic sensor for measuring the distance and shape of objects inside the pipe and generating ultrasonic reflection data; a gas sensor for detecting the gas composition inside the pipe and generating gas composition data; an infrared imager for capturing thermal imaging data inside the pipe; an acoustic wave imaging sensor for generating acoustic wave image data; an electromagnetic induction sensor for detecting metals and conductive materials and generating electromagnetic induction data; and a flow rate sensor for measuring the liquid flow rate inside the drainage pipe and generating flow rate data.

[0013] Preferably, the preprocessing includes denoising the collected raw data, specifically filtering and denoising the optical image data, three-dimensional point cloud data, ultrasonic reflection data, gas composition data, infrared imaging data, acoustic image data, electromagnetic induction data and flow rate data to generate a clean data set; and normalizing the denoised data to ensure consistency and comparability of the data to generate a preprocessed raw feature data set.

[0014] Preferably, the multi-scale feature extraction includes:

[0015] The convolutional neural network performs multi-layer convolution and pooling operations on the preprocessed optical image feature data and infrared imaging feature data, extracts spatial features, and generates pooled optical feature data and pooled infrared feature data;

[0016] The 3D convolutional neural network performs voxel processing on the preprocessed 3D point cloud feature data, converts the 3D point data into voxel data, and performs multi-layer 3D convolution and pooling operations to extract 3D spatial features and generate pooled 3D point cloud feature data;

[0017] The long short-term memory network performs time series processing on the preprocessed ultrasonic reflection characteristic data and gas composition characteristic data, extracts time-related features, and generates time-characteristic ultrasonic data and time-characteristic gas data.

[0018] Preferably, the self-attention mechanism dynamically adjusts the weight of each modal feature, specifically including generating weighted feature data according to the importance of each modal feature in different tasks, including weighted pooled optical feature data, weighted pooled infrared feature data, weighted pooled three-dimensional point cloud feature data, weighted time feature ultrasonic data, weighted time feature gas data, weighted acoustic image feature data, weighted electromagnetic induction feature data and weighted flow velocity feature data.

[0019] Preferably, the graph neural network processes the weighted feature data, specifically including:

[0020] Construct a feature graph, and use weighted pooled optical feature data, weighted pooled infrared feature data, weighted pooled three-dimensional point cloud feature data, weighted time feature ultrasonic data, weighted time feature gas data, weighted acoustic wave image feature data, weighted electromagnetic induction feature data, and weighted flow velocity feature data as graph nodes;

[0021] Use graph neural networks to extract high-level correlation features between graph nodes and generate graph feature data.

[0022] Preferably, the discriminant model performs a comprehensive analysis on the optimized multimodal comprehensive feature data, specifically including:

[0023] Use a fully connected neural network to perform multi-layer fully connected processing on the optimized multi-modal comprehensive feature data to generate fully connected feature data;

[0024] Based on the fully connected feature data, the types, quantities, and sizes of blockages are classified, and regression analysis of the degree of blockage and difficulty of cleaning is performed to generate discrimination result data.

[0025] Preferably, the dynamic adaptive correction of the discriminant model parameters by real-time data feedback and self-learning mechanism specifically includes:

[0026] Based on real-time data feedback, an adaptive learning algorithm is used to adjust the parameters of the discrimination model to generate corrected discrimination result data;

[0027] Through the self-learning mechanism, based on historical judgment results and real-time data feedback, the reinforcement learning algorithm is used to optimize the model structure and learning strategy to generate optimized model parameters.

[0028] Preferably, the preliminary discrimination, secondary discrimination and final decision making of the correction discrimination result data specifically include:

[0029] Use the rule engine to make a preliminary judgment on the correction judgment result data to generate preliminary judgment result data;

[0030] Conduct secondary data collection and analysis based on the preliminary discrimination result data, use the discrimination model to conduct secondary discrimination, and generate secondary discrimination result data;

[0031] The preliminary discrimination result data and the secondary discrimination result data are integrated, and a decision tree or random forest model is used to make the final decision to generate the final cleaning solution data.

[0032] A system for implementing the tunnel drainage pipe blockage monitoring and processing method, comprising:

[0033] A multimodal sensor array, including a high-resolution optical camera, a lidar, an ultrasonic sensor, a gas sensor, an infrared imager, an acoustic imaging sensor, an electromagnetic induction sensor, and a flow rate sensor, for collecting optical image data, three-dimensional point cloud data, ultrasonic reflection data, gas composition data, infrared imaging data, acoustic image data, electromagnetic induction data, and flow rate data inside the drainage pipe;

[0034] The data preprocessing module is used to perform denoising and normalization on the collected raw data to generate a preprocessed raw feature data set;

[0035] The multi-scale feature extraction module includes a convolutional neural network, a three-dimensional convolutional neural network, and a long short-term memory network, which is used to process the preprocessed original feature data set, extract multi-scale features, and generate a pooled feature data set;

[0036] The self-attention mechanism module is used to dynamically adjust the weights of each modality feature and generate weighted feature data;

[0037] The graph neural network module is used to process weighted feature data, extract high-level correlation features, and generate graph feature data;

[0038] The discrimination model module includes a fully connected neural network, which is used to discriminate the type, quantity, size, degree of obstruction and difficulty of cleaning of the obstruction based on the graph feature data, and generate discrimination result data;

[0039] The real-time data feedback and self-learning mechanism module is used to dynamically and adaptively correct the discrimination model parameters according to the real-time data feedback and historical discrimination results, generate the corrected discrimination result data and optimize the model parameters;

[0040] The decision module includes a rule engine and a machine learning model, which is used to make preliminary judgments, secondary judgments and final decisions on the correction judgment result data and generate final cleaning solution data;

[0041] A robot arm execution module, used for executing the cleaning task according to the final cleaning plan data;

[0042] The real-time monitoring and feedback module is used to monitor the cleaning task in real time through multimodal sensors, collect feedback data, optimize the cleaning strategy and discrimination model, and generate the final cleaning feedback data.

[0043] Compared with the prior art, the advantages and beneficial effects of the present invention are:

[0044] The present invention uses a multimodal sensor array to comprehensively collect optical images, three-dimensional point clouds, ultrasonic reflections, gas composition, infrared imaging, acoustic wave images, electromagnetic induction and flow rate data, thereby achieving comprehensiveness and accuracy in data collection;

[0045] The present invention extracts multi-scale features through multiple deep learning models such as convolutional neural networks, three-dimensional convolutional neural networks, and long short-term memory networks, achieving the depth and breadth of feature extraction;

[0046] The present invention uses the self-attention mechanism and graph neural network to dynamically adjust the weights of each modal feature, extract high-level related features, and realize the intelligent and dynamic adjustment capabilities of multi-modal feature fusion;

[0047] The present invention realizes accurate identification of the type, quantity, size, degree of blockage and difficulty of cleaning of blockages through a fully connected neural network and a self-learning mechanism, thereby improving the complexity and accuracy of the identification model.

[0048] The present invention realizes a dynamic adaptive correction mechanism through real-time data feedback and a self-learning mechanism, thereby improving the real-time performance and self-learning ability of the discrimination result. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a schematic diagram of the process of the present invention;

[0050] Figure 2 It is a data processing flow chart in the present invention;

[0051] Figure 3 This is a flow chart of identification and correction in the present invention;

[0052] Figure 4 It is a structural block diagram of the system of the present invention. DETAILED DESCRIPTION

[0053] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is obvious that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0054] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0055] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0056] In the case of using expressions such as "at least one of A, B, and C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.). In the case of using expressions such as "at least one of A, B, or C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, or C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0057] Some block diagrams and / or flow charts are shown in the accompanying drawings. It should be understood that some boxes or combinations thereof in the block diagrams and / or flow charts can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that these instructions can create a device for implementing the functions / operations described in these block diagrams and / or flow charts when executed by the processor. The technology of the present disclosure can be implemented in the form of hardware and / or software (including firmware, microcode, etc.). In addition, the technology of the present disclosure can take the form of a computer program product on a computer-readable storage medium storing instructions, which can be used by an instruction execution system or used in conjunction with an instruction execution system.

[0058] A method for monitoring and treating blockage of a tunnel drainage pipe comprises the following steps:

[0059] Use a multimodal sensor array to collect optical image data, three-dimensional point cloud data, ultrasonic reflection data, gas composition data, infrared imaging data, acoustic image data, electromagnetic induction data and flow rate data inside the drainage pipe, denoise and normalize the collected raw data to generate a preprocessed raw feature data set;

[0060] Preferably, the multimodal sensor array includes a high-resolution optical camera for capturing optical image data inside the drainage pipe; a lidar for acquiring three-dimensional point cloud data; an ultrasonic sensor for measuring the distance and shape of objects inside the pipe and generating ultrasonic reflection data; a gas sensor for detecting the gas composition inside the pipe and generating gas composition data; an infrared imager for capturing thermal imaging data inside the pipe; an acoustic wave imaging sensor for generating acoustic wave image data; an electromagnetic induction sensor for detecting metals and conductive materials and generating electromagnetic induction data; and a flow rate sensor for measuring the liquid flow rate inside the drainage pipe and generating flow rate data.

[0061] One of the key steps in the present invention is to use a multimodal sensor array to collect various data inside the tunnel drainage pipe. The multimodal sensor array can capture different types of data, including optical image data, three-dimensional point cloud data, ultrasonic reflection data, gas composition data, infrared imaging data, acoustic image data, electromagnetic induction data, and flow rate data. Each sensor can capture different features in a specific environment, thereby forming a comprehensive feature data set.

[0062] High-resolution optical cameras are used to capture optical image data from inside the sewer pipes. These image data provide an intuitive view of the inside of the pipes, helping to identify obvious blockages and structural damage. The advantages of optical image data are its high resolution and intuitiveness, but its effectiveness may be limited in low-light conditions or when the pipe interior is obscured.

[0063] Laser radar (LiDAR) is used to obtain 3D point cloud data. By emitting laser light and measuring the time difference of reflected light, LiDAR can generate a 3D model of the inside of the pipeline. This 3D point cloud data can accurately describe the geometry of the inside of the pipeline and the spatial distribution of blockages. The advantages of LiDAR are its high accuracy and wide range of applications, and its ability to work normally in low light conditions.

[0064] Ultrasonic sensors are used to measure distances and object shapes inside pipes, generating ultrasonic reflection data. Ultrasonic sensors determine the position and shape of objects by emitting sound waves and measuring the time difference between reflected waves. This technology is very effective in detecting objects in liquid and solid media, and is particularly suitable for detecting solid blockages in liquids.

[0065] Gas sensors are used to detect the gas composition inside the pipeline and generate gas composition data. These sensors can detect different types of gases, such as toxic, flammable or corrosive gases, thereby providing key information about the environment inside the pipeline. Gas composition data can help determine chemical reactions and gas leaks in the pipeline.

[0066] Infrared imagers are used to capture thermal imaging data from inside pipelines. Infrared imagers detect infrared radiation emitted by objects to generate thermal images that can show temperature distribution and thermal anomalies inside pipelines. Thermal imaging data is very useful for identifying heat sources, cold spots, and heat losses within pipelines.

[0067] Acoustic imaging sensors are used to generate acoustic image data. Acoustic imaging sensors generate acoustic images of the interior of a pipe by transmitting and receiving sound waves. These images can show the sound reflection characteristics within the pipe and help identify different materials and structural features. Acoustic image data is very effective in detecting hidden blockages and structural damage.

[0068] Electromagnetic induction sensors are used to detect metal and conductive materials in pipes and generate electromagnetic induction data. Electromagnetic induction sensors can identify metal objects and conductive blockages in pipes by detecting the electromagnetic properties of conductive materials. This technology is very effective in detecting metal pipes and metal blockages.

[0069] Flow rate sensors are used to measure the flow rate of liquid inside drainage pipes and generate flow rate data. Flow rate data can provide real-time information on the flow of liquid in the pipes and help determine the degree of blockage and flow resistance of the pipes.

[0070] All the data from these sensors are denoised and normalized to generate a pre-processed raw feature data set. Denoising can remove noise from the data and improve the accuracy of the data; normalization can ensure that different types of data have a consistent scale, which is convenient for subsequent data fusion and analysis.

[0071] In summary, using a multimodal sensor array to collect various data inside the drainage pipe can provide comprehensive information about the inside of the pipe, thereby improving the accuracy and efficiency of monitoring and handling blockages. By combining data from different sensors, the system can identify and locate various types of blockages, determine the severity of the blockage, and develop corresponding cleaning plans.

[0072] Preferably, the preprocessing includes denoising the collected raw data, specifically filtering and denoising the optical image data, three-dimensional point cloud data, ultrasonic reflection data, gas composition data, infrared imaging data, acoustic image data, electromagnetic induction data and flow rate data to generate a clean data set; and normalizing the denoised data to ensure consistency and comparability of the data to generate a preprocessed raw feature data set.

[0073] First, the collected optical image data, three-dimensional point cloud data, ultrasonic reflection data, gas composition data, infrared imaging data, acoustic wave image data, electromagnetic induction data and flow rate data are filtered and denoised. The purpose of denoising is to eliminate noise in the data and improve the quality and accuracy of the data. Noise is an inevitable interference factor in the data acquisition process, which will have a negative impact on subsequent data analysis and processing. Filtering denoising technology uses various mathematical methods, such as mean filtering, Gaussian filtering and median filtering, to smooth data and reduce noise. For example, Gaussian filtering of optical image data can effectively remove random noise in the image, making the image clearer; median filtering of ultrasonic reflection data can eliminate pulse noise and retain the edge details of the signal. Through denoising, the generated clean data set is more accurate and reliable, laying the foundation for subsequent analysis.

[0074] Secondly, the denoised data is normalized to ensure the consistency and comparability of the data. Since different types of data have different scales and dimensions, directly comparing and analyzing these data will lead to deviations and errors. Normalization converts the data into a standard range so that different types of data are processed at the same scale. Commonly used normalization methods include minimum-maximum normalization and z-score normalization. For example, flow velocity data may range from 0 to several meters per second, while electromagnetic induction data may range from 0 to several thousand millivolts. Through normalization, they can be converted to the same range (such as 0 to 1 or -1 to 1) to facilitate subsequent feature extraction and model training. Normalization not only improves the consistency of the data, but also reduces the deviation caused by different scales, so that different types of data can play a more effective role when fused.

[0075] Through the above preprocessing steps, the preprocessed original feature data set is generated. This data set contains all the denoised and normalized data, with high quality and standardized characteristics, providing a reliable data basis for subsequent multi-scale feature extraction, feature fusion and blockage situation identification. Preprocessing is not only a necessary step in data processing, but also a key link in improving the accuracy and efficiency of the entire system. By eliminating noise and standardizing data, preprocessing ensures the accuracy and stability of subsequent data analysis, and ultimately improves the overall effect of monitoring and processing of tunnel drainage pipe blockage.

[0076] The preprocessed original feature data set is processed by convolutional neural network, three-dimensional convolutional neural network and long short-term memory network to extract multi-scale features and generate a pooled feature data set;

[0077] Preferably, the multi-scale feature extraction includes:

[0078] The convolutional neural network performs multi-layer convolution and pooling operations on the preprocessed optical image feature data and infrared imaging feature data, extracts spatial features, and generates pooled optical feature data and pooled infrared feature data;

[0079] The 3D convolutional neural network performs voxel processing on the preprocessed 3D point cloud feature data, converts the 3D point data into voxel data, and performs multi-layer 3D convolution and pooling operations to extract 3D spatial features and generate pooled 3D point cloud feature data;

[0080] The long short-term memory network performs time series processing on the preprocessed ultrasonic reflection characteristic data and gas composition characteristic data, extracts time-related features, and generates time-characteristic ultrasonic data and time-characteristic gas data.

[0081] First, the convolutional neural network (CNN) performs multi-layer convolution and pooling operations on the preprocessed optical image feature data and infrared imaging feature data. Through the local receptive field and weight sharing mechanism, CNN can effectively extract spatial features in the image, such as edges, corners, and textures. In the multi-layer convolution process, the low-level convolution layer extracts the basic features of the image, while the high-level convolution layer extracts more complex features. Pooling operations (such as maximum pooling and average pooling) reduce the dimension of the data, reduce the amount of calculation, and retain important features, thereby generating pooled optical feature data and pooled infrared feature data. These feature data can reflect the morphological characteristics and thermal distribution inside the pipeline, which helps to identify blockages and structural abnormalities.

[0082] Secondly, the three-dimensional convolutional neural network (3D CNN) voxelizes the preprocessed three-dimensional point cloud feature data, converts the three-dimensional point data into voxel data, and performs multi-layer three-dimensional convolution and pooling operations. Through convolution operations in three-dimensional space, 3D CNN can capture the spatial features in the three-dimensional point cloud data, such as the shape, size, and spatial position of objects. During the voxelization process, the three-dimensional point data is divided into multiple small voxels (volume pixels), and each voxel contains point data within a certain range. Through three-dimensional convolution operations, 3D CNN can extract local features in three-dimensional space and generate pooled three-dimensional point cloud feature data. These feature data can accurately describe the geometric shape inside the pipeline and help locate and identify the spatial position of the blockage.

[0083] Finally, the long short-term memory network (LSTM) performs time series processing on the preprocessed ultrasonic reflection feature data and gas composition feature data. LSTM is a special recurrent neural network (RNN) that can effectively process long-term dependencies in time series data. In LSTM processing, ultrasonic reflection data and gas composition data are used as time series inputs. LSTM extracts time-related features through memory and forgetting mechanisms to generate time-featured ultrasonic data and time-featured gas data. These feature data can reflect the dynamic changes in the internal environment and materials of the pipeline, and help detect and analyze the formation process and change laws of blockages.

[0084] Through the above multi-scale feature extraction process, the generated pooled feature data set contains multi-dimensional feature information such as optical, infrared, three-dimensional space and time series, providing a comprehensive and detailed data basis for subsequent multi-modal feature fusion and blockage situation identification. This step not only improves the efficiency of data processing, but also enhances the system's ability to identify complex blockage situations, ultimately improving the overall effect of monitoring and processing tunnel drainage pipe blockage.

[0085] For example, in practical applications, after optical image data and infrared imaging data are processed by CNN, clear images of the interior of the pipeline can be generated to show the location and shape of the blockage; after 3D point cloud data are processed by 3D CNN, detailed 3D models can be generated to accurately describe the spatial distribution of the blockage; after ultrasonic reflection data and gas composition data are processed by LSTM, time series features can be generated to show the dynamic changes of the blockage. The fusion and analysis of these feature data will help to accurately determine the blockage situation and formulate effective cleaning plans.

[0086] Use the self-attention mechanism to dynamically adjust the weights of each modal feature, perform graph neural network processing on the pooled feature data set, extract high-level correlation features, and generate optimized multi-modal comprehensive feature data;

[0087] Preferably, the self-attention mechanism dynamically adjusts the weight of each modal feature, specifically including generating weighted feature data according to the importance of each modal feature in different tasks, including weighted pooled optical feature data, weighted pooled infrared feature data, weighted pooled three-dimensional point cloud feature data, weighted time feature ultrasonic data, weighted time feature gas data, weighted acoustic image feature data, weighted electromagnetic induction feature data and weighted flow velocity feature data.

[0088] In the present invention, the use of self-attention mechanism to dynamically adjust the weight of each modal feature and perform graph neural network (GNN) processing on the pooled feature data set is a key step to improve feature fusion and discrimination accuracy. The combination of self-attention mechanism and graph neural network can make full use of the data of each modal feature, extract high-level related features, and generate optimized multi-modal comprehensive feature data, thereby improving the system's ability to identify complex congestion situations.

[0089] The self-attention mechanism dynamically adjusts the weight of each modal feature by calculating its importance in different tasks. Specifically, the self-attention mechanism assigns corresponding weights according to the contribution of each modal feature in the task and generates weighted feature data. These weighted feature data include weighted pooled optical feature data, weighted pooled infrared feature data, weighted pooled three-dimensional point cloud feature data, weighted time feature ultrasonic data, weighted time feature gas data, weighted acoustic image feature data, weighted electromagnetic induction feature data and weighted flow velocity feature data. The advantage of the self-attention mechanism is that it can adaptively focus on the most critical features in the task, thereby retaining important information and ignoring irrelevant noise data during the data fusion process.

[0090] When dealing with blockages in tunnel drainage pipes, different sensor data has different importance for different types of blockages. For example, optical image feature data is very effective in identifying obvious physical blockages, while gas composition feature data is more critical in detecting chemical blockages and gas leaks. The self-attention mechanism learns these differences and dynamically adjusts the weights of each modal feature during the feature fusion process, thereby improving the effectiveness of the overall feature data.

[0091] After obtaining the weighted feature data, the graph neural network (GNN) is used to process these weighted feature data and extract high-level correlation features. GNN can capture complex relationships and patterns in the data by calculating nodes and edges on the graph structure. In the present invention, the constructed feature graph uses each modal feature data as a graph node, and through the processing of the graph neural network, the high-level correlation features between the nodes are extracted to generate graph feature data.

[0092] In the specific implementation, the graph neural network first constructs a feature graph, using weighted pooled optical feature data, weighted pooled infrared feature data, weighted pooled 3D point cloud feature data, weighted time feature ultrasonic data, weighted time feature gas data, weighted acoustic image feature data, weighted electromagnetic induction feature data, and weighted flow velocity feature data as graph nodes. Then, through the graph convolution operation, the correlation between nodes is calculated to extract high-level graph feature data. These graph feature data reflect the complex relationship and mutual influence between different modal features, and can more accurately describe the blockage inside the pipeline.

[0093] For example, in a specific application scenario, optical image data and three-dimensional point cloud data can be combined to provide information on the morphology and spatial distribution of the interior of the pipeline; at the same time, ultrasonic data and gas composition data provide information on the properties and dynamic changes of the material. The self-attention mechanism dynamically adjusts the weights of these features according to the needs of the task, so that the most important information is strengthened and irrelevant information is weakened during the graph neural network processing. The resulting graph feature data not only contains useful information about each modal feature, but also captures the high-level relationship between features through the graph structure, thereby providing more accurate and comprehensive input data for subsequent discriminant models.

[0094] In this way, the combination of self-attention mechanism and graph neural network realizes the efficient fusion of multimodal features and the extraction of deep correlation features, significantly improving the system's monitoring and processing capabilities for tunnel drainage pipe blockage. This method not only improves the accuracy and efficiency of data processing, but also enhances the robustness and adaptability of the system, and can cope with complex and changeable practical application scenarios.

[0095] Preferably, the graph neural network processes the weighted feature data, specifically including:

[0096] Construct a feature graph, and use weighted pooled optical feature data, weighted pooled infrared feature data, weighted pooled three-dimensional point cloud feature data, weighted time feature ultrasonic data, weighted time feature gas data, weighted acoustic wave image feature data, weighted electromagnetic induction feature data, and weighted flow velocity feature data as graph nodes;

[0097] Use graph neural networks to extract high-level correlation features between graph nodes and generate graph feature data.

[0098] In the present invention, the processing of weighted feature data by graph neural network (GNN) is one of the key steps. This step generates graph feature data by constructing a feature graph and using graph neural network to extract high-level correlation features between graph nodes, thereby improving the recognition ability and discrimination accuracy of complex congestion situations.

[0099] First of all, constructing a feature graph is the basis of graph neural network processing. The construction of the feature graph includes taking weighted pooled optical feature data, weighted pooled infrared feature data, weighted pooled 3D point cloud feature data, weighted time feature ultrasonic data, weighted time feature gas data, weighted acoustic image feature data, weighted electromagnetic induction feature data, and weighted flow velocity feature data as graph nodes. In this feature graph, each node represents a feature data, and the edges between nodes represent the association between these feature data. The construction of the feature graph needs to consider the correlation and mutual influence of feature data. In this way, the feature graph can fully reflect the complex relationship between various feature data.

[0100] Graph neural networks (GNNs) process feature graphs to extract high-level correlation features between graph nodes. Specifically, GNNs aggregate the information of a node and its neighboring nodes through graph convolution operations and calculate high-level features of each node. This graph convolution operation can capture the complex correlation between feature data, so that the generated graph feature data contains richer and deeper information. For example, the spatial correlation between optical feature data and infrared feature data, the physical correlation between three-dimensional point cloud feature data and flow velocity feature data, and the dynamic change correlation between ultrasonic feature data and gas composition feature data can all be effectively modeled and extracted through graph neural networks.

[0101] In the specific implementation, the graph neural network first initializes the feature graph, that is, weighted pooled optical feature data, weighted pooled infrared feature data, weighted pooled 3D point cloud feature data, weighted time feature ultrasonic data, weighted time feature gas data, weighted acoustic image feature data, weighted electromagnetic induction feature data and weighted flow velocity feature data are used as initial node features. Then, through multi-layer graph convolution operations, high-level features of the nodes are extracted layer by layer. In each layer of graph convolution operations, the information of the node comes not only from itself, but also from its neighboring nodes. In this way, the features of the node gradually integrate the information of its neighboring nodes to form more comprehensive and high-level graph feature data.

[0102] For example, in the specific application of monitoring and handling blockages in tunnel drainage pipes, optical feature data can provide visual information inside the pipe, while infrared feature data can display the heat distribution inside the pipe. Through the processing of graph neural networks, the associated information of these two feature data is fully extracted and fused, so that the generated graph feature data can not only reflect the morphology and temperature distribution inside the pipe, but also capture the correlation pattern between them. Similarly, the three-dimensional point cloud feature data and flow velocity feature data are processed by graph neural networks, and the generated graph feature data can comprehensively reflect the geometric morphology and liquid flow conditions inside the pipe, providing more accurate and comprehensive information for the location of blockages and the analysis of flow resistance.

[0103] By processing weighted feature data with graph neural networks, the generated graph feature data has higher expressiveness and discrimination capabilities. This method can not only effectively extract high-level correlation features between multimodal feature data, but also retain and utilize the advantages of different feature data in the process of data fusion and discrimination, thereby significantly improving the overall effect of monitoring and processing tunnel drainage pipe blockage. Ultimately, through this comprehensive and in-depth feature extraction and analysis method, the system can more accurately identify and judge blockages, providing reliable data support for formulating effective cleaning plans.

[0104] like Figure 3 As shown, based on the optimization of multimodal comprehensive feature data, a discriminant model is used to discriminate the type, quantity, size, degree of blockage and difficulty of cleaning of blockages, generate discrimination result data, and dynamically and adaptively correct the discriminant model parameters through real-time data feedback and self-learning mechanism to generate corrected discrimination result data;

[0105] Preferably, the discriminant model performs a comprehensive analysis on the optimized multimodal comprehensive feature data, specifically including:

[0106] Use a fully connected neural network to perform multi-layer fully connected processing on the optimized multi-modal comprehensive feature data to generate fully connected feature data;

[0107] Based on the fully connected feature data, the types, quantities, and sizes of blockages are classified, and regression analysis of the degree of blockage and difficulty of cleaning is performed to generate discrimination result data.

[0108] In the present invention, based on the optimization of multi-modal comprehensive feature data, the use of a discriminant model to discriminate the type, quantity, size, degree of blockage and difficulty of cleaning of the blockage is one of the core links of the solution. This link generates discriminant result data by comprehensively analyzing the feature data of different modes, and dynamically and adaptively corrects the discriminant model through real-time data feedback and self-learning mechanism to ensure the accuracy and reliability of the discriminant results.

[0109] The optimized multimodal comprehensive feature data is the high-level correlation feature data processed by the self-attention mechanism and graph neural network in the previous steps. These data integrate the feature information of multiple modes, such as optical images, infrared imaging, 3D point clouds, ultrasonic reflection, gas composition, acoustic wave images, electromagnetic induction and flow rate data. These optimized feature data provide a comprehensive description of the internal conditions of the pipeline, providing a solid foundation for subsequent judgment.

[0110] The discriminant model uses a fully connected neural network (FCNN) to perform multi-layer fully connected processing on the optimized multimodal comprehensive feature data. The fully connected neural network can effectively extract deep patterns and relationships in the input feature data through the connection of multiple layers of neurons. In the present invention, the multi-layer processing of the fully connected neural network can convert the optimized multimodal comprehensive feature data into fully connected feature data, which contains rich information about the type, quantity, size, degree of blockage and cleaning difficulty of the blockage.

[0111] Specifically, the discriminant model performs classification and regression analysis on the fully connected feature data. In the classification task, the model classifies the type, quantity, and size of the blockage and generates corresponding classification labels. For example, for the type of blockage, the model may output categories such as "mud", "roots", and "garbage"; for the number of blockages, the model will output a specific quantity value; for the size of the blockage, the model will output the corresponding size classification. In the regression analysis task, the model performs regression analysis on the degree of blockage and the difficulty of cleaning to generate continuous numerical results. For example, the degree of blockage may be quantified as a range of values ​​from 0 to 1, indicating the severity of the blockage; the difficulty of cleaning may be quantified as a range of values ​​from 1 to 10, indicating the complexity and time-consuming nature of the cleaning task. Through these analyses, the discriminant model generates detailed discriminant result data, which provides an important basis for the evaluation of pipeline blockage and the formulation of cleaning plans.

[0112] In order to ensure the accuracy and dynamic adaptability of the discrimination results, real-time data feedback and self-learning mechanisms are introduced into the scheme. Real-time data feedback collects new data in real time through the sensor array and monitoring system and feeds it back to the discrimination model. The self-learning mechanism enables the discrimination model to adapt to the dynamically changing pipeline environment and blockage conditions by continuously updating and optimizing the model parameters. In the specific implementation, the self-learning mechanism includes an adaptive learning algorithm based on real-time data feedback and a reinforcement learning algorithm based on historical data. The adaptive learning algorithm generates corrected discrimination result data by adjusting the model parameters in real time; the reinforcement learning algorithm optimizes the model structure and learning strategy by learning the successful experiences and failure lessons in the historical data, and generates optimized model parameters. These dynamic correction and optimization mechanisms ensure the accuracy and robustness of the discrimination model, enabling it to provide reliable discrimination results in complex and changing actual environments.

[0113] For example, in a specific application scenario, by optimizing multimodal comprehensive feature data, the fully connected neural network can accurately identify the type of obstruction in the pipeline as "tree roots", the number as "2 roots", the size as "5 cm in diameter", the degree of obstruction as "0.7" (indicating moderate obstruction), and the cleaning difficulty as "7" (indicating high difficulty). Real-time data feedback and self-learning mechanisms ensure that even when the pipeline environment changes (such as the emergence of new obstructions or the movement of existing obstructions), the discrimination model can still accurately update the discrimination results and provide the latest assessment of the blockage situation. This dynamic adaptability enables the system to maintain efficient and accurate monitoring and processing capabilities when facing complex and dynamically changing pipeline environments.

[0114] Through this series of feature extraction, comprehensive analysis, real-time feedback and self-learning mechanisms, the present invention realizes accurate monitoring and efficient processing of blockages in tunnel drainage pipes, providing strong technical support for ensuring the normal operation and timely cleaning of pipes.

[0115] Preferably, the dynamic adaptive correction of the discriminant model parameters by real-time data feedback and self-learning mechanism specifically includes:

[0116] Based on real-time data feedback, an adaptive learning algorithm is used to adjust the parameters of the discrimination model to generate corrected discrimination result data;

[0117] Through the self-learning mechanism, based on historical judgment results and real-time data feedback, the reinforcement learning algorithm is used to optimize the model structure and learning strategy to generate optimized model parameters.

[0118] Based on real-time data feedback, an adaptive learning algorithm is used to adjust the parameters of the discriminant model. The real-time data feedback mechanism collects new data inside the pipeline in real time through a multimodal sensor array, such as optical image data, infrared imaging data, and three-dimensional point cloud data. These real-time data reflect the latest status and changes inside the pipeline. The adaptive learning algorithm uses these real-time data to dynamically adjust the parameters of the discriminant model to generate more accurate corrected discrimination result data. For example, when the sensor detects a new blockage, the real-time data feedback passes this information to the discriminant model, and the adaptive learning algorithm adjusts the model parameters according to the new data, so that the model can timely update the discrimination results of the type, quantity, size, degree of blockage, and difficulty of cleaning. Through this dynamic adjustment mechanism, the discriminant model can always maintain high-precision monitoring and identification of the internal conditions of the pipeline.

[0119] Through the self-learning mechanism, the reinforcement learning algorithm is used to optimize the model structure and learning strategy based on historical discrimination results and real-time data feedback. Reinforcement learning is a machine learning method that continuously optimizes the decision-making process through trial and error and reward mechanisms. In the present invention, the self-learning mechanism continuously optimizes the structure and learning strategy of the discrimination model by analyzing historical discrimination results and real-time data feedback. Specifically, the reinforcement learning algorithm identifies and strengthens successful discrimination strategies through a reward mechanism, and reduces and corrects incorrect discrimination strategies through a penalty mechanism. In this way, the discrimination model can improve its overall performance and robustness in the process of continuous learning and optimization.

[0120] For example, in a specific application scenario, the discriminant model initially detected a blockage in the pipeline, but was unable to accurately identify its type and quantity. Through real-time data feedback, the adaptive learning algorithm adjusts the model parameters so that the model can update the discrimination results in a timely manner and accurately identify the blockage as "silt" and the quantity as "3 piles". At the same time, the self-learning mechanism analyzes historical discrimination data to discover some previous misjudgments, and adjusts the model structure and learning strategy through the reinforcement learning algorithm to avoid similar errors in the future. This dynamic adaptive correction mechanism ensures the accuracy and stability of the discrimination model, enabling it to provide reliable discrimination results in complex and dynamically changing pipeline environments.

[0121] Through the combination of real-time data feedback and self-learning mechanism, the present invention can not only make rapid adjustments with the support of real-time data, but also continuously optimize the model through the accumulation and analysis of historical data. In this way, the discriminant model can always maintain efficient and accurate monitoring and processing capabilities when facing various complex blockage situations, and improve the overall effect of monitoring and processing of tunnel drainage pipe blockage.

[0122] In summary, through real-time data feedback and self-learning mechanism, the parameters of the discrimination model are dynamically and adaptively corrected. By using adaptive learning algorithm and reinforcement learning algorithm, not only can the model parameters be adjusted and optimized in real time, but also the model structure and learning strategy can be continuously improved and optimized, thereby providing more accurate and stable discrimination results in complex and dynamically changing actual application environments, and providing strong technical support for the monitoring and treatment of pipeline blockages.

[0123] Perform preliminary judgment, secondary judgment and final decision on the correction judgment result data, use the rule engine and machine learning model to evaluate the necessity of cleaning, and generate the final cleaning plan data;

[0124] Preferably, the preliminary discrimination, secondary discrimination and final decision making of the correction discrimination result data specifically include:

[0125] Use the rule engine to make a preliminary judgment on the correction judgment result data to generate preliminary judgment result data;

[0126] Conduct secondary data collection and analysis based on the preliminary discrimination result data, use the discrimination model to conduct secondary discrimination, and generate secondary discrimination result data;

[0127] The preliminary discrimination result data and the secondary discrimination result data are integrated, and a decision tree or random forest model is used to make the final decision to generate the final cleaning solution data.

[0128] In the present invention, the preliminary judgment, secondary judgment and final decision of the correction judgment result data are the key links to ensure the accuracy and effectiveness of the cleaning plan. This link combines the rule engine and the machine learning model to make multi-level judgments and decisions on the blockage situation, generate the final cleaning plan data, and ensure the necessity and effectiveness of the cleaning measures.

[0129] First, a preliminary judgment is made on the corrected judgment result data. The preliminary judgment uses the rule engine to perform a preliminary analysis on the corrected judgment result data. The rule engine quickly judges and classifies the input data based on a predefined set of rules. The rule set can include standards and thresholds for different types, quantities, sizes, degrees of blockage, and difficulty of clearing. For example, if the judgment result shows that the type of blockage is "silt" and the degree of blockage exceeds a certain threshold, the rule engine can immediately determine that the blockage needs to be cleared. Through the preliminary judgment of the rule engine, preliminary judgment result data is generated to ensure that obvious blockages that need to be dealt with are quickly identified.

[0130] Next, secondary data collection and analysis are performed based on the preliminary discrimination result data. For uncertain or complex situations in the preliminary discrimination results, further data verification and analysis are required. At this time, the system performs secondary data collection to obtain more detailed and accurate data. For example, if the preliminary discrimination results show that the degree of blockage is high but the type is unclear, higher resolution optical images or more detailed gas composition data can be re-collected. After the secondary data is collected, the discrimination model is used for secondary discrimination to generate secondary discrimination result data. Secondary discrimination uses more detailed data to improve the accuracy and reliability of discrimination.

[0131] Finally, the preliminary discrimination result data and the secondary discrimination result data are combined to make the final decision. The final decision uses a machine learning model, such as a decision tree or a random forest model, to analyze the comprehensive data. The decision tree model can handle complex decision-making problems by splitting nodes layer by layer and gradually determining the optimal decision path. The random forest model improves the accuracy and stability of decisions by building multiple decision trees and combining the judgment results of multiple trees. Combining the preliminary discrimination result data and the secondary discrimination result data, these models can comprehensively analyze all relevant factors of the congestion and generate the final cleaning plan data. The final cleaning plan data includes detailed information such as specific cleaning measures, required tools, and expected cleaning time to ensure that the cleaning work can be carried out efficiently and orderly.

[0132] For example, in actual applications, the system initially identified through the rule engine that there was sediment blockage in the pipeline, but the difficulty of cleaning was uncertain. The system collected secondary data to obtain more detailed sediment distribution and density data. The discrimination model performed secondary discrimination based on the secondary data to determine the degree of blockage and the difficulty of cleaning. Subsequently, based on the preliminary and secondary discrimination results, the random forest model made the final decision and generated a cleaning plan, including the use of high-pressure water flow cleaning, and the estimated cleaning time was 2 hours. This process ensured the accuracy and effectiveness of the cleaning plan, avoided unnecessary cleaning work, and improved cleaning efficiency.

[0133] Through this multi-level discrimination and decision-making mechanism, the present invention can accurately assess the blockage situation and formulate an efficient cleaning plan in the complex and changeable tunnel drainage pipe environment. The rapid response of the rule engine, the accurate analysis of secondary data collection and discrimination, and the comprehensive decision-making of the machine learning model together constitute a complete monitoring and processing system to ensure the smooth and safe operation of the tunnel drainage pipe.

[0134] According to the final cleaning plan data, a robotic arm is used to perform the cleaning task, and multimodal sensors are used for real-time monitoring to collect feedback data, optimize the cleaning strategy and discrimination model, and generate the final cleaning feedback data.

[0135] In the present invention, according to the final cleaning plan data, the robot arm is used to perform the cleaning task, and real-time monitoring is performed through multimodal sensors to collect feedback data, optimize the cleaning strategy and discrimination model, and generate the final cleaning feedback data, which is the execution and feedback link of the plan. This link not only ensures the effective execution of the cleaning task, but also continuously optimizes the performance of the entire system through real-time monitoring and feedback mechanism.

[0136] First, according to the final cleaning plan data, the cleaning task is performed using a robotic arm. The final cleaning plan data includes information such as detailed cleaning measures, required tools, operation steps, and expected cleaning time. The robotic arm is a high-precision, multi-degree-of-freedom automated equipment that can operate flexibly in a complex pipeline environment. The robotic arm is equipped with a variety of tools, such as a high-pressure water gun, a mechanical gripper, and a drill, etc., which can perform corresponding cleaning operations according to different blockage situations. The automated operation of the robotic arm not only improves the cleaning efficiency, but also reduces the risk of manual entry into narrow and dangerous environments.

[0137] During the cleaning task, multimodal sensors are used for real-time monitoring. These sensors include optical cameras, infrared imagers, ultrasonic sensors, and flow rate sensors, which can monitor the situation inside the pipeline in real time to ensure that the cleaning task proceeds as planned. The optical camera provides real-time images so that operators can intuitively see the cleaning process; the infrared imager detects temperature changes, monitors the working status of the equipment and thermal anomalies inside the pipeline; the ultrasonic sensor measures the cleaning effect to ensure that the blockage is completely removed; the flow rate sensor monitors the flow of liquid to verify whether the cleaning effect is as expected. The data from these sensors together constitute a comprehensive real-time monitoring system, which provides a guarantee for the smooth progress of the cleaning task.

[0138] During real-time monitoring, the system continuously collects feedback data. This feedback data includes information such as cleaning progress, equipment status, changes in blockages, and the environment inside the pipeline. By analyzing this data, the system can evaluate the execution effect of the cleaning task and adjust the cleaning strategy as needed. For example, if the feedback data shows that the cleaning progress is slow, the system can adjust the operating force of the robot arm or change the cleaning tool; if a new blockage is detected, the system can re-evaluate the situation and formulate a new cleaning plan.

[0139] Feedback data is also used to optimize cleaning strategies and discriminant models. By analyzing the actual data from the cleaning task, the system is able to identify bottlenecks and problems in the cleaning process and improve the cleaning strategy. At the same time, feedback data is also used to optimize the parameters and structure of the discriminant model to make it more accurate and efficient in discriminating future blockages. For example, if feedback data indicates that a certain type of blockage is prone to reoccurrence under certain conditions, the system can adjust the parameters of the discriminant model to enhance the detection and early warning capabilities of these conditions.

[0140] For example, in a specific cleaning task, the final cleaning plan data indicates the use of a robotic arm equipped with a high-pressure water gun to clean up the silt blockage. The robotic arm enters the pipeline and starts cleaning. The cleaning progress is monitored in real time through an optical camera, and the infrared imager detects the working status of the robotic arm and the temperature changes inside the pipeline. The ultrasonic sensor measures the cleaning effect of the silt, and the flow rate sensor monitors the recovery of the liquid flow. During the cleaning process, the system found that the silt was relatively hard and the cleaning progress was slow. Through feedback data analysis, the system adjusted the pressure of the high-pressure water gun and improved the cleaning efficiency. After the task was completed, the system analyzed the feedback data of the cleaning task, optimized the cleaning strategy and discrimination model, and provided more accurate and effective technical support for future cleaning tasks.

[0141] Through this execution and feedback mechanism, the present invention can not only efficiently complete the cleaning task of the tunnel drainage pipe, but also continuously optimize the system performance through real-time monitoring and feedback data, and improve the overall monitoring and processing effect. Real-time monitoring ensures the safety and smooth progress of the cleaning task, and feedback data provides a basis for continuous optimization, enabling the system to maintain an efficient and accurate working state in a constantly changing actual environment.

[0142] like Figure 4 As shown, a system for implementing the tunnel drainage pipe blockage monitoring and processing method comprises:

[0143] A multimodal sensor array, including a high-resolution optical camera, a lidar, an ultrasonic sensor, a gas sensor, an infrared imager, an acoustic imaging sensor, an electromagnetic induction sensor, and a flow rate sensor, for collecting optical image data, three-dimensional point cloud data, ultrasonic reflection data, gas composition data, infrared imaging data, acoustic image data, electromagnetic induction data, and flow rate data inside the drainage pipe;

[0144] The system first collects data through a multimodal sensor array. The multimodal sensor array includes high-resolution optical cameras, lidars, ultrasonic sensors, gas sensors, infrared imagers, acoustic imaging sensors, electromagnetic induction sensors, and flow rate sensors. These sensors are used to collect optical image data, three-dimensional point cloud data, ultrasonic reflection data, gas composition data, infrared imaging data, acoustic image data, electromagnetic induction data, and flow rate data inside the drainage pipe. Through this multimodal data collection method, the system can fully obtain multi-dimensional information inside the pipe, making up for the shortcomings of a single data collection method. For example, optical cameras and infrared imagers can provide visual images of the inside of the pipe, lidars and ultrasonic sensors can capture three-dimensional spatial features and object morphology, and gas sensors and acoustic imaging sensors can detect chemical and acoustic features inside the pipe.

[0145] The data preprocessing module is used to perform denoising and normalization on the collected raw data to generate a preprocessed raw feature data set;

[0146] The collected data will be sent to the data preprocessing module for processing. This module denoises and normalizes the raw data to generate a preprocessed raw feature data set. Denoising removes noise from the data through filtering and other techniques to improve the quality and accuracy of the data. Normalization converts the data to the same scale range to ensure the consistency and comparability of different types of data. For example, Gaussian filtering is used to remove random noise in optical images, and z-score normalization is used to normalize the flow rate data to the same range.

[0147] The multi-scale feature extraction module includes a convolutional neural network, a three-dimensional convolutional neural network, and a long short-term memory network, which is used to process the preprocessed original feature data set, extract multi-scale features, and generate a pooled feature data set;

[0148] The preprocessed data will enter the multi-scale feature extraction module. This module includes convolutional neural network (CNN), three-dimensional convolutional neural network (3D CNN) and long short-term memory network (LSTM), which are used to process the preprocessed original feature data set, extract multi-scale features, and generate pooled feature data sets. Specifically, CNN performs multi-layer convolution and pooling operations on optical images and infrared imaging data to extract spatial features; 3D CNN performs voxelization and three-dimensional convolution on three-dimensional point cloud data to extract three-dimensional spatial features; LSTM performs time series processing on ultrasonic reflection data and gas composition data to extract time-related features. Through this multi-scale feature extraction method, the system can capture deep-level features in the data and improve the richness and accuracy of feature expression.

[0149] The self-attention mechanism module is used to dynamically adjust the weights of each modality feature and generate weighted feature data;

[0150] After feature extraction, the self-attention mechanism module dynamically adjusts the weights of each modal feature to generate weighted feature data. The self-attention mechanism calculates the importance of each modal feature in different tasks and adaptively adjusts their weights to optimize the feature fusion effect. For example, when identifying the type of blockage, the optical image feature may be more important than the gas composition feature, and the self-attention mechanism will give the optical image feature a higher weight.

[0151] The graph neural network module is used to process weighted feature data, extract high-level correlation features, and generate graph feature data;

[0152] The weighted feature data then enters the graph neural network (GNN) module. This module constructs a feature graph, uses the weighted feature data as graph nodes, uses GNN to process the relationship between nodes, extracts high-level correlation features, and generates graph feature data. GNN captures the complex relationships and patterns between multimodal features through graph convolution operations, making the feature data more representative and discriminative.

[0153] The discrimination model module includes a fully connected neural network, which is used to discriminate the type, quantity, size, degree of obstruction and difficulty of cleaning of the obstruction based on the graph feature data, and generate discrimination result data;

[0154] The graph feature data will be transmitted to the discriminant model module. This module includes a fully connected neural network (FCNN), which is used to discriminate the type, quantity, size, degree of blockage and difficulty of cleaning based on the graph feature data, and generate discrimination result data. Specifically, FCNN deeply processes the input feature data through the connection of multiple layers of neurons, and outputs classification and regression results. For example, FCNN can identify the type of blockage as "tree roots", the number as "2 roots", the size as "5 cm in diameter", the degree of blockage as "0.7", and the difficulty of cleaning as "7".

[0155] The real-time data feedback and self-learning mechanism module is used to dynamically and adaptively correct the discrimination model parameters according to the real-time data feedback and historical discrimination results, generate the corrected discrimination result data and optimize the model parameters;

[0156] In order to ensure the accuracy and adaptability of the discrimination results, the system introduces real-time data feedback and self-learning mechanism modules. This module dynamically and adaptively corrects the discrimination model parameters based on real-time data feedback and historical discrimination results, generates corrected discrimination result data and optimizes model parameters. Through the adaptive learning algorithm, the model parameters are adjusted in real time to adapt to the latest pipeline environment and blockage conditions; through the self-learning mechanism and reinforcement learning algorithm, historical data and feedback are analyzed, the model structure and learning strategy are optimized, and the overall performance and robustness of the model are improved.

[0157] The decision module includes a rule engine and a machine learning model, which is used to make preliminary judgments, secondary judgments and final decisions on the correction judgment result data and generate final cleaning solution data;

[0158] The corrected discrimination result data will enter the decision module. This module includes a rule engine and a machine learning model, which are used to perform preliminary discrimination, secondary discrimination and final decision on the corrected discrimination result data and generate the final cleaning solution data. The rule engine quickly classifies and judges the discrimination results based on predefined rules; the machine learning model (such as decision tree and random forest) makes the final decision and generates a detailed cleaning solution by comprehensively analyzing the preliminary and secondary discrimination results.

[0159] A robot arm execution module, used for executing the cleaning task according to the final cleaning plan data;

[0160] According to the final cleaning plan data, the robot arm execution module will perform specific cleaning tasks. The robot arm is equipped with various tools, such as high-pressure water guns, mechanical grippers and drills, which can flexibly operate in the pipeline to complete the cleaning work. The real-time monitoring and feedback module monitors the cleaning task in real time through multimodal sensors to ensure that the task is carried out as planned, and collects feedback data to further optimize the cleaning strategy and discrimination model to generate the final cleaning feedback data.

[0161] The real-time monitoring and feedback module is used to monitor the cleaning task in real time through multimodal sensors, collect feedback data, optimize the cleaning strategy and discrimination model, and generate the final cleaning feedback data.

[0162] The real-time monitoring and feedback module is used to monitor the cleaning task in real time through multimodal sensors, collect feedback data, optimize the cleaning strategy and discrimination model, and generate the final cleaning feedback data. This module is introduced to ensure the accurate execution of the cleaning task and the continuous optimization of the system.

[0163] Real-time monitoring: During the cleaning task, a multimodal sensor array (including high-resolution optical cameras, lidar, ultrasonic sensors, gas sensors, infrared imagers, acoustic imaging sensors, electromagnetic induction sensors, and flow rate sensors) will monitor the cleaning process in real time. These sensors collect data in real time to ensure that the cleaning task proceeds as planned. For example, optical cameras can monitor visual changes in the cleaning area in real time, infrared imagers can detect temperature changes in the working state of the equipment, ultrasonic sensors can measure the progress of cleaning, and flow rate sensors can monitor the restoration of liquid flow. These monitoring data provide a comprehensive perspective on the execution of the cleaning task, ensuring that any abnormalities are discovered and handled in a timely manner.

[0164] Collect feedback data: Various data collected during real-time monitoring are transmitted to the system for analysis. These data are called feedback data. Feedback data include the progress of the cleaning task, the working status of the robot arm, changes in blockages, changes in the environment inside the pipeline, etc. By analyzing these feedback data, the system can evaluate the effectiveness and execution effect of the current cleaning strategy. For example, if the feedback data shows that the cleaning progress is slow or new blockages appear, the system can immediately adjust the cleaning strategy.

[0165] Optimize cleaning strategy: Based on feedback data, the system can optimize the current cleaning strategy. For example, if the robot encounters a hard blockage during the cleaning process, the feedback data will show that the cleaning progress is slow. At this time, the system can adjust the operating parameters of the robot based on the feedback data, such as increasing water pressure or changing cleaning tools to improve cleaning efficiency. Through this real-time adjustment, it ensures that the cleaning task can be completed efficiently and smoothly.

[0166] Optimizing the discriminant model: Feedback data is also used to optimize the discriminant model. The system identifies deficiencies and areas for improvement in the discriminant model by analyzing the actual data in the cleaning task. For example, if the discriminant model has errors in discriminating a certain type of blockage, the system can self-learn and optimize through feedback data, adjusting model parameters or structure to improve the accuracy of future discrimination. The self-learning mechanism and reinforcement learning algorithm play a key role here, continuously optimizing the discriminant model by continuously learning historical data and real-time feedback.

[0167] Generate final cleaning feedback data: Through real-time monitoring and feedback mechanisms, the system finally generates cleaning feedback data. This data not only reflects the execution and effect of the cleaning task, but also contains optimization suggestions for the cleaning strategy and discrimination model. The final cleaning feedback data is used to update the system's knowledge base and optimize the model, making the system more efficient and accurate in future cleaning tasks.

[0168] For example, in a cleaning task, the real-time monitoring module monitored the robotic arm through optical cameras and lidar to clean the mud and sand blockages in the pipe. Real-time feedback data showed that the cleaning progress was slow and the ultrasonic sensor detected that the mud and sand were relatively hard. After analyzing the feedback data, the system adjusted the operating parameters of the robotic arm, increased the water pressure, and replaced the tool that was more suitable for cleaning hard blockages. After the cleaning task was completed, the system generated the final cleaning feedback data, recorded the adjustments and optimization measures during the cleaning process, further optimized the discrimination model and cleaning strategy, and provided valuable experience and data support for similar cleaning tasks in the future.

[0169] The system of the present invention comprehensively collects the internal data of the tunnel drainage pipe through a multimodal sensor array, performs denoising and normalization preprocessing, uses convolutional neural networks, three-dimensional convolutional neural networks and long short-term memory networks to extract multi-scale features, combines self-attention mechanisms and graph neural networks to perform feature fusion and associated feature extraction, uses fully connected neural networks to identify blockages, and performs dynamic correction through real-time data feedback and self-learning mechanisms. Finally, a cleaning plan is generated and the cleaning task is performed by a robotic arm. The real-time monitoring and feedback module ensures the execution of the cleaning task and the continuous optimization of the system, which significantly improves the monitoring and processing efficiency and accuracy of the blockage of the tunnel drainage pipe.

[0170] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0171] The present application is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0172] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0173] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0174] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0175] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0176] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0177] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0178] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A method for monitoring and handling blockage of a tunnel drainage pipe, characterized in that: The following steps are involved: Use a multimodal sensor array to collect optical image data, three-dimensional point cloud data, ultrasonic reflection data, gas composition data, infrared imaging data, acoustic image data, electromagnetic induction data and flow rate data inside the drainage pipe, denoise and normalize the collected raw data to generate a preprocessed raw feature data set; The preprocessed original feature data set is processed by convolutional neural network, three-dimensional convolutional neural network and long short-term memory network to extract multi-scale features and generate a pooled feature data set; Use the self-attention mechanism to dynamically adjust the weights of each modal feature, perform graph neural network processing on the pooled feature data set, extract high-level correlation features, and generate optimized multi-modal comprehensive feature data; Based on the optimization of multi-modal comprehensive feature data, the discriminant model is used to discriminate the type, quantity, size, degree of blockage and difficulty of cleaning of the blockage, and the discriminant result data is generated. The discriminant model parameters are dynamically and adaptively corrected through real-time data feedback and self-learning mechanism to generate corrected discriminant result data; Perform preliminary judgment, secondary judgment and final decision on the correction judgment result data, use the rule engine and machine learning model to evaluate the necessity of cleaning, and generate the final cleaning plan data; According to the final cleaning plan data, a robotic arm is used to perform the cleaning task, and multimodal sensors are used for real-time monitoring to collect feedback data, optimize the cleaning strategy and discrimination model, and generate the final cleaning feedback data.

2. The method according to claim 1, characterized in that The multimodal sensor array includes a high-resolution optical camera for capturing optical image data inside a drainage pipe; a lidar for acquiring three-dimensional point cloud data; an ultrasonic sensor for measuring the distance and shape of objects inside the pipe and generating ultrasonic reflection data; a gas sensor for detecting the gas composition inside the pipe and generating gas composition data; an infrared imager for capturing thermal imaging data inside the pipe; an acoustic wave imaging sensor for generating acoustic wave image data; an electromagnetic induction sensor for detecting metals and conductive materials and generating electromagnetic induction data; and a flow rate sensor for measuring the liquid flow rate inside the drainage pipe and generating flow rate data.

3. The method according to claim 1, characterized in that The preprocessing includes denoising the collected raw data, specifically filtering and denoising the optical image data, three-dimensional point cloud data, ultrasonic reflection data, gas composition data, infrared imaging data, acoustic image data, electromagnetic induction data and flow rate data to generate a clean data set; and normalizing the denoised data to ensure the consistency and comparability of the data to generate a preprocessed raw feature data set.

4. The method according to claim 1, characterized in that The multi-scale feature extraction comprises: The convolutional neural network performs multi-layer convolution and pooling operations on the preprocessed optical image feature data and infrared imaging feature data, extracts spatial features, and generates pooled optical feature data and pooled infrared feature data; The 3D convolutional neural network performs voxel processing on the preprocessed 3D point cloud feature data, converts the 3D point data into voxel data, and performs multi-layer 3D convolution and pooling operations to extract 3D spatial features and generate pooled 3D point cloud feature data; The long short-term memory network performs time series processing on the preprocessed ultrasonic reflection characteristic data and gas composition characteristic data, extracts time-related features, and generates time-characteristic ultrasonic data and time-characteristic gas data.

5. The method according to claim 4, characterized in that The self-attention mechanism dynamically adjusts the weight of each modal feature, specifically generating weighted feature data according to the importance of each modal feature in different tasks, including weighted pooled optical feature data, weighted pooled infrared feature data, weighted pooled three-dimensional point cloud feature data, weighted time feature ultrasonic data, weighted time feature gas data, weighted acoustic image feature data, weighted electromagnetic induction feature data and weighted flow velocity feature data.

6. The method according to claim 5, characterized in that The graph neural network processes the weighted feature data, specifically including: Construct a feature graph, and use weighted pooled optical feature data, weighted pooled infrared feature data, weighted pooled three-dimensional point cloud feature data, weighted time feature ultrasonic data, weighted time feature gas data, weighted acoustic wave image feature data, weighted electromagnetic induction feature data, and weighted flow velocity feature data as graph nodes; Use graph neural networks to extract high-level correlation features between graph nodes and generate graph feature data.

7. The method according to claim 1, characterized in that The discriminant model performs a comprehensive analysis on the optimized multi-modal comprehensive feature data, specifically including: Use a fully connected neural network to perform multi-layer fully connected processing on the optimized multi-modal comprehensive feature data to generate fully connected feature data; Based on the fully connected feature data, the types, quantities, and sizes of blockages are classified, and regression analysis of the degree of blockage and difficulty of cleaning is performed to generate discrimination result data.

8. The method according to claim 1, characterized in that The dynamic adaptive correction of the discriminant model parameters through real-time data feedback and self-learning mechanism specifically includes: Based on real-time data feedback, an adaptive learning algorithm is used to adjust the parameters of the discrimination model to generate corrected discrimination result data; Through the self-learning mechanism, based on historical judgment results and real-time data feedback, the reinforcement learning algorithm is used to optimize the model structure and learning strategy to generate optimized model parameters.

9. The method according to claim 1, characterized in that: The preliminary judgment, secondary judgment and final decision making of the correction judgment result data specifically include: Use the rule engine to make a preliminary judgment on the correction judgment result data to generate preliminary judgment result data; Conduct secondary data collection and analysis based on the preliminary discrimination result data, use the discrimination model to conduct secondary discrimination, and generate secondary discrimination result data; The preliminary discrimination result data and the secondary discrimination result data are integrated, and a decision tree or random forest model is used to make the final decision to generate the final cleaning solution data.

10. A system for implementing the tunnel drainage pipe blockage monitoring and processing method according to any one of claims 1 to 9, characterized in that: include: A multimodal sensor array, including a high-resolution optical camera, a lidar, an ultrasonic sensor, a gas sensor, an infrared imager, an acoustic imaging sensor, an electromagnetic induction sensor, and a flow rate sensor, for collecting optical image data, three-dimensional point cloud data, ultrasonic reflection data, gas composition data, infrared imaging data, acoustic image data, electromagnetic induction data, and flow rate data inside the drainage pipe; The data preprocessing module is used to perform denoising and normalization on the collected raw data to generate a preprocessed raw feature data set; The multi-scale feature extraction module includes a convolutional neural network, a three-dimensional convolutional neural network, and a long short-term memory network, which is used to process the preprocessed original feature data set, extract multi-scale features, and generate a pooled feature data set; The self-attention mechanism module is used to dynamically adjust the weights of each modality feature and generate weighted feature data; The graph neural network module is used to process weighted feature data, extract high-level correlation features, and generate graph feature data; The discrimination model module includes a fully connected neural network, which is used to discriminate the type, quantity, size, degree of obstruction and difficulty of cleaning of the obstruction based on the graph feature data, and generate discrimination result data; The real-time data feedback and self-learning mechanism module is used to dynamically and adaptively correct the discrimination model parameters according to the real-time data feedback and historical discrimination results, generate the corrected discrimination result data and optimize the model parameters; The decision module includes a rule engine and a machine learning model, which is used to make preliminary judgments, secondary judgments and final decisions on the correction judgment result data and generate final cleaning solution data; A robot arm execution module, used for executing the cleaning task according to the final cleaning plan data; The real-time monitoring and feedback module is used to monitor the cleaning task in real time through multimodal sensors, collect feedback data, optimize the cleaning strategy and discrimination model, and generate the final cleaning feedback data.

Images