Pipe network defect diagnosis method and system based on resnet and lstm fusion

Abstract

The application discloses a pipe network defect diagnosis method and system based on ResNet and LSTM fusion, and belongs to the technical field of urban drainage system monitoring. The method comprises the following steps: collecting pipe inner wall images, water quality parameters, hydraulic parameters and pipe environment data through multiple types of sensors; after image data preprocessing, spatial features are extracted by using ResNet, after time series data preprocessing, time series dependency is extracted by using LSTM, and joint training and optimization are carried out; an attention mechanism is used to weight and fuse image features and time series features, and a classifier is input to output defect types and grades; according to the comparison between the diagnosis result and a preset safety threshold, grades are divided, early warning is triggered, maintenance work orders are generated, and disposal feedback data is received for model updating. The application fuses multiple source heterogeneous data, improves the accuracy and comprehensiveness of defect diagnosis, realizes intelligent closed-loop management from detection to disposal, and can be widely applied to the health monitoring and intelligent operation and maintenance of urban drainage pipe networks.

Description

Technical Field

[0001] This invention relates to the field of urban drainage system monitoring technology, specifically to a method and system for diagnosing pipeline defects based on the fusion of ResNet and LSTM. Background Technology

[0002] With the acceleration of urbanization, urban drainage systems, as crucial infrastructure ensuring the normal operation of cities, directly impact the city's flood control and drainage capabilities, as well as its environmental protection level. The introduction of smart monitoring technology is of paramount strategic importance for improving the management efficiency of drainage systems, reducing operation and maintenance costs, and preventing urban flooding. By integrating the Internet of Things (IoT) and big data analytics, comprehensive perception and dynamic control of the urban water cycle system can be achieved, ensuring the unimpeded flow of this vital urban lifeline under extreme weather conditions. Within urban drainage systems, pipe networks are the core channels for transporting and discharging water. Long-term operation is susceptible to corrosion, geological subsidence, and human factors, making them prone to defects such as rupture, deformation, and blockage. Failure to detect and repair these defects in a timely manner can lead to serious accidents such as ground subsidence and water pollution. Therefore, as a sub-field, the improvement of pipe network defect diagnosis technology has extremely high practical value for ensuring the healthy operation of drainage pipe networks throughout their entire lifecycle.

[0003] Existing methods for diagnosing pipeline defects mainly rely on manual inspection or single computer vision technology. Manual inspection is inefficient and poses safety hazards, while single technical means often only focus on image recognition of the pipeline's appearance, ignoring dynamic environmental data such as hydraulics and water quality during pipeline operation. This leads to a high false alarm rate under complex lighting or shading conditions, and it is difficult to determine the actual impact of defects on pipeline function. This non-intelligent detection mode cannot meet the needs of real-time processing of massive pipeline network data and lacks the ability to predict defect development trends. As a result, maintenance work is often in a reactive emergency repair state, making it difficult to achieve proactive preventive maintenance. This seriously restricts the improvement of the refined management level of urban drainage systems and also causes a large consumption of human and material resources. Summary of the Invention

[0004] The purpose of this invention is to provide a pipeline defect diagnosis method and system based on the fusion of ResNet and LSTM, in order to solve the technical problems of existing pipeline defect diagnosis methods that rely too much on a single image data source, lack comprehensive consideration of hydraulic, water quality and pipeline operating environment time series data, cannot accurately identify pipeline functional defects, and are difficult to achieve intelligent closed-loop management and prediction based on the degree of defect.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A pipeline defect diagnosis method based on the fusion of ResNet and LSTM is characterized by the following steps: S1. Data Acquisition: Real-time acquisition of visual image data of the inner wall of the pipeline, as well as water quality, hydraulic and pipeline environment data during pipeline operation; S2. Model Construction and Training: The collected image data and structured data are preprocessed separately. A deep convolutional neural network model based on ResNet is constructed to extract spatial features of the image. A long short-term memory network model based on LSTM is constructed to extract time series dependencies. The two models are jointly trained and their parameters are optimized using labeled sample data. S3. Model Fusion and Diagnosis: A feature fusion strategy is adopted to deeply fuse the image feature vector output by the ResNet model and the temporal feature vector output by the LSTM model at the feature level. The weights of different features are dynamically adjusted through the attention mechanism. The fused feature vector is input into the classifier to output the defect diagnosis result. S4. Closed-loop control: Automatically matches the corresponding operation and maintenance handling strategy based on the output defect diagnosis results, receives on-site handling feedback data and enters the results into the database for model retraining or parameter updates.

[0007] Furthermore, in step S1: Collect video stream data inside the pipeline, extract the video stream frame by frame and filter key frames to obtain image data reflecting the condition of the pipeline's inner wall; Collect at least one of the following water quality data for the liquid inside the pipeline: chemical oxygen demand, ammonia nitrogen content, pH, and suspended solids concentration; Collect at least one of the following hydraulic data: instantaneous flow rate of liquid in the pipeline, liquid velocity, and liquid level height change over time; Collect at least one of the following: ambient temperature and humidity data inside the pipeline, and gas pressure data inside the pipeline.

[0008] Further, step S2 includes: S21: Perform denoising, image enhancement and standardization transformation on the acquired image data, and perform missing value imputation, outlier removal and normalization on the water quality, hydraulic and environmental data to make the input data meet the model input requirements; S22: The cross-entropy loss function is used to calculate the error between the model's predicted value and the true label. The Adam optimizer is used to iteratively update the weight parameters of the ResNet model and the LSTM model through the backpropagation algorithm until the loss function converges to the preset range.

[0009] Further, step S3 includes: S31. Perform dimension unification processing on the image feature vector output by the ResNet model and the temporal feature vector output by the LSTM model, and perform standardization operation on the two feature vectors to eliminate the difference in dimensions and achieve feature dimension alignment. S32. An attention mechanism is introduced to calculate the importance weights of image features and temporal features in the defect determination task. The two types of features are weighted and fused to generate a fused feature vector containing rich multidimensional information. S33. Input the generated fusion feature vector into the fully connected layer for mapping transformation, and then use the Softmax classifier to calculate the probability distribution of each category, and output the specific type and severity level of the pipeline defect.

[0010] Furthermore, the feature weighting process of model fusion is expressed as:

[0011]

[0012]

[0013] in: This represents the image feature vector after dimension alignment, with dimensions of... ; This represents the temporal feature vector after dimension alignment, with dimension 1. ; Represents the concatenated vector; This represents the weight matrix in attention calculation; This represents the bias term in attention calculation; This represents the intermediate state vector after processing by the activation function. This represents the weight matrix used to generate attention weights; This represents the bias term used to generate attention weights; Represents the normalized exponential function; This represents the generated attention weight coefficient vector, with values ​​ranging from 0 to 1. This represents element-wise multiplication; This represents the fused feature vector of the final output.

[0014] Further, step S4 includes: S41. Analyze the defect type, location coordinates, and severity level data output by the fusion model to generate a diagnostic conclusion including a confidence score; S42. Compare the defect severity index with the preset safety threshold, classify the defect into several levels based on the comparison results, and set differentiated response logic. S43. When the defect level is determined to be at or above the set level, the system automatically pushes real-time early warning information containing defect location images and risk assessments to the operation and maintenance personnel through the management platform, and generates a standardized electronic diagnostic report for archiving. S44. The system automatically generates maintenance work orders based on defect level and geographical location information, intelligently assigns the work orders to the corresponding maintenance teams, receives feedback information on on-site handling and re-inspection data after repair, and updates the historical database.

[0015] Furthermore, in step S42, the preset security threshold system includes three levels of thresholds. , , Defects are classified into four levels: normal, attention, warning, and emergency. Defect scoring calculation formula:

[0016] in: Representing the Input values ​​for defect characteristic indicators, such as crack width or deformation rate; Representing the Preset weight coefficients corresponding to each feature indicator; Represents the total number of feature indicators involved in the calculation; This represents the calculated comprehensive defect score. Defect level determination rules:

[0017] in: This represents the level of defect determined based on the scoring. Represents the first-level threshold. Represents the second-level threshold. Represents the third-level threshold. This represents the normal level. Represents the level of attention. Represents the warning level. This indicates the level of emergency.

[0018] A pipeline defect diagnosis system based on the fusion of ResNet and LSTM includes: The data acquisition unit is used to collect visual image information of the inner wall of the pipeline in real time, as well as water quality parameters, hydraulic parameters and pipeline environmental data during pipeline operation. The model building and training unit is used to preprocess the collected image data and structured data respectively, build a deep convolutional neural network model based on ResNet to extract image spatial features, build a long short-term memory network model based on LSTM to extract time series dependencies, and use labeled sample data to jointly train and optimize the parameters of the two models. The feature fusion and diagnosis unit is used to perform deep fusion of the image feature vector output by the ResNet model and the temporal feature vector output by the LSTM model at the feature level using a feature fusion strategy. The weights of different features are dynamically adjusted through an attention mechanism. The fused feature vector is then input into the classifier to output the defect diagnosis result. The closed-loop control unit is used to automatically match the corresponding operation and maintenance handling strategy based on the defect diagnosis results output by the model, and to receive on-site handling feedback data and record the results into the database for model retraining or parameter updates.

[0019] Furthermore, the data acquisition unit includes an image acquisition module for acquiring image data of the inner wall of the pipe; The water quality acquisition module is used to collect at least one of the following water quality data: chemical oxygen demand, ammonia nitrogen content, pH, and suspended solids concentration. The hydraulic data acquisition module is used to acquire at least one hydraulic data point from the following: instantaneous flow rate, liquid velocity, and liquid level height change over time in the pipeline. An environmental data acquisition module is used to acquire at least one of the following: ambient temperature and humidity data inside the pipeline, and gas pressure data inside the pipeline.

[0020] Furthermore, in the feature fusion and diagnosis unit, the feature fusion process is represented as follows:

[0021]

[0022]

[0023] in: This represents the image feature vector after dimension alignment, with dimensions of... ; This represents the temporal feature vector after dimension alignment, with dimension 1. ; Represents the concatenated vector; This represents the weight matrix in attention calculation; This represents the bias term in attention calculation; This represents the intermediate state vector after processing by the activation function. This represents the weight matrix used to generate attention weights; This represents the bias term used to generate attention weights; Represents the normalized exponential function; This represents the generated attention weight coefficient vector, with values ​​ranging from 0 to 1. This represents element-wise multiplication; This represents the fused feature vector of the final output.

[0024] The beneficial effects of this invention are: This invention extracts pipeline image features using a ResNet model and extracts temporal features of water quality, hydraulic, and environmental parameters using an LSTM model. Feature fusion processing is performed at the feature layer, and a weighted fusion feature vector is generated using an attention mechanism. This enables the model to simultaneously process pipeline visual information and operational status information, outputting diagnostic results that include defect type and level. Based on the diagnostic results, the system performs threshold comparison, graded early warning, report generation, and work order dispatch, and records the handling feedback data into the database. This forms a closed-loop processing flow of data acquisition, model diagnosis, and operation and maintenance control, realizing joint analysis of multi-source heterogeneous data and automated operation and maintenance management. By introducing temporal data features to assist image recognition, the model's defect recognition capability under complex operating conditions is improved, ensuring the comprehensiveness, accuracy, and real-time nature of pipeline network defect diagnosis. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating the pipeline defect diagnosis method based on the fusion of ResNet and LSTM of the present invention. Figure 2 This is a schematic diagram of the module structure of the pipeline defect diagnosis system based on the fusion of ResNet and LSTM of the present invention; Detailed Implementation The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention are within the scope of protection of the present invention.

[0026] Embodiments of the present invention: like Figure 1 and 2 As shown, a pipeline defect diagnosis method based on the fusion of ResNet and LSTM includes the following steps: S1: Data Acquisition A multi-source heterogeneous sensor network is used to monitor the urban drainage network in an all-round way. Intelligent sensing devices deployed inside the pipeline and around the nodes simultaneously collect visual images of the pipeline inner wall, water quality chemical composition, water flow dynamic parameters, and physical quantities of the surrounding environment. The collected data is aggregated to the data center in real time using the Internet of Things transmission protocol to ensure the temporal synchronization and spatial consistency of data collection. A multi-dimensional raw database covering spatial and temporal features is constructed, and the raw data is subjected to integrity verification, preliminary cleaning and format conversion to provide high-quality data support for the subsequent training and inference of deep learning models.

[0027] Step S1 specifically includes the following: S11: A CCTV inspection robot equipped with a high-definition camera moves inside the pipeline to collect video stream data. The video stream is extracted frame by frame and key frames are filtered to obtain high-resolution image data reflecting the condition of the pipeline's inner wall.

[0028] Image data indicators include the morphology, width, and length of cracks in the pipe's inner wall, corrosion area, corrosion depth, pipe deformation rate, interface misalignment, branch pipe concealed connections, sediment thickness, sediment distribution, root intrusion, root intrusion depth, obstacle type, obstacle size, and characteristics of pipe attachments. Image data acquisition utilizes a CCTV inspection robot equipped with a high-brightness LED supplementary lighting system and an optically stabilized lens. The image acquisition resolution is set to 1920x1080 pixels, with an acquisition frequency of 25 frames per second. Video data is compressed using H.264 encoding to save storage space. The system automatically extracts a key image frame every 5 seconds, stores it in JPEG format on a network-attached storage server, and establishes a dual-index database based on timestamps and pipe segment locations for rapid retrieval and recall. The equipment also features automatic ranging to record the pipe ring number corresponding to each image frame.

[0029] S12: By deploying online water quality monitoring sensors at key nodes in the pipeline network, real-time data on water quality parameters such as chemical oxygen demand, ammonia nitrogen content, pH, and suspended solids concentration of the liquid inside the pipeline are collected. Data acquisition is performed using an online water quality analyzer with integrated multi-probes. The sensors employ electrochemical and optical fluorescence detection principles, and the acquisition frequency is set to once per minute. Data is transmitted to the edge computing gateway via RS485 bus through the Modbus protocol and stored in the time-series database in real time. The system automatically performs zero-point calibration and range calibration daily and records calibration parameters to correct measurement deviations. It is also equipped with an ultrasonic automatic cleaning device to prevent probe surface contamination and ensure measurement stability during long-term operation.

[0030] S13: Using flow meters and level gauges installed at specific locations in the pipeline, the instantaneous flow rate, flow velocity, and hydraulic data of the liquid level change over time are collected in real time.

[0031] It adopts an external clamp-on ultrasonic Doppler flow meter and a high-precision submersible hydrostatic level gauge, which are installed non-contactly at the top or bottom of the pipeline. The acquisition frequency is set to once per second. It uses the Doppler frequency shift effect to calculate the fluid velocity, uses the pressure principle to measure the liquid level height, and calculates the instantaneous flow rate and cumulative flow rate through an integral algorithm. The data is uploaded to the cloud server in real time via a 4G wireless network and outlier filtering is performed. The device also has a power failure storage function to prevent data loss.

[0032] S14: Collect ambient temperature and relative humidity data inside the pipeline using temperature and humidity sensors, and simultaneously collect gas pressure data inside the pipeline using pressure sensors to obtain pipeline environmental data. The pipeline environmental data indicators include ambient temperature, relative humidity, and gas pressure. Data acquisition is achieved using a data acquisition device that integrates temperature and humidity sensors and gas pressure sensors, installed inside the pipeline inspection well. The acquisition frequency is once per minute. The data is transmitted to the aggregation node via a LoRaWAN low-power wireless network. The system has a built-in temperature compensation algorithm to correct sensor drift. Long-term monitoring data is used to assess the pipeline operating environment status. At the same time, the equipment protection level reaches IP68 to adapt to the humid environment underground.

[0033] S2: Model Building Preprocessing operations are performed on the collected image data and structured data respectively. A deep convolutional neural network model based on the ResNet architecture is constructed to extract spatial features of the image. The output of LSTM is used as a temporal feature encoding, which is fused with the image features before classification. The model parameters are trained and optimized through the backpropagation algorithm.

[0034] Preprocessing: Denoising, image enhancement, and standardization transformations are performed on the acquired image data. Missing value imputation, outlier removal, and normalization are performed on the water quality, hydraulic, and environmental data to ensure that the input data meets the model input requirements.

[0035] Step S2 specifically includes the following steps: S21: Visual Model Construction An image feature extraction model based on ResNet (Deep Residual Network) is constructed. This model includes multiple convolutional layers, batch normalization layers, modified linear unit activation function layers, and max pooling layers. The input is a pipe inner wall image that has been resized and normalized, and the output is a fixed-dimensional image feature vector. The texture and geometric features of the image are extracted through convolution operations. The gradient vanishing problem in deep network training is solved by using a residual skip connection structure. The Adam optimizer and cross-entropy loss function are used to iteratively update the model parameters until the loss function converges to a preset range.

[0036] The feature extraction process of the image data model is represented by the following mathematical formula:

[0037]

[0038]

[0039] in: The tensor representing the input image of the inner wall of the pipe, after preprocessing, has a size of [size missing]. ; This represents the image feature vector output by the ResNet network, with dimensions of... ; The mapping function representing the residual network contains multiple convolutional and residual connection operations; Representing the Output feature maps of each residual block; Representing the The weight parameter matrix of each convolutional layer; Representing the Bias terms for each convolutional layer; This represents a non-linear activation function; here, the ReLU function is used. Represents the residual function, which typically contains two... Convolution operation; Represents the th after residual connection Output feature map of the layer.

[0040] S22: Structured Model Construction A structured data model based on LSTM (Long Short-Term Memory) network is constructed. Water quality, hydraulic, and environmental data are aligned according to timestamps and a multi-dimensional time series input vector is constructed. This vector is then input into the input layer of the LSTM network. The cell state update and information flow are controlled by three gating units: forget gate, input gate, and output gate. This effectively extracts long-term dependencies and periodic change patterns from the time series data. The output is a time series feature vector. The model parameters are optimized using the cross-entropy classification loss function to capture the dynamic change characteristics of the pipeline network operation status.

[0041] The process of extracting time-series features from structured data models is represented by the following mathematical formula:

[0042]

[0043]

[0044]

[0045]

[0046]

[0047] in: represent The structured data vector input at any time contains water quality, hydraulic and environmental parameters; represent The hidden layer state output at each time step; represent The hidden layer state output at each time step, i.e., the extracted temporal features; represent The cell state at any given moment is used to store long-term memory information; represent The output vector of the forget gate determines which information is discarded. represent The output vector of the input gate at each time step determines which new information is updated. represent Vector values ​​of candidate cell states at each time step; represent The output vector of the output gate at each time step determines which hidden state information is output; These represent the weight matrices for the forget gate, input gate, cell state, and output gate, respectively. These represent the corresponding bias term parameters; Represents the Sigmoid activation function; Represents the hyperbolic tangent activation function; This represents the element-wise multiplication operation.

[0048] S3: Model Fusion and Diagnosis A feature fusion strategy is adopted to deeply integrate the image feature vectors extracted by the ResNet model and the temporal feature vectors extracted by the LSTM model. The heterogeneous features are mapped to a unified feature space through a fully connected layer. The correlation weights of image features and temporal features in defect determination are dynamically calculated using a multi-head attention mechanism. The feature channels are adaptively weighted to generate the final fused feature vector. This vector integrates pipeline appearance defect information and operating environment status information, providing a more comprehensive and accurate decision basis for subsequent defect classification.

[0049] Step S3 specifically includes the following steps: S31: Input the image feature vector output by the ResNet network and the temporal feature vector output by the LSTM network into the feature alignment module. Use a fully connected layer to map the dimensions of the two feature vectors to the same preset dimension, and perform L2 normalization on the feature vectors to eliminate numerical differences caused by different data units. This ensures that the image features and temporal features are consistent in numerical scale, laying the foundation for subsequent fusion operations.

[0050] S32: The dimension-aligned image feature vector and temporal feature vector are input into the attention fusion layer. The correlation matrix between the two feature vectors is calculated, and the attention weight distribution is obtained through the Softmax function. This weight is used to weight the temporal features, highlighting environmental factors highly correlated with the current image defects, suppressing noise interference, and achieving adaptive deep fusion of image features and temporal features. The weighted fused features are then output. For example, when a pipe ruptures, the image feature weight increases; when there is siltation and blockage, the hydraulic temporal feature weight increases.

[0051] S33: The fused feature vector is input into the fully connected classification layer. After nonlinear transformation by the multilayer perceptron, the predicted probability of each category is finally output through the Softmax activation function. The category with the highest probability is the diagnosis result. The categories include defect types such as normal pipeline, rupture, deformation, corrosion, and blockage. At the same time, the corresponding confidence score is output to evaluate the certainty of the model for the current diagnosis result.

[0052] The feature weighting process of model fusion is represented by the following mathematical formula:

[0053]

[0054]

[0055] in: This represents the image feature vector after dimension alignment, with dimensions of... ; This represents the temporal feature vector after dimension alignment, with dimension 1. ; A concatenated vector representing image features and temporal features; This represents the weight matrix of the intermediate layer in attention calculation; This represents the bias term in the intermediate layer of attention calculation; This represents the intermediate state vector after processing by the activation function. This represents the weight matrix used to generate attention weights; This represents the bias term used to generate attention weights; This represents the normalization exponential function, used to generate the weight distribution; This represents the generated attention weight coefficient vector, with values ​​ranging from 0 to 1; This represents the element-wise multiplication operation. The fused feature vector representing the final output combines visual and temporal information.

[0056] S4: Closed-loop control Based on the defect diagnosis results output by the model, the system automatically executes a closed-loop control strategy. The system compares the diagnosis results with a preset threshold system at multiple levels and triggers corresponding response mechanisms according to the severity and type of the defect. For minor defects, records are automatically generated and archived in the database. For serious defects, an early warning process is immediately initiated. Alarm information is sent to maintenance personnel through the management platform and maintenance orders are automatically generated. At the same time, the system receives on-site handling feedback data and re-inspection results to update model parameters, thereby realizing intelligent and automated management of pipeline network operation and maintenance.

[0057] Step S4 specifically includes the following steps: S41: The system receives the defect type and confidence score output by the fusion model, combines the pipeline's geographical location information and historical defect records, comprehensively analyzes the impact of the current defect on the hydraulic transmission function of the pipeline network, predicts the development trend of the defect, generates a detailed diagnostic analysis report including defect location coordinates, defect category, risk level assessment and recommended treatment measures, and stores the report in the pipeline health record database.

[0058] S42: The system has a multi-level safety threshold system. It compares the severity index of defects in the diagnostic analysis report with the threshold in real time. Based on the comparison results, the pipeline status is divided into four levels: normal, attention, warning, and emergency. Different response time limits and handling strategies are set for different levels. For defects that reach the emergency level, the highest level alarm signal is directly triggered to ensure that critical risks are dealt with in a timely manner.

[0059] S43: When the defect level reaches the concern level or above, the system sends an immediate warning notification to the operation and maintenance management personnel via SMS, email or mobile application push. The notification includes on-site image screenshots of the defect, a description of the specific location, a risk level description, and preliminary handling suggestions. At the same time, a standardized diagnostic report PDF file is automatically generated and archived together with relevant data in the pipeline network life cycle management system for future reference.

[0060] S44: The operation and maintenance management system intelligently assigns the nearest suitable maintenance team based on the geographical location information of the defect and the real-time distribution of the current maintenance team. It automatically generates a repair work order containing task details and safety tips. After the maintenance personnel complete the on-site handling, they upload before-and-after comparison photos and text descriptions via mobile terminal. The system records the feedback data into the historical database for optimizing subsequent diagnostic models and control strategies.

[0061] The formulas used in the closed-loop control process are as follows:

[0062]

[0063] in: Representing the Input values ​​for defect characteristic indicators, such as crack width or deformation rate; Representing the Preset weight coefficients corresponding to each feature indicator; Represents the total number of feature indicators involved in the calculation; This represents the calculated comprehensive defect score. This represents the level of defect determined based on the scoring. This represents the first-level threshold, used to distinguish between normal and attention status; This represents the second-level threshold, used to distinguish between the attention and warning status; This represents the third-level threshold, used to distinguish between a warning and an emergency. This represents the normal level, indicating that the pipeline network is operating well; This indicates a level of concern, meaning there are minor defects that require regular inspection. This indicates a warning level, meaning there is a significant defect that needs to be repaired as soon as possible; This indicates an emergency level, signifying a serious defect that requires immediate attention.

[0064] It should be noted that, unless otherwise specified, the embodiments and features described in this invention can be combined with each other.

[0065] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of this invention is in use, or the orientation or positional relationship commonly understood by those skilled in the art. They are only used for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention. In addition, the terms "first," "second," etc., are only used to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0066] In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0067] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A pipeline defect diagnosis method based on the fusion of ResNet and LSTM, characterized in that, Includes the following steps: S1. Data Acquisition: Real-time acquisition of visual image data of the inner wall of the pipeline, as well as water quality, hydraulic and pipeline environment data during pipeline operation; S2. Model Building and Training: The collected image data and structured data are preprocessed separately to build a deep convolutional neural network model based on ResNet for extracting image spatial features. We construct a long short-term memory network model based on LSTM to extract time series dependencies, and use labeled sample data to jointly train and optimize the parameters of the two models. S3. Model Fusion and Diagnosis: A feature fusion strategy is adopted to deeply fuse the image feature vector output by the ResNet model and the temporal feature vector output by the LSTM model at the feature level. The weights of different features are dynamically adjusted through the attention mechanism. The fused feature vector is input into the classifier to output the defect diagnosis result. S4. Closed-loop control: Automatically matches the corresponding operation and maintenance handling strategy based on the output defect diagnosis results, receives on-site handling feedback data and enters the results into the database for model retraining or parameter updates.

2. The pipeline defect diagnosis method based on the fusion of ResNet and LSTM according to claim 1, characterized in that, In step S1: Collect video stream data inside the pipeline, extract the video stream frame by frame and filter key frames to obtain image data reflecting the condition of the pipeline's inner wall; Collect at least one of the following water quality data for the liquid inside the pipeline: chemical oxygen demand, ammonia nitrogen content, pH, and suspended solids concentration; Collect at least one of the following hydraulic data: instantaneous flow rate of liquid in the pipeline, liquid velocity, and liquid level height change over time; Collect at least one of the following: ambient temperature and humidity data inside the pipeline, and gas pressure data inside the pipeline.

3. The pipeline defect diagnosis method based on the fusion of ResNet and LSTM according to claim 2, characterized in that, Step S2 includes: S21: Perform denoising, image enhancement and standardization transformation on the acquired image data, and perform missing value imputation, outlier removal and normalization on the water quality, hydraulic and environmental data to make the input data meet the model input requirements; S22: The cross-entropy loss function is used to calculate the error between the model's predicted value and the true label. The Adam optimizer is used to iteratively update the weight parameters of the ResNet model and the LSTM model through the backpropagation algorithm until the loss function converges to the preset range.

4. The pipeline defect diagnosis method based on the fusion of ResNet and LSTM according to claim 1, characterized in that, Step S3 includes: S31. Perform dimension unification processing on the image feature vector output by the ResNet model and the temporal feature vector output by the LSTM model, and perform standardization operation on the two feature vectors to eliminate the difference in dimensions and achieve feature dimension alignment. S32. An attention mechanism is introduced to calculate the importance weights of image features and temporal features in the defect determination task. The two types of features are weighted and fused to generate a fused feature vector containing rich multidimensional information. S33. Input the generated fusion feature vector into the fully connected layer for mapping transformation, and then use the Softmax classifier to calculate the probability distribution of each category, and output the specific type and severity level of the pipeline defect.

5. The pipeline defect diagnosis method based on the fusion of ResNet and LSTM according to claim 4, characterized in that: The feature weighting process of model fusion is represented as: in: This represents the image feature vector after dimension alignment, with dimensions of... ; This represents the temporal feature vector after dimension alignment, with dimension 1. ; Represents the concatenated vector; This represents the weight matrix in attention calculation; This represents the bias term in attention calculation; This represents the intermediate state vector after processing by the activation function. This represents the weight matrix used to generate attention weights; This represents the bias term used to generate attention weights; Represents the normalized exponential function; This represents the generated attention weight coefficient vector, with values ​​ranging from 0 to 1. This represents element-wise multiplication; This represents the fused feature vector of the final output.

6. The pipeline defect diagnosis method based on the fusion of ResNet and LSTM according to claim 1, characterized in that, Step S4 includes: S41. Analyze the defect type, location coordinates, and severity level data output by the fusion model to generate a diagnostic conclusion including a confidence score; S42. Compare the defect severity index with the preset safety threshold, classify the defect into several levels based on the comparison results, and set differentiated response logic. S43. When the defect level is determined to be at or above the set level, the system automatically pushes real-time early warning information containing defect location images and risk assessments to the operation and maintenance personnel through the management platform, and generates a standardized electronic diagnostic report for archiving. S44. The system automatically generates maintenance work orders based on defect level and geographical location information, intelligently assigns the work orders to the corresponding maintenance teams, receives feedback information on on-site handling and re-inspection data after repair, and updates the historical database.

7. The pipeline defect diagnosis method based on the fusion of ResNet and LSTM according to claim 6, characterized in that: In S42, the preset security threshold system includes three levels of thresholds. , , Defects are classified into four levels: normal, attention, warning, and emergency. Defect scoring calculation formula: in: Representing the Input values ​​for defect characteristic indicators, such as crack width or deformation rate; Representing the Preset weight coefficients corresponding to each feature indicator; Represents the total number of feature indicators involved in the calculation; This represents the calculated comprehensive defect score. Defect level determination rules: in: This represents the level of defect determined based on the scoring. Represents the first-level threshold. Represents the second-level threshold. Represents the third-level threshold. This represents the normal level. Represents the level of attention. Represents the warning level. This indicates the level of emergency.

8. A pipeline defect diagnosis system based on the fusion of ResNet and LSTM, characterized in that, include: The data acquisition unit is used to collect visual image information of the inner wall of the pipeline in real time, as well as water quality parameters, hydraulic parameters and pipeline environmental data during pipeline operation. The model building and training unit is used to preprocess the collected image data and structured data respectively, build a deep convolutional neural network model based on ResNet to extract image spatial features, build a long short-term memory network model based on LSTM to extract time series dependencies, and use labeled sample data to jointly train and optimize the parameters of the two models. The feature fusion and diagnosis unit is used to perform deep fusion of the image feature vector output by the ResNet model and the temporal feature vector output by the LSTM model at the feature level using a feature fusion strategy. The weights of different features are dynamically adjusted through an attention mechanism. The fused feature vector is then input into the classifier to output the defect diagnosis result. The closed-loop control unit is used to automatically match the corresponding operation and maintenance handling strategy based on the defect diagnosis results output by the model, and to receive on-site handling feedback data and record the results into the database for model retraining or parameter updates.

9. The pipeline defect diagnosis system based on the fusion of ResNet and LSTM according to claim 8, characterized in that: The data acquisition unit includes an image acquisition module for acquiring image data of the inner wall of the pipe; The water quality acquisition module is used to collect at least one of the following water quality data: chemical oxygen demand, ammonia nitrogen content, pH, and suspended solids concentration. The hydraulic data acquisition module is used to acquire at least one hydraulic data point from the following: instantaneous flow rate, liquid velocity, and liquid level height change over time in the pipeline. An environmental data acquisition module is used to acquire at least one of the following: ambient temperature and humidity data inside the pipeline, and gas pressure data inside the pipeline.

10. The pipeline defect diagnosis system based on the fusion of ResNet and LSTM according to claim 8, characterized in that: In the feature fusion and diagnosis unit, the feature fusion process is represented as follows: in: This represents the image feature vector after dimension alignment, with dimensions of... ; This represents the temporal feature vector after dimension alignment, with dimension 1. ; Represents the concatenated vector; This represents the weight matrix in attention calculation; This represents the bias term in attention calculation; This represents the intermediate state vector after processing by the activation function. This represents the weight matrix used to generate attention weights; This represents the bias term used to generate attention weights; Represents the normalized exponential function; This represents the generated attention weight coefficient vector, with values ​​ranging from 0 to 1. This represents element-wise multiplication; This represents the fused feature vector of the final output.

Images