Water conservancy project defect identification method and system based on deep learning

Abstract

The invention discloses a water conservancy project defect identification method and system based on deep learning, and relates to the technical field of water conservancy project surveying and mapping, and the method comprises the steps: obtaining a multi-temporal remote sensing image of a water conservancy project facility, and then extracting a change pattern spot as a to-be-identified region based on a time sequence change enhanced image; according to the change detection, stable areas which do not change are screened out, the recognition range is concentrated at positions where problems possibly exist, and local texture changes, material surface information and macrostructure features are obtained by constructing a deep learning model comprising a convolutional feature coding layer, a space structure extraction layer and a cross-scale fusion layer. And training the model by using labels of different defect types, and learning a discrimination mode of local texture change and overall structure deformation by adopting cross entropy loss and structural consistency constraint optimization. According to the method, through feature fusion of multi-scale features, the model can sense microcracks and macrostructure anomalies at the same time.

Description

A Deep Learning-Based Method and System for Defect Identification in Hydraulic Engineering Technical Field

[0001] This invention relates to the field of water conservancy engineering surveying technology, specifically a method and system for identifying defects in water conservancy projects based on deep learning. Background Technology

[0002] In the existing surveying and monitoring of water conservancy facilities, remote sensing data is mainly used for macroscopic inspection or manual interpretation of water conservancy facilities. It usually relies on single-phase images or visual comparison to detect anomalies. This method is difficult to quantify and analyze the subtle changes in the engineering facilities over time, which makes it easy for minor changes such as weathering of the structural surface and local cracks to be masked by the overall image background and difficult to identify accurately.

[0003] Furthermore, existing water conservancy surveying methods lack systematic processing of multi-temporal remote sensing images, and their ability to analyze the spatial geometry and cross-scale structural characteristics of water conservancy engineering facilities is insufficient. This makes it difficult to distinguish subtle differences between different types of defects during the identification stage. This further makes it difficult to effectively screen out significant anomaly areas related to water conservancy engineering facilities from surveying data in the context of large-scale surface changes, thus limiting the accuracy and reliability of defect identification. Summary of the Invention

[0004] (I) Technical problem to be solved This invention provides a method and system for identifying defects in water conservancy projects based on deep learning, which can classify defects by utilizing the fusion features of the spatial geometry and structure of water conservancy facilities.

[0005] (II) Technical Solution To achieve the above objectives, the present invention provides the following technical solution: A method for identifying defects in water conservancy projects based on deep learning, comprising the following steps: constructing a time series of remote sensing images of the same area in a target water conservancy project facility; extracting a pixel change intensity distribution map using correlation analysis and expectation-maximization algorithms; combining threshold segmentation and morphological processing to extract areas from the pixel change intensity distribution map that have a continuous spatial distribution of change intensity and exceed a set threshold as the area to be identified; performing scale normalization processing on the area to be identified; and then organizing the extracted multispectral features, texture features, and structural attributes of the engineering facility in parallel according to the channel dimension to generate multi-channel input samples; constructing a multi-branch convolutional deep learning model for defect identification of the input samples, wherein the convolutional feature encoding layer in the deep learning model uses small-scale convolution kernels for the multispectral and texture channels in the input samples. Multi-layer convolutional operations are performed to encode local features such as surface reflection differences, crack textures, and local erosion traces. A spatial structure extraction layer uses dilated convolutions and spatial aggregation operations with different dilation rates on the structural attribute channels of the input samples to capture the overall structural anomalies of the engineering facilities. A cross-scale fusion layer spatially aligns the output local features with the overall structural anomalies to generate unified fusion features. A classification layer outputs defect type classifications of the fusion features. During the training phase of the deep learning model, corresponding defect type labels are constructed for each region sample to be identified. Cross-entropy loss is used to supervise the classification output, while structural consistency constraints are introduced to optimize the synergistic relationship between local texture changes and overall structural deformation in the fusion features. The multi-channel input samples corresponding to the region are input into the trained deep learning model, which generates fusion features internally and outputs defect type evaluation results through the classification layer.

[0006] In some feasible embodiments, for the same spatial area of ​​the target water conservancy project, remote sensing image data collected at different time points are acquired, and the spatial position of each remote sensing image is aligned with a unified geographic coordinate system through geometric correction to construct a time series of remote sensing images for the area.

[0007] In some feasible embodiments, pixel-level time-series spectral analysis is performed on each phase of the remote sensing image in the time series of the remote sensing image. Correlation analysis is used to measure the spectral consistency of each pixel in the time dimension, thereby distinguishing stable pixels from potentially changing pixels. Then, the expectation-maximization algorithm is used to iteratively solve the probability of the change state of each pixel to obtain the change intensity value of each pixel, forming a pixel change intensity distribution map.

[0008] In some feasible embodiments, after obtaining the pixel change intensity distribution map, the Otsu segmentation algorithm is applied to analyze the histogram of the pixel change intensity distribution, and a threshold corresponding to the maximum inter-class variance is selected. Pixels with change intensity higher than the threshold are marked as candidate change pixels, and a set of candidate change pixels is selected. Connectivity analysis is performed on the candidate change pixel set to combine spatially adjacent pixels into continuous regions, and a unique label is assigned to each continuous region to form a preliminary set of regions to be identified. Then, morphological filtering is performed on each independent region in the set of regions to be identified to remove isolated noise smaller than a set area threshold, generating the regions to be identified for the classification and identification of defects in engineering facilities.

[0009] In some feasible embodiments, the deep learning model employs a multi-branch convolutional structure. The input samples include standardized multispectral channels, texture channels, and structural attribute channels, with each type of feature being input into the deep learning model as a separate branch. The convolutional feature encoding layer consists of multiple convolutional units stacked sequentially, each of which includes a small-scale convolutional kernel for nonlinear activation and feature normalization operations. The small-scale convolutional kernels perform sliding convolution operations on the inputs corresponding to the multispectral and texture channels within their local neighborhoods, progressively extracting local features such as material surface reflection differences, crack texture orientation, and local erosion traces. The spatial structure extraction layer uses dilated convolutions with different dilation rates to process the inputs of the structural attribute channels in parallel. The structural features obtained under different dilation rates are integrated through spatial aggregation operations to form overall structural anomaly features reflecting the construction of engineering facilities. The cross-scale fusion layer integrates the local features and overall structural anomaly features, performs spatial size alignment processing, maps them to a unified spatial resolution, and weights and fuses them according to the contribution of each branch feature in defect discrimination, forming a fused feature that simultaneously includes local texture changes and overall structural anomaly information.

[0010] In some feasible embodiments, during the training phase of the deep learning model, multi-channel input samples that have undergone scale normalization and channel alignment are paired with their corresponding engineering defect type labels to construct a sample set for model training. During training, the input samples are sequentially input into the convolutional feature encoding layer and the spatial structure extraction layer to extract the local features and overall structural anomaly features of the region to be identified, respectively. Then, the local features and overall structural anomaly features from different branches are input into the cross-scale fusion layer for spatial alignment and fusion to obtain the fused features used for defect discrimination.

[0011] The classification layer in the deep learning model maps fused features to predict various engineering defects. During backpropagation, the cross-entropy loss function is used to measure the difference between the predicted results and the corresponding defect type labels, and the difference is used to iteratively optimize the parameters of each layer in the deep learning model.

[0012] In some feasible embodiments, during the parameter update process of the deep learning model, the cross-entropy loss between the predicted result of the defect category and the corresponding label is used as the main supervision signal. At the same time, structural consistency constraints are introduced. By constraining the response distribution generated by the local feature branches and the overall structural anomaly feature branches in the fusion features, the local texture change features in the same region to be identified and their corresponding structural features maintain a consistent response relationship in space. The structural consistency constraint term and the cross-entropy loss are jointly backpropagated to update the parameters of the deep learning model.

[0013] In some feasible embodiments, after the deep learning model is trained, the input samples corresponding to the acquired region are input into the deep learning model. The convolutional feature encoding layer performs forward computation on the input samples, extracts and outputs local features characterizing the spectral changes and crack textures of the material surface. The spatial structure extraction layer extracts and outputs overall structural anomaly features characterizing the region based on the input samples and their spatial context information. The local feature representations and overall structural anomaly features are aligned with the cross-scale fusion layer according to spatial correspondence, and a unified fusion feature vector is formed by channel splicing. The fusion feature vector is then input into the classification layer, where it is mapped to the predicted response values ​​of each defect category through the parameters learned during the training phase, and the corresponding category probability distribution is obtained after normalization. The deep learning model determines the defect category corresponding to the region based on the category probability distribution and outputs the predicted probability of each defect category.

[0014] A deep learning-based system for identifying defects in water conservancy projects includes: a data monitoring module configured to: construct a time series of remote sensing images of the same area within a target water conservancy project facility, and extract a pixel intensity distribution map using correlation analysis and expectation-maximization algorithms; a pre-identification module configured to: extract areas with continuously distributed spatial intensity distribution exceeding a set threshold from the pixel intensity distribution map using threshold segmentation and morphological processing as regions to be identified; after scale normalization of the regions to be identified, organize the extracted multispectral features, texture features, and structural attributes of the engineering facility in parallel according to channel dimensions to generate multi-channel input samples; and a model building module configured to: construct a multi-branch convolutional deep learning model for defect identification of the input samples, wherein the convolutional feature encoding layer in the deep learning model uses small-scale convolutional kernels to perform multi-layer convolution on the multispectral and texture channels of the input samples. Convolutional operations encode local features such as surface reflection differences, crack textures, and local erosion traces. A spatial structure extraction layer uses dilated convolutions and spatial aggregation operations with different dilation rates on the structural attribute channels of the input samples to capture the overall structural anomalies of the engineering facilities. A cross-scale fusion layer spatially aligns the output local features with the overall structural anomalies to generate unified fusion features. A classification layer classifies the fusion features to reflect defect types. During the training phase of the deep learning model, a corresponding defect type label is constructed for each region sample to be identified. Cross-entropy loss is used to supervise the classification output, while structural consistency constraints are introduced to optimize the synergistic relationship between local texture changes and overall structural deformation in the fusion features. The defect classification module is configured to input multi-channel input samples corresponding to the region into the trained deep learning model, generate fusion features internally, and output defect type evaluation results through the classification layer.

[0015] (III) Beneficial effects: Compared with the prior art, the invention has the following beneficial effects: By introducing a region screening mechanism based on time-series change constraints, the invention transforms defect identification from indiscriminate analysis of the entire region to targeted processing of regions that have undergone significant changes, thereby effectively narrowing the analysis range in the initial stage of identification and reducing the interference of stable regions on subsequent feature extraction and classification judgment.

[0016] During the defect identification process, by modeling local convolutional features and spatial structural features separately and fusing them at a unified scale, the defect identification process no longer relies solely on a single texture or local abnormal response. Instead, it simultaneously utilizes surface change information within the region and spatial structural differences at the component level, thereby enhancing the ability to distinguish complex engineering defect morphologies.

[0017] By using fusion features for unified classification and discrimination, the same model can judge engineering defects of different scales and forms within a consistent feature space, avoiding the instability of recognition caused by differences in defect manifestations, thereby improving the overall consistency and reliability of defect recognition results. Attached Figure Description

[0018] Figure 1 is a flowchart illustrating a deep learning-based method for identifying defects in water conservancy projects according to an embodiment of the present invention; Figure 2 is a schematic diagram illustrating the data processing logic of each process stage in the deep learning-based method for identifying defects in water conservancy projects according to an embodiment of the present invention; Figure 3 is a block diagram illustrating the principle of a deep learning-based system for identifying defects in water conservancy projects according to an embodiment of the present invention; Figure 4 is a schematic diagram illustrating the data processing logic of the constructed deep learning model in the deep learning-based method for identifying defects in water conservancy projects according to an embodiment of the present invention; In the figures: 100, data monitoring module; 200, pre-identification module; 300, model construction module; 400, defect classification module. Detailed Implementation

[0019] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0020] Furthermore, the terms "first" and "second" are used only to distinguish descriptions and should not be interpreted as indicating or implying relative importance.

[0021] It should be noted that, where there is no conflict, the features in the embodiments of the present invention can be combined with each other.

[0022] This paper presents a deep learning-based method and system for identifying defects in hydraulic engineering projects, as shown in Figures 1 to 4. After identifying and classifying specific engineering defects, corresponding data collection and preprocessing are required. Referring to Figure 1, this involves step S10: constructing a time series of remote sensing images of the same area within the target hydraulic engineering facility, and extracting pixel change intensity distribution maps using correlation analysis and expectation-maximization algorithms. The purpose of this step is to provide consistent and reliable spatiotemporal foundational data for any subsequent time-series-based change analysis and defect identification. This step is also implemented by the data monitoring module 100 in the system.

[0023] By systematically acquiring multi-temporal remote sensing images and performing geometric alignment, radiometric quantization, and noise suppression on each temporal phase, taking the use of UAVs for remote sensing image data acquisition as an example, artifacts introduced by imaging conditions such as sensor differences, flight attitude, solar altitude, and atmospheric conditions can be reduced to a controllable range. This ensures that the differences in pixel values ​​of the same surface pixel at different time points mainly reflect the real changes of the ground object or engineering structure itself, rather than observation errors or sensor noise.

[0024] Once this data consistency is achieved, subsequent change detection only needs to focus on the true signals, reducing the root causes of false alarms and false negatives, while providing a reliable coordinate basis for spatial integration with engineering structure information and terrain data.

[0025] More specifically, data collection planning should be carried out during the project initiation phase. In the embodiments of this invention, the monitoring object can be clearly defined first, namely the spatial range of the water conservancy engineering facilities, the expected monitoring frequency (e.g., seasonal, monthly, or re-monitoring after an emergency), the required spatial resolution and spectral band, such as visible / near-infrared multispectral, high-resolution visible light, thermal infrared, or SAR / lidar as needed.

[0026] The platforms for acquiring and monitoring remote sensing images include, but are not limited to, satellite imagery and drone platforms. In some embodiments of this invention, drones equipped with multispectral or high-resolution cameras or lidar are used to acquire localized remote sensing images suitable for high resolution and flexible scheduling. Furthermore, aerial photography of a specific area under specific weather conditions can also serve as an intermediate solution.

[0027] It is important to note that when planning, the imaging time period should be determined, which means ensuring that the shooting scene is within the observation window with a similar sun angle and no or few clouds, as well as the required overlap rate and resolution for each acquisition.

[0028] During the data transmission and initial management phase, the collected remote sensing data, or additional metadata, needs to be completely transferred to the project storage. In some embodiments of this invention, the acquired raw imagery, GNSS / INS tracks, GCP coordinates, environmental records, etc., should be archived together according to a unified naming convention, timestamp, and coordinate reference, with proper backup and version control. A preliminary quality check should also be performed at this step, such as checking for severe blurring, overexposure / underexposure, cloud obstruction, or abnormal tracks, thus recording and determining whether a reflight / reacquisition is necessary.

[0029] When entering the preprocessing stage, geometric correction and registration are performed first. Taking remote sensing image data acquired by UAV as an example, the exterior orientation elements of the image are calculated based on the camera intrinsic parameters, attitude data and GCP configured by the UAV, and orthorectification and mosaicking are performed to ensure that each image or mosaic image has a comparable planar position in a unified geographic coordinate system, so as to ensure that the pixels are aligned with the actual position on the ground.

[0030] Radiometric and atmospheric corrections are then performed to convert the digital (DN) or grayscale values ​​of the image into physical reflectance or radiance. Radiometric correction includes, but is not limited to, sensor dark current / zero-point correction, gain correction, and the application of calibration coefficients. Atmospheric correction can employ relative calibration methods, such as relative radiometric normalization based on temporal images of the same region, or absolute atmospheric correction methods, with correction parameters recorded for traceability. For multi-temporal sequences, a consistent correction procedure and parameters are typically used to avoid radiometric baseline drift between different images.

[0031] During the data preprocessing stage, it is also important to incorporate engineering structural information and topographic data into the spatial reference system. The engineering structural information comes from design drawings, BIM models, on-site surveys, or historical inspection records. Topographic data may be existing DEMs, aerial lidar point clouds, or differential measurement results. These vector or grid data need to be converted to the same coordinate reference system as the remote sensing image, and projection transformations or raster resampling should be performed as needed to ensure a one-to-one correspondence between structural boundaries and image pixels.

[0032] Finally, data acceptance and storage are crucial. All preprocessed images and auxiliary data should be accompanied by metadata information, specifically recording data such as acquisition time, sensor parameters, calibration parameters, and quality assessment indicators. It is important to note that only qualified data can proceed to subsequent time-series analysis and change detection stages; unqualified data must have its issues recorded, and a decision made regarding whether to re-acquire the data or use it as backup data must be made.

[0033] After acquiring and preprocessing multi-temporal remote sensing images, a systematic temporal analysis of these images obtained at different times is required. However, considering that structural defects in water conservancy facilities often have the characteristics of accumulation or abrupt change over time, such as the gradual deepening of local settlement, the spread of degradation of slope protection surface materials, and the extension of dam cracks, etc.

[0034] Such changes are often difficult to reliably identify in images from a single time phase, but in a time series, subtle differences between images can form more stable trend information. Therefore, before formally extracting change patches, it is necessary to enhance the time series images to magnify the areas that truly represent structural changes in the time dimension, while suppressing non-engineering changes caused by background, lighting differences, seasonal noise, etc.

[0035] Based on the above, it can be understood that before extracting the changed area (the area to be identified), this step first performs two processing steps on the time-series remote sensing image: calculating the change intensity and enhancing its saliency. This operation has two main functions: first, it separates the real change signal from observation noise, seasonality, or imaging differences, providing a continuous and quantifiable measure (change intensity) for subsequent patch extraction, rather than directly using the binary judgment that is easily affected by noise, as in the original differential image.

[0036] Furthermore, based on the generated continuous change saliency image, its pixel values ​​can directly reflect the strength of the response difference over time, which makes it easy to use statistical methods (such as GMM / EM) to model the distribution of changed and unchanged pixels, and then use an automatic thresholding method (that is, the Otsu algorithm mentioned above) to separate the regions of significant change.

[0037] This approach can more robustly identify candidate regions that are truly worthy of subsequent classification, reducing false positives and false negatives.

[0038] First, remote sensing images of different time phases in the same spatial region are organized into a sequence according to time order, and correlation analysis is performed on the changes in spectral values ​​at the same pixel location in the sequence.

[0039] Based on the above explanation, correlation analysis uses the common patterns of pixel values ​​over time to determine whether a region belongs to a stable area. If the spectral response of a pixel is highly consistent across all time phases, its correlation coefficient is close to 1, indicating that the location is stable over a long period. Conversely, if a region exhibits abnormal changes in one or more time phases, such as a sudden increase in brightness or texture breaks, the correlation will decrease significantly. In this way, a preliminary distribution of the significance of changes across the entire sequence can be obtained. For example, in continuous images, areas where a dam gradually brightens will show a persistently low correlation.

[0040] Considering that correlation analysis can only provide a preliminary estimate of the trend of change and it is difficult to directly and quantitatively describe different types and magnitudes of change, in some embodiments of the present invention, it is necessary to further utilize the expectation-maximization algorithm (EM) for pixel state classification.

[0041] The Expectation-Maximization (EM) algorithm treats the behavior of each pixel in a remote sensing image at different time points as several potential states. For example, the various behaviors can be initially divided into different levels of potential states such as "no change", "slight change", and "significant change", and the probability of the pixel in these states is solved iteratively.

[0042] Specifically, as shown in Figure 2, the Expectation-Maximization (EM) algorithm first initializes the distribution parameters of each state using preliminary features derived from correlation analysis as observation data. Then, it alternately executes the expectation and maximization steps, gradually converging to a distribution model that can accurately distinguish different change patterns. Ultimately, the EM algorithm outputs a continuous distribution map representing the intensity of change, where the more significant the change in a pixel throughout the time series, the higher its intensity value; if the overall situation remains stable, the intensity value approaches the background level.

[0043] After obtaining this distribution of intensity changes, it is necessary to further transform it into an image with a clearer structure, which is easier for subsequent extraction of change patches.

[0044] Therefore, in some embodiments, the obtained change intensity map is specifically used with the Otsu segmentation algorithm mentioned above. The Otsu algorithm automatically determines the optimal segmentation threshold by maximizing the inter-class variance, thus optimally distinguishing the change region from the background region. In areas with high change intensity values, Otsu segmentation will classify them into the "significant change" category, while areas with consistent temporal behavior will be classified into the "background" category.

[0045] Therefore, it can be understood that the result of applying Otsu's algorithm is a variation feature enhancement image, in which variation areas are highlighted, such as subsidence locations that may form bright patches, while structurally stable areas remain at a lower brightness.

[0046] In summary, the entire S10 step can be broken down into three steps: first, acquiring the processed time-series image; second, obtaining the preliminary trend of change based on correlation analysis; third, using the expectation-maximization algorithm to obtain a more accurate expression of the intensity of change; and finally, using Otsu segmentation to form a structured enhanced image.

[0047] Through this continuous processing, engineering changes that were previously difficult to detect in multi-temporal images are made explicit, laying the foundation for subsequent stable and accurate extraction of change patches.

[0048] After obtaining the enhanced change image, it is necessary to further identify the truly engineering-significant change regions. This is because the enhanced change image only amplifies brightness changes over time, but still contains noise from changes in lighting, environment, or vegetation. Therefore, spatial analysis and morphological processing are needed to extract continuous and morphologically sound change regions.

[0049] Specifically, the process begins with S20: combining threshold segmentation and morphological processing, regions with continuously distributed spatial intensity variations exceeding a set threshold are extracted from the pixel intensity variation distribution map as the regions to be identified. After obtaining the regions to be identified using the variation patterns, S30 is performed: after scale normalization of the regions to be identified, the extracted multispectral features, texture features, and structural attributes of engineering facilities are combined and organized in parallel according to channel dimensions to generate multi-channel input samples. This step is implemented by the pre-identification module 200 in the system.

[0050] These two steps can be understood as first separating these significantly enhanced change areas from the overall change feature enhancement image, so that the locations where engineering facilities have changed can be extracted individually. The direct effect of this is to remove most of the unchanged background unrelated to engineering defects from the vast remote sensing imagery, limiting the scope that the subsequent deep learning model needs to process to truly change-informed local areas. This avoids wasting computational resources on a large amount of invalid background and improves the targeting and stability of subsequent feature learning.

[0051] Specifically, the first step is to perform threshold segmentation on the enhanced image. This is because the brightness of the pixel values ​​in the enhanced image can already reflect the degree of change over time. Therefore, a global or adaptive threshold can be selected based on the pixel grayscale distribution to separate the areas with obvious changes from the background area.

[0052] For example, when the intensity of change in a certain area exceeds a certain threshold, that area is classified as a change category; otherwise, it belongs to the background. The segmented binary image may still contain isolated noise points, so connected component analysis is needed to spatially aggregate the segmented bright areas. By checking the adjacency relationship between pixels, pixels belonging to the same change area are merged into independent connected patches, while isolated points containing only a few pixels can be identified as noise and removed.

[0053] After obtaining the connected regions, morphological filtering is needed to further clean the boundaries. This is because, due to residual noise, lighting differences, or instrument noise in remote sensing images, the edges of segmented change regions often appear jagged or have holes. Therefore, morphological operations such as dilation, erosion, and opening can smooth the region outlines and fill internal voids, making the change patches more consistent with the spatial morphology of actual engineering structures.

[0054] The final output of the change patches will be marked as regions to be identified, representing the locations of structural changes that need to be examined in the subsequent classification stages.

[0055] After separating the regions of change, it is necessary to extract the representational features from these regions that can be effectively utilized by the subsequent machine learning model.

[0056] In the embodiments of the present invention, considering that the defects of water conservancy engineering facilities often manifest simultaneously as changes in the spectral properties of the material surface, disturbances in the texture structure, and local spatial deformation, it is necessary to construct features from multiple perspectives so that the subsequent deep learning model can comprehensively interpret the engineering structural information behind these changes.

[0057] Specifically, the first step is to crop the changed patches so that the corresponding local window of each changed area in the original image is extracted, and then standardize them by uniformly cropping the areas to a fixed size so that subsequent features can be processed in the same model structure.

[0058] Subsequently, various features are extracted from the cropped area, including but not limited to spectral features, texture structure features, and spatial geometric features. In addition, structural attributes from engineering data need to be incorporated, including the structural type, material category, or design form of the facility at that location. These attributes are often not directly reflected in actual remote sensing imagery, but they play a decisive role in identifying the nature of the defects.

[0059] To facilitate understanding of the relationships and functions among these features, please refer to Table 1 below.

[0060] Table 1. Various features extracted from the changed patches In order for these multi-source features to be used as inputs to the subsequent model, it is necessary to perform numerical normalization on each type of feature to ensure that features of different dimensions and ranges have a consistent scale in the model.

[0061] After standardization, these features need to be precisely aligned with the original remote sensing image fragments according to their spatial location to ensure that when the model receives image information and structural attribute information simultaneously, they correspond to the same spatial region. After alignment, spectral information, texture structure (and geometric features), and engineering construction attributes are encapsulated into structured input samples, which serve as the basic data units for the entire deep learning recognition stage.

[0062] After constructing the input samples, a deep learning model needs to be built based on the characteristics of these samples to simultaneously understand spectral information, texture details, and structural morphological features. Since defects in hydraulic engineering facilities often exhibit multi-scale and multi-source variations—for example, cracks are fine-scale linear texture anomalies, subsidence is a large-scale geometric contour migration, and material weathering manifests as a slow shift in spectral reflectance—a single type of network structure cannot cover all feature types.

[0063] Next, in this embodiment of the invention, a deep learning model is used for defect type classification and identification. Regarding the model architecture, please refer to Figures 1 and 4, specifically S40: A multi-branch convolutional deep learning model is constructed to identify defects in the input samples. The convolutional feature encoding layer in the deep learning model uses small-scale convolution kernels to perform multi-layer convolution operations on the multispectral and texture channels in the input samples, encoding local features such as material surface reflection differences, crack textures, and local erosion traces; the spatial structure extraction layer uses dilated convolution and spatial aggregation operations with different dilation rates on the structural attribute channels in the input samples to capture the overall structural anomaly features of the engineering facilities; the cross-scale fusion layer spatially aligns the output local features with the overall structural anomaly features to generate a unified fusion feature; and the classification layer outputs the defect type classification based on the fusion feature.

[0064] Before explaining the specific architecture of each layer, it is necessary to understand why the model is designed as a multi-branch and cross-scale fusion structure. The core purpose is to enable the model to recognize both micro-textures and macro-structures.

[0065] Considering the defects of the water conservancy projects applied in the embodiments of the present invention, they usually exhibit obvious scale differences: cracks and micro-erosion are fine-scale signals at the pixel level or several pixels wide, while settlement, block displacement, and slip are macroscopic deformations spanning tens to hundreds of pixels; in addition, spectral information (reflection in different bands) provides physical clues such as material / humidity, and geometric / structural properties provide engineering semantic constraints.

[0066] Learning these signals by placing them on the same branch often leads to mutual interference. Specifically, small convolutional kernels are better at capturing details but have a small receptive field, while large receptive fields tend to smooth out details. Therefore, a multi-branch approach is used, focusing on different types of features, and then merging semantic information at different levels through stacking operations and convolutional layers. This way, fine-scale branches retain minute textures and edge information, while structural branches expand the receptive field to perceive the overall shape. The fused result can identify both cracks and subsidence. Different branches focus on different physical phenomena, facilitating the tracing of judgment criteria from feature maps; for example, the determination of a certain defect is mainly driven by the response of the structural branch.

[0067] First, the input samples for the deep learning model consist of standardized spectral distribution, texture structure, and structural attribute features. These features are spatially aligned with the image region to form joint input data, ensuring that the model can learn material surface and engineering structure information in the same spatial coordinates.

[0068] The input data is first fed into two parallel branches: a convolutional feature encoding layer and a spatial structure extraction layer. As shown in Figure 4, it is important to note that these two processing layers process the input data in parallel. The two branches extract fine-scale and large-scale information respectively during parallel operation without interfering with each other, ensuring that the model is sensitive to both local texture and overall structure.

[0069] The convolutional feature encoding layer extracts local features from the input sample through multiple stacked operations of small convolutional kernels, capturing microscopic information such as differences in surface reflection, crack textures, and local erosion traces. Each convolutional layer is combined with batch normalization and nonlinear activation functions to ensure stable features and rich expression.

[0070] Meanwhile, the spatial structure extraction layer employs dilated / atrous convolution and spatial aggregation strategies to expand the receptive field and capture macroscopic structural features at the engineering component level, such as settlement, block displacement, and overall deformation patterns. Dilated / atrous convolution, also known as dilated convolution or expanding convolution in neural network operations, skips sampling points at fixed intervals within the kernel sampling locations of ordinary convolution, allowing the kernel to see a larger range without significantly increasing the number of parameters or computational load. The use of dilated convolution is based on the fact that defects in hydraulic engineering exhibit two distinctly different scale characteristics. For example, in remote sensing images, a crack may appear as a narrow, dark groove, with texture differences varying only within a few pixels; simultaneously, some weathered areas may exhibit slight but stable brightness differences in the combination of visible and near-infrared bands.

[0071] If ordinary convolutional kernels can only see local variations, they will miss cross-regional deformations and overall shape changes of the component. Dilated convolutions, however, can allow the model to see a larger range without increasing the number of parameters or the size of the convolutional kernel. Convolutional layers, by stacking multiple convolutional kernels, enable the model to search for patterns in local space at different directions and frequencies, such as striped textures, patchy noise dissipation, and changes in material surface roughness. As the number of layers increases, these local encodings are gradually compressed into multi-layered feature vectors representing materials and textures from fine to abstract.

[0072] Considering that structural deformations of water conservancy facilities cannot all be observed in local pixels, such as the overall downward movement of the slope protection plate, block slippage on the embankment, and non-uniform deflection of the gate body edge, these phenomena require the model to have a larger receptive field.

[0073] Therefore, in this spatial structure extraction layer, the effective receptive field of the convolution kernel is first extended to a larger spatial region, enabling the model to observe a more complete structural morphology. For example, for a 128×128 cropped region, the convolutional layer may only be able to observe local textures of 3×3 or 5×5 pixel size, while the spatial structure extraction layer can expand the receptive field to 30×30 or even larger range through convolution, so that the model can see the geometric contour of the entire region.

[0074] In this way, the spatial structure extraction layer can capture the hierarchical relationship of the engineering structure from the image, such as whether the edge of a slope protection exhibits overall outward expansion, whether there is stepped displacement between slabs, or whether there is a regional brightness change gradient caused by settlement.

[0075] After the convolutional feature encoding layer and the spatial structure extraction layer generate features of different scales and semantic depths, the cross-scale fusion layer in the model architecture integrates these features in a unified manner so that fine-scale textures and large-scale structures can be utilized simultaneously in the final classification stage.

[0076] Based on the examples above, suppose some landslide precursor areas simultaneously possess characteristics of fine cracks (small scale) and overall block displacement (large scale). If the model only considers one of these characteristics, the identification will be incomplete. Therefore, this cross-scale fusion layer combines the high-resolution details of the early convolutional layers with the low-resolution global semantics of the mid-to-late structure extraction layers through multi-scale feature stacking, skip connections, or weighted fusion. This allows the deep model to understand the region to be identified in a hierarchical structure. For example, crack textures may be detected in the detail layer, while settlement patterns are extracted in the structure layer; the cross-scale fusion layer superimposes these two features, giving the model both a "magnifying glass perspective" and a "top-down perspective."

[0077] Subsequently, features from both branches are input into a cross-scale fusion layer of the deep learning model for integration. This layer first spatially aligns the features, upsampling structural branch features or downsampling local branch features to a uniform scale, and then synthesizes local texture and macroscopic structural information into a unified feature representation through channel concatenation and weighted fusion.

[0078] In some embodiments, the fusion layer may also incorporate an attention mechanism, which assigns higher weights to important features to further enhance the defect detection capability.

[0079] The final fused features are mapped to each defect category through a fully connected classification layer, outputting the defect type and corresponding confidence score for each region to be identified. For easier understanding of the processing and relationship logic of each architectural layer, please refer to Table 2 below.

[0080] Table 2 Processing Layers of Deep Learning Model Architecture Overall, this multi-branch parallel (convolutional feature encoding layer and spatial structure extraction layer are processed in parallel) and cross-scale fusion model architecture can take into account the feature representation of different scales in a single model. It not only preserves local signals such as fine cracks, but also captures macroscopic deformations such as settlement and slippage, providing a comprehensive ability to identify defects in water conservancy projects. At the same time, it has good debuggability and interpretability.

[0081] The training process of the model is specifically as follows: In the training phase of the deep learning model, a corresponding defect type label is constructed for each sample of the region to be identified. Cross-entropy loss is used to supervise the classification output. At the same time, structural consistency constraints are introduced to optimize the synergistic relationship between local texture changes and overall structural deformation in the fused features. The above two steps are implemented by the model building module 300 in the system.

[0082] The training of the model is also aimed at the identification of defects in water conservancy projects. Defects usually have multi-scale characteristics: local cracks and erosion are micro-texture level defects, while settlement and block displacement are macro-structural level defects.

[0083] Therefore, it is understandable that if the training model relies on only a single supervision signal (such as cross-entropy classification), the model is prone to focusing only on obvious texture or surface features and ignoring overall structural information.

[0084] Therefore, in the embodiments of the present invention, for deep learning models used for classification, introducing structural consistency constraints during the training process can enable the model to learn local texture features while maintaining sensitivity to the macroscopic structure of engineering components, so that it can learn a discrimination pattern that combines cross-scale, global and local aspects.

[0085] First, based on existing defect annotation data, each region to be identified is labeled with a corresponding defect type, such as cracks, erosion, and subsidence. These labels provide supervisory signals to guide the model in learning the typical features of different defects. A cross-entropy loss is then applied to measure the difference between the model's predicted class distribution and the true class labels, prompting the model to output a probability distribution that more closely approximates the true defect types.

[0086] Introducing structural consistency constraints during training is typically achieved through additional loss terms, with the goal of maintaining the stability of structural information in the fused feature maps. For example, spatial smoothing or shape consistency constraints can be applied to the model's output in adjacent pixels or regions, preserving the continuity or symmetry of macroscopic structures and thus helping the model capture large-scale deformation patterns.

[0087] Finally, the cross-entropy loss and structural consistency constraint are combined into a total loss function for end-to-end training. In this way, the model optimizes classification accuracy while preserving structural information when updating parameters, achieving joint discrimination of local texture and overall structure.

[0088] Now let's understand how the loss definition set above is applied to the workflow of a deep learning model.

[0089] Data training phase: Input samples: cropped and standardized features of the region to be identified (spectral, texture, structural attributes), labels: corresponding defect types.

[0090] Forward propagation stage: Data is input into the deep learning model, and after passing through the convolutional feature encoding layer, the spatial structure extraction layer, and the cross-scale fusion layer, a fused feature representation is obtained.

[0091] Loss calculation stage: Cross-entropy loss is used to calculate the difference between the predicted class probability and the true class label; structural consistency constraint loss is used to evaluate the spatial consistency of fused features or output predictions, such as keeping the output smooth in local regions or constraining macroscopic deformation patterns.

[0092] Backpropagation to update model parameters stage: Using the total loss function, the network parameters are updated through the gradient descent algorithm, enabling the model to learn both texture changes and overall structural features simultaneously.

[0093] Iterative training (repeat the above steps until the total loss converges or the preset number of training rounds is reached).

[0094] Based on the above, it can be understood that because the model learns both local texture changes and overall structural deformation patterns during training, it enhances its ability to distinguish different types of defects. The joint optimization of cross-entropy and structural constraints enables the entire model to learn iteratively within the architecture, without relying on manual feature extraction or step-by-step processing.

[0095] Finally, in the specific model usage phase S60, the multi-channel input samples corresponding to the region are input into the trained deep learning model. The model internally generates fused features and outputs the defect type evaluation result through the classification layer. This step is performed by the classification and recognition module 400 in the system.

[0096] After the deep learning model has been trained, the main purpose of the application phase is to use the model to identify defects in newly acquired regions to be identified. This involves applying the local texture features and overall structural features learned by the model to the actual remote sensing images, thereby outputting the defect type and its confidence level for each region. This phase does not involve updating the model parameters; instead, it uses the trained parameters directly for inference.

[0097] First, the input data is consistent with that of the training phase, including: spectral distribution characteristics (reflecting the surface properties of the material); texture structure characteristics (reflecting microscopic changes such as local cracks and erosion); and engineering structural attribute characteristics (reflecting structural information such as dams and gates).

[0098] The aforementioned features are aligned in space and standardized to form a fusion vector, ensuring that the model can correctly match the feature patterns learned during the training phase.

[0099] The fused vector is input into a deep learning model (which has been iteratively trained) for classification. The model first performs the following steps: multi-layer convolution operations on the input fused vector to extract local texture and material surface differences, forming local feature representations; and then uses dilated convolution and spatial aggregation operations to capture deformation features at the macroscopic engineering component level. In this process, the convolutional feature encoding layer and the spatial structure extraction layer are processed in parallel without interference, and then integrated into a comprehensive feature through the fusion layer.

[0100] The fused feature representations are input into the classification layer. The model calculates the probability of each defect category based on the feature vectors. The output includes: 1. Defect type: the category determined by the model (such as cracks, erosion, settlement, etc.); 2. Confidence value: the probability predicted by the model for this category, used to measure the reliability of the identification.

[0101] This output is the model's ability to quickly identify the type of defect in a new region (newly acquired image) by using the local texture and overall structure discrimination patterns learned in the previous training phase.

[0102] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. The scope of protection of the present invention is defined by the claims. Similarly, any equivalent structural changes made based on the description and drawings of the present invention should also be included within the scope of protection of the present invention.

Claims

1. A method for identifying defects in hydraulic engineering based on deep learning, characterized in that, Includes the following steps: A time series of remote sensing images of the same area within a target water conservancy project is constructed. Pixel intensity distribution maps are extracted using correlation analysis and expectation-maximization algorithms. Threshold segmentation and morphological processing are combined to extract areas from these pixel intensity distribution maps where spatial intensity distribution is continuous and exceeds a set threshold, as regions to be identified. After scale normalization of these regions, the extracted multispectral features, texture features, and structural attributes of the project are organized in parallel according to channel dimensions to generate multi-channel input samples. A multi-branch convolutional deep learning model is constructed to identify defects in the input samples. The convolutional feature encoding layer in the deep learning model uses small-scale convolution kernels to perform multi-layer convolution operations on the multispectral and texture channels of the input samples, encoding local features such as surface reflection differences, crack textures, and local erosion traces. The spatial structure extraction layer uses dilated convolutions with different dilation rates and spatial aggregation operations on the structural attribute channels of the input samples to capture the overall structural anomalies of the project. The cross-scale fusion layer spatially aligns the output local features with the overall structural anomaly features to generate unified fused features; The classification layer outputs a defect type classification for the fused features. During the training phase of the deep learning model, a corresponding defect type label is constructed for each sample of the region to be identified. Cross-entropy loss is used to supervise the classification output, and structural consistency constraints are introduced to optimize the synergistic relationship between local texture changes and overall structural deformation in the fused features. The multi-channel input samples corresponding to the region are input into the trained deep learning model, and the model generates fused features and outputs the defect type evaluation results through the classification layer.

2. The method for identifying defects in hydraulic engineering based on deep learning according to claim 1, characterized in that, For the same spatial area of ​​the target water conservancy project, remote sensing image data collected at different time points are acquired. Through geometric correction, the spatial position of each remote sensing image is aligned with a unified geographic coordinate system to construct a time series of remote sensing images for the area.

3. The method for identifying defects in hydraulic engineering based on deep learning according to claim 2, characterized in that, For each phase of the remote sensing image in the time series, pixel-level time-series spectral analysis is performed. Correlation analysis is used to measure the spectral consistency of each pixel in the time dimension, thereby distinguishing stable pixels from potentially changing pixels. Then, the expectation-maximization algorithm is used to iteratively solve the probability of change state of each pixel to obtain the change intensity value of each pixel, forming a pixel change intensity distribution map.

4. The method for identifying defects in hydraulic engineering based on deep learning according to claim 1, characterized in that, After obtaining the pixel change intensity distribution map, the Otsu segmentation algorithm is applied to analyze the histogram of pixel change intensity distribution, and the threshold corresponding to the maximum inter-class variance is selected. Pixels with change intensity higher than the threshold are marked as candidate change pixels, and a set of candidate change pixels is selected. Connectivity analysis is performed on the candidate set of changed pixels to combine spatially adjacent pixels into continuous regions, and a unique label is assigned to each continuous region to form a preliminary set of regions to be identified. Then, morphological filtering is performed on each independent region in the set of regions to be identified to remove isolated noise smaller than a set area threshold, generating the regions to be identified for the classification and identification of defects in engineering facilities.

5. The method for identifying defects in hydraulic engineering based on deep learning according to claim 1, characterized in that, The deep learning model employs a multi-branch convolutional structure. The input samples include standardized multispectral channels, texture channels, and structural attribute channels. Each type of feature is input into the deep learning model as a separate branch. The convolutional feature encoding layer consists of multiple convolutional units stacked sequentially. Each convolutional unit includes a small-scale convolutional kernel for nonlinear activation and feature normalization operations. The small-scale convolutional kernel performs sliding convolution operations on the inputs corresponding to the multispectral and texture channels within their local neighborhoods, progressively extracting local features such as material surface reflection differences, crack texture orientation, and local erosion traces. The spatial structure extraction layer uses dilated convolutions with different dilation rates to process the inputs of the structural attribute channels in parallel. The structural features obtained under different dilation rates are integrated through spatial aggregation operations to form overall structural anomaly features reflecting the construction of engineering facilities. The cross-scale fusion layer integrates the local features and overall structural anomaly features, performs spatial size alignment processing, maps them to a unified spatial resolution, and performs weighted fusion based on the contribution of each branch feature in defect discrimination, forming a fused feature that simultaneously contains local texture changes and overall structural anomaly information.

6. The method for identifying defects in hydraulic engineering based on deep learning according to claim 5, characterized in that, During the training phase of the deep learning model, multi-channel input samples that have undergone scale normalization and channel alignment are paired with their corresponding engineering defect type labels to construct a sample set for model training. During training, the input samples are sequentially fed into the convolutional feature encoding layer and the spatial structure extraction layer to extract the local features and overall structural anomaly features of the region to be identified, respectively. Then, local features from different branches and overall structural anomaly features are input into the cross-scale fusion layer for spatial alignment and fusion to obtain the fused features used for defect discrimination; The classification layer in the deep learning model maps fused features to predict various engineering defects. During backpropagation, the cross-entropy loss function is used to measure the difference between the predicted results and the corresponding defect type labels, and the difference is used to iteratively optimize the parameters of each layer in the deep learning model.

7. The method for identifying defects in hydraulic engineering based on deep learning according to claim 6, characterized in that, During the parameter update process of the deep learning model, the cross-entropy loss between the predicted result of the defect category and the corresponding label is used as the main supervision signal. At the same time, structural consistency constraints are introduced. By constraining the response distribution generated by the local feature branches and the overall structural anomaly feature branches in the fusion features, the local texture change features in the same region to be identified and their corresponding structural features maintain a consistent response relationship in space. The structural consistency constraint term and the cross-entropy loss are jointly backpropagated to update the parameters of the deep learning model.

8. The method for identifying defects in hydraulic engineering based on deep learning according to claim 1, characterized in that, After the deep learning model is trained, the input samples corresponding to the acquired region are input into the deep learning model. The convolutional feature encoding layer performs forward calculation on the input samples, extracts and outputs local features that characterize the spectral changes and crack textures of the material surface. The spatial structure extraction layer extracts and outputs overall structural anomaly features that characterize the region based on the input samples and their spatial context information. The local feature representations and the overall structural anomaly features are aligned in the cross-scale fusion layer according to spatial correspondence, and a unified fusion feature vector is formed by channel splicing. The fusion feature vector is then input into the classification layer, where it is mapped to the predicted response value of each defect category through the parameters learned during the training phase, and the corresponding category probability distribution is obtained after normalization. The deep learning model determines the defect category corresponding to the region based on the category probability distribution and outputs the predicted probability of each defect category.

9. A deep learning-based defect identification system for hydraulic engineering projects, characterized in that, A defect identification method according to any one of claims 1 to 8, comprising: a data monitoring module configured to: construct a time series of remote sensing images of the same area in a target water conservancy engineering facility, and extract a pixel change intensity distribution map using correlation analysis and expectation-maximization algorithms; a pre-identification module configured to: extract areas with continuously distributed spatial change intensity exceeding a set threshold from the pixel change intensity distribution map by combining threshold segmentation and morphological processing as areas to be identified; after scale normalization of the areas to be identified, organize the extracted multispectral features, texture features, and structural attributes of the engineering facility in parallel according to the channel dimension to generate multi-channel input samples; and a model building module configured to: construct a multi-branch convolutional deep learning model for defect identification of the input samples, wherein the convolutional feature encoding layer in the deep learning model uses small-scale convolutional kernels for the multispectral and texture channels in the input samples. Multi-layer convolution operations are performed to encode local features such as surface reflection differences, crack textures, and local erosion traces. A spatial structure extraction layer uses dilated convolutions and spatial aggregation operations with different dilation rates on the structural attribute channels of the input samples to capture the overall structural anomaly features of the engineering facilities. A cross-scale fusion layer spatially aligns the output local features with the overall structural anomaly features to generate unified fusion features. A classification layer classifies the fusion features to reflect defect types. During the training phase of the deep learning model, a corresponding defect type label is constructed for each region sample to be identified. Cross-entropy loss is used to supervise the classification output, while structural consistency constraints are introduced to optimize the synergistic relationship between local texture changes and overall structural deformation in the fusion features. The defect classification module is configured to input the multi-channel input samples corresponding to the region into the trained deep learning model, generate fusion features internally, and output defect type evaluation results through the classification layer.

Images