A method and system for detecting concrete cracks in hydraulic engineering

Abstract

This invention relates to a method and system for detecting concrete cracks in hydraulic engineering, belonging to the field of image processing technology. It addresses the shortcomings of current crack detection technologies and includes the following steps: acquiring multiple frames of concrete images based on a time series and their corresponding motion data; extracting features from each frame to obtain an original input feature map; processing the original input feature map using a spatial attention mechanism to obtain a detail-enhanced feature map; performing frequency domain enhancement processing on the detail-enhanced feature map to obtain a frequency domain-enhanced feature map; performing residual concatenation between the frequency domain-enhanced feature map and the original input feature map to obtain an output feature map; transforming and aligning the output feature maps of historical frames in the time series to the coordinate system of the current frame's output feature map to form an aligned frame feature sequence; fusing the feature maps of the aligned frame feature sequence, and segmenting the fused feature map to obtain crack segmentation results. The method of this invention is low-cost and highly flexible.

Description

Technical Field

[0001] This invention relates to the field of computer and image processing technology, specifically to a method and system for detecting concrete cracks in hydraulic engineering projects. Background Technology

[0002] The statements herein provide only background information in relation to this invention and do not necessarily constitute prior art.

[0003] In the field of hydraulic engineering, traditional crack detection methods mainly rely on manual inspections or handheld devices to collect single-frame images, which are then classified and located using image processing or deep learning models. These methods suffer from the following main problems: single-frame images are difficult to extract complete crack features under conditions such as low contrast, non-uniform lighting, texture interference, or crack fracture; some methods rely on high-resolution industrial cameras, specific angles, or static installation platforms, limiting their flexibility in practical applications; existing models are mostly for static image classification or segmentation tasks, failing to fully utilize the spatial and temporal correlations between consecutive frames during camera movement; traditional methods lack stability in dynamic environments such as vibration, occlusion, and changes in viewing angle, easily leading to false positives and false negatives. To address these issues, some research has attempted to introduce video streams as a detection data source or fuse information from multiple sensors such as structured light and lidar. However, these solutions typically involve high-cost sensor configurations and require high calibration accuracy and data synchronization, making them difficult to deploy in resource-constrained environments or field settings. Summary of the Invention

[0004] In view of the shortcomings of the existing technology, the purpose of this invention is to provide a method and system for detecting concrete cracks in water conservancy projects, which overcomes the defects of current crack detection methods.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solution:

[0006] In a first aspect, embodiments of the present invention provide a method for detecting cracks in concrete in hydraulic engineering projects, comprising the following steps:

[0007] Acquire multi-frame images of concrete based on time series and the self-motion data of the image acquisition device corresponding to the acquisition of each frame image. The self-motion data includes the triaxial acceleration and angular velocity of the image acquisition device.

[0008] Feature extraction is performed on each frame of the image to obtain the original input feature map;

[0009] The original input feature map is processed using a spatial attention mechanism to obtain a detail-enhanced feature map;

[0010] The detail-enhanced feature map is subjected to frequency domain enhancement processing to obtain a frequency domain enhanced feature map;

[0011] The frequency domain enhanced feature map is residually concatenated with the original input feature map to obtain the output feature map;

[0012] For the output feature map of the historical frame in the time series, an affine transformation is performed according to the corresponding pose transformation matrix to align the output feature map of the historical frame to the coordinate system of the current frame output feature map, forming an aligned frame feature sequence. The pose transformation matrix is ​​obtained from the self-motion data.

[0013] A cross-frame frequency domain cross-attention mechanism is used to fuse the aligned frame feature maps of the aligned frame feature sequence, outputting a fused feature map. The fused feature map is then used to obtain crack segmentation results through a segmentation network, thereby achieving crack detection.

[0014] Optionally, the frequency domain enhancement processing method is as follows: perform Fourier transform on the detail enhancement feature map, and use sliding window and mask multiplication operations to realize frequency domain feature reweighting to complete frequency domain processing; map the result of frequency domain processing back to the spatial domain through inverse Fourier transform to obtain the frequency domain enhanced feature map.

[0015] Optionally, after mapping the result of frequency domain processing back to the spatial domain through inverse Fourier transform to obtain the frequency domain processed feature map, a 1X1 convolution operation is performed, and then the frequency domain enhanced feature map is obtained based on the convolution operation result and the detail enhancement feature map.

[0016] Optionally, the spatial attention mechanism can be used to process the feature map as follows: perform convolution and layer normalization on the original input feature map, obtain the feature map after spatial attention weighting and the feature map after channel and spatial joint weighting based on the processed feature map, add the feature map after spatial attention weighting and the feature map after channel and spatial joint weighting, and then perform convolution to obtain the weight matrix. Multiply the weight matrix with the original input feature map to obtain the detail enhancement feature map.

[0017] Optionally, the method for obtaining the spatial attention-weighted feature map is as follows:

[0018] The processed feature maps are then subjected to global average pooling and global max pooling to obtain average pooling features and max pooling features.

[0019] The average pooling features and the max pooling features are concatenated along the channel dimension of the processed feature map to form a concatenated feature map that incorporates contextual information.

[0020] The concatenated feature map is convolved, and the result of the convolution operation is normalized to obtain the spatial attention weight matrix.

[0021] The processed feature map is multiplied element-wise with the spatial attention weight matrix to obtain the spatial attention-weighted feature map.

[0022] Alternatively, the method for obtaining the feature map after joint channel and spatial weighting is as follows:

[0023] Global average pooling is performed on the processed feature maps to obtain global feature descriptions for each channel. Convolution transformation is then applied to the global feature descriptions to form channel attention maps.

[0024] The processed feature map is convolved to obtain the spatial attention map;

[0025] Add the channel attention map and the spatial attention map together and normalize them to obtain the joint attention weight matrix;

[0026] The processed feature map is multiplied element-wise with the joint attention weight matrix to obtain the channel- and spatially weighted feature map.

[0027] Optionally, a method for processing the aligned frame feature sequence using a cross-frame frequency domain cross-attention mechanism is as follows:

[0028] Perform Fourier transform on all aligned frame feature maps in the aligned frame feature sequence to extract the amplitude map;

[0029] Attention calculations are performed based on the amplitude maps of the current frame and the amplitude maps of historical frames, and a frequency domain fused representation is output.

[0030] The frequency domain features are reconstructed by combining the phase spectrum of the current frame amplitude map with the frequency domain fusion representation and inversely transformed back to the spatial domain, and the fused feature map is output, thus completing the processing of the cross-frame frequency domain cross-attention mechanism.

[0031] Optionally, the pose transformation matrix can be obtained by pre-integration and state estimation based on the self-motion data corresponding to each frame of the image.

[0032] Optionally, the pose transformation matrix can be obtained by filtering the self-motion data and then pre-integrating and estimating the state.

[0033] Secondly, embodiments of the present invention provide a concrete crack detection system for hydraulic engineering projects, comprising:

[0034] Image acquisition module: configured to acquire multi-frame images of concrete based on time series, and acquire the self-motion data of the image acquisition device corresponding to each frame of image acquisition, including the triaxial acceleration and angular velocity of the image acquisition device;

[0035] Feature extraction module: configured to extract features from each frame of the image to obtain the original input feature map;

[0036] The detail enhancement processing module is configured to process the original input feature map using a spatial attention mechanism to obtain a detail-enhanced feature map.

[0037] Frequency domain enhancement processing module: configured to perform frequency domain enhancement processing on the detail enhancement feature map to obtain the frequency domain enhanced feature map;

[0038] The residual connection module is configured to perform a residual connection between the frequency-domain enhanced feature map and the original input feature map to obtain an output feature map.

[0039] Feature alignment module: It is configured to perform affine transformation on the output feature map of the historical frame in the time series according to the corresponding pose transformation matrix so that the output feature map of the historical frame is aligned to the coordinate system of the output feature map of the current frame, forming an aligned frame feature sequence. The pose transformation matrix is ​​obtained from the self-motion data.

[0040] Crack segmentation module: It is configured to use a cross-frame frequency domain cross-attention mechanism to fuse the aligned frame feature maps of the aligned frame feature sequence, output a fused feature map, and use the fused feature map to obtain crack segmentation results through a segmentation network to achieve crack detection.

[0041] The beneficial effects of this invention are as follows:

[0042] 1. The crack detection method of the present invention utilizes a spatial attention mechanism to enhance the details of the original input feature map, and simultaneously performs frequency domain enhancement processing on the feature map with enhanced details, thus obtaining a frequency domain enhanced feature map. This effectively enhances the features of tiny cracks in each frame of the image, especially in low-light or structurally blurred scenes, accurately capturing the structural contours and details of cracks, improving the robustness of target detection in complex scenes, and facilitating the extraction of complete crack features in complex scenes. Image acquisition can be done with a regular camera, without the need for a high-resolution industrial camera or a static mounting platform, making it highly flexible in practical applications.

[0043] 2. The crack detection method of the present invention performs an affine transformation on the output feature map of historical frames in the time series according to the corresponding pose transformation matrix, aligning it to the coordinate system of the output feature map of the current frame, forming an aligned frame feature sequence; the aligned frame feature map of the aligned feature sequence is fused using a cross-frame frequency domain cross-attention mechanism to output a fused feature map, and the fused feature map is used to obtain crack segmentation results through a segmentation network to realize crack detection. This method realizes the use of time series as the detection data source, overcomes the defects of detection using static data sources, and does not require high-cost sensors and complex calibration environments, achieving high-precision detection of crack targets and has strong applicability. Attached Figure Description

[0044] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0045] Figure 1This is a flowchart of the method in Embodiment 1 of the present invention;

[0046] Figure 2 These are images of concrete collected in Embodiment 1 of the present invention;

[0047] Figure 3 This is a flowchart of the spatial attention mechanism processing and frequency domain enhancement processing in Embodiment 1 of the present invention;

[0048] Figure 4 This is an image showing the crack identification result of Embodiment 1 of the present invention;

[0049] Figure 5 This is an image showing the crack identification results from existing technologies; Detailed Implementation

[0050] Example 1

[0051] This embodiment provides a method for detecting concrete cracks in hydraulic engineering projects, such as... Figure 1 As shown, it includes the following steps:

[0052] Step 1: Obtain multi-frame images of concrete based on time series and the self-motion data corresponding to each frame.

[0053] In this embodiment, an image acquisition device is used to acquire multiple frames of images based on a time series. The image acquisition device is a camera, which is mounted on a mobile device such as a regular mobile terminal, a robot platform, or a drone for image acquisition. The mobile device is also equipped with an inertial measurement unit to record the camera's self-motion data in real time. In this embodiment, the self-motion data includes the camera's three-axis acceleration and angular velocity. The acquired image data and self-motion data are time-aligned using the mobile device's built-in hardware clock or synchronization protocol to ensure that the acquired data has temporal consistency.

[0054] In this embodiment, a camera is used to acquire multiple frames of images based on a time series and obtain the self-motion data corresponding to each frame.

[0055] The acquired multiple frames constitute an image frame sequence.

[0056] Among them, I t Indicates the first t A frame image with a resolution of HxW, where H represents the height of the image, W represents the width of the image, and T represents the total number of frames in the image frame sequence. The images are temporally sequential, meaning there are slight viewpoint shifts or displacements between images.

[0057] The self-motion data corresponding to multiple frames of images constitutes a self-motion data sequence S. t :

[0058] Where a tLet ω be the triaxial acceleration vector. t Let be the angular velocity vector, representing the linear and rotational motion state of the camera at time t.

[0059] The data synchronization structure D is:

[0060] Each frame corresponds to a set of motion data. In this embodiment, the image of concrete captured by the camera is as follows: Figure 2 As shown.

[0061] Step 2: Extract features from each frame of the image to obtain the original input feature map.

[0062] In this embodiment, a ResNet feature extraction network is used to extract features from each frame of the image to obtain the original input feature map corresponding to each frame of the image.

[0063] The method of feature extraction using the ResNet feature extraction network can be achieved using existing techniques, and will not be described in detail here.

[0064] Step 3: As Figure 3 As shown, the original input feature map is processed using a spatial attention mechanism to obtain a detail-enhanced feature map, including the following specific steps:

[0065] Step 3.1: Perform layer normalization and convolution processing on the original input feature map. Specifically, perform 3x3 convolution operation (3x3 Conv), layer normalization operation (LNorm), 1x1 convolution operation (1x1 Conv) and 3x3 convolution operation (3x3 Conv) on the original input feature map in sequence.

[0066] Step 3.2: Based on the feature map processed in Step 3.1, obtain the feature map after spatial attention weighting (SC) and the feature map after channel and spatial joint weighting (SCA).

[0067] Spatial attention-weighted feature maps are feature maps that have been weighted by the spatial location of the processed feature maps. This enhances the feature responses of important regions while suppressing irrelevant background information. The method for obtaining these feature maps includes the following steps:

[0068] Step a: Perform global average pooling and global max pooling on the processed feature map to obtain average pooling features and max pooling features, which respectively reflect the overall strength and the maximum response of the layout.

[0069] Step b: Concatenate the average pooling features and max pooling features along the channel dimension of the processed feature map to form a concatenated feature map that incorporates contextual information.

[0070] Step c: Perform a convolution operation on the concatenated feature map with a kernel size of 7x7, and normalize the convolution result with Sigmoid to obtain the spatial attention weight matrix.

[0071] No activation function is used in step c in order to preserve the complete feature response.

[0072] Step d: Multiply the processed feature map element by element with the spatial attention weight matrix to achieve spatial weighting of the input processed feature map, and obtain the spatial attention weighted feature map.

[0073] The feature map after joint channel and spatial weighting can enhance the feature response by utilizing channel and spatial information, realizing a lightweight joint attention mechanism. The acquisition method includes the following steps:

[0074] Step 1): Perform global average pooling on the processed feature map to obtain the global feature description for each channel. Perform 1x1 convolution transformation on the global feature description to form the channel attention map.

[0075] Step 2): Perform a 7x7 convolution transformation on the processed feature map to obtain the spatial attention map;

[0076] Step 3): Add the channel attention map and the spatial attention map together, and normalize them using Sigmoid to obtain the joint attention weight matrix.

[0077] Step 4): Multiply the processed feature map element by element with the joint attention weight matrix to obtain a channel- and spatially weighted feature map, which can better highlight the target features and retain important information, while maintaining a lightweight structure.

[0078] In this embodiment, neither the spatial attention-weighted feature map (SC) nor the channel- and spatially weighted feature map (SCA) was obtained using activation functions such as ReLU. This helps to preserve the complete feature response and avoids nonlinear operations from weakening weak target signals.

[0079] Step 3.3: Add the spatial attention-weighted feature map and the channel and spatial joint-weighted feature map together, and then perform 3x3 convolution operation (3x3 Conv) and 1x1 convolution operation (1x1 Conv) in sequence to obtain the weight matrix.

[0080] Step 3.4: Multiply the weight matrix obtained in Step 3.3 element by element with the original input feature map to obtain the detail enhancement feature map. Each frame of the image corresponds to a detail enhancement feature map.

[0081] Step 4: Perform frequency domain enhancement processing on the detail enhancement feature map to obtain the frequency domain enhanced feature map. This includes the following steps:

[0082] Step 4.1: Perform Fourier transform (FFT) on the detail enhancement feature map.

[0083]

[0084] Wherein: F t Enhance feature maps for details. For the frequency domain representation of the detail-enhanced feature map of frame t, FFT, the Fourier transform operator, transforms the feature map from the spatial domain to the frequency domain for analyzing its spectral information. The amplitude map, obtained by Fourier transforming the detail enhancement map of the t-th frame image, represents the intensity of each frequency component and reflects the image's texture, structure, and other features. θ t Indicates the first t The phase spectrum of the Fourier transform of the detail enhancement feature map of a frame image represents the phase information of each frequency component, which determines the geometric structure and details of the image. j The imaginary unit, Combined into the complex form of the phase term.

[0085] In this embodiment, after layer normalization (LNorm) is performed on the detail enhancement feature map, an FFT transformation is applied. The purpose of the FFT transformation is to convert the input spatial domain image into a frequency domain representation. In this representation, the content of the spatial domain image is resolved into a combination of different frequency components. The low-frequency part reflects the overall contour and background information of the spatial domain image, while the high-frequency part contains the edges, textures, and detailed structures in the spatial domain image. This frequency domain representation is more advantageous than the spatial domain representation for directly analyzing and enhancing certain target features, especially those anomalous targets that are sparsely distributed in space but have obvious energy concentration in frequency (such as cracks, noise, etc.).

[0086] Step 4.2: After Fourier transform, the transformed frequency domain image is subjected to sliding window (FFT Shift) and mask multiplication operations to realize frequency domain feature reweighting to complete frequency domain processing.

[0087] The FFT-transformed spectrum by default concentrates low-frequency information in the four corners of the image, while high-frequency information is distributed around the center. This layout is not conducive to understanding and manipulation. Therefore, a sliding window operation is used to move the low-frequency part of the spectrum to the center of the frequency domain image, while the high-frequency part is redistributed to the edges. This centralized spectrum form is more intuitive for processing and also facilitates the design of masks for frequency selection and control.

[0088] Mask multiplication is one of the core operations in the entire frequency domain enhancement process. It uses a preset or learnable frequency domain mask to adjust the weights of different frequency regions in a spectral image. By designing an appropriate mask, high-frequency components (such as texture and edges) can be emphasized while low-frequency background components are suppressed, thereby improving the model's responsiveness to details. For example, in crack detection tasks, high frequencies contain the vast majority of crack features; enhancing this part helps the model focus on key regions. Since the FFT transform output is a complex number, mask multiplication is applied to the frequency domain image in a complex multiplication manner, maintaining the consistency of amplitude and phase information.

[0089] Step 4.3: Map the result of frequency domain processing back to the spatial domain through inverse Fourier transform (IFFT) to obtain the frequency domain enhanced feature map.

[0090]

[0091] IFFT stands for Inverse Fourier Transform operator. This indicates the result after applying sliding window and mask multiplication operations. t Frame amplitude diagram μ t The phase spectrum of the feature map after frequency domain processing. j The imaginary unit, Combined into the complex form of the phase term.

[0092] Since specific enhancements have been applied in the frequency domain, the image output in this step already exhibits more prominent structural features in space, such as clearer edges and more obvious details.

[0093] In this embodiment, the frequency domain processed feature map (FFTback) is obtained after inverse Fourier transform. Then, a 1X1 convolution operation is performed on the frequency domain processed feature map to obtain a weight matrix. The weight matrix obtained by the convolution operation is then multiplied element-wise with the detail enhancement feature map to obtain the frequency domain enhanced feature map.

[0094] Step 5: Perform residual concatenation between the frequency domain enhanced feature map and the original input feature map to obtain the output feature map.

[0095] Specifically, the frequency domain enhanced feature map is added element-wise to the corresponding original input feature map to obtain the output feature map. This preserves global information and basic features from the original input feature map while avoiding excessive changes caused by frequency domain enhancement that could lead to image distortion or noise amplification. Residual connections make it easier for the network to learn "incremental features," meaning enhancements are only performed where necessary, thereby improving the model's convergence speed, robustness, and generalization ability.

[0096] The residual connection method can be achieved using existing technologies, and will not be described in detail here.

[0097] Step 6: Perform an affine transformation on the output feature map of the historical frame in the time series according to the corresponding pose transformation matrix to align it to the coordinate system of the output feature map of the current frame, forming an aligned frame feature sequence. The pose transformation matrix is ​​obtained from the self-motion data.

[0098] Since the output feature maps of adjacent frames are affected by viewpoint changes and self-motion, direct fusion will lead to misalignment. Therefore, in this embodiment, the self-motion data corresponding to each acquired frame image is filtered, and a pre-integration and state estimation method is used to obtain the pose transformation matrix between adjacent frames. :

[0099]

[0100] The pre-integration and state estimation algorithms can be derived using existing techniques and will not be described in detail here. Represents the output feature map of frame t and the... The pose transformation matrix between the frame output feature maps.

[0101] In this embodiment, the filtering is performed using an extended Kalman filter, and the state estimation is performed using a VIO system estimation.

[0102] Based on the pose transformation matrix Output feature maps of historical frame images in the time series Align the coordinates of the output feature map of the current frame image with the coordinate system through an affine transformation.

[0103]

[0104] Warp is an affine transformation algorithm, which can be implemented using existing techniques and will not be described in detail here. Indicates the first The frame output feature map is transformed by affine transformation to the 1st frame. t The output feature map after the frame output feature map coordinate system.

[0105] The output feature maps of all neighboring frames are aligned, forming an aligned frame feature sequence W composed of multiple aligned frame feature maps. t :

[0106]

[0107] To align the feature sequence of the first frame tk Frame alignment feature map k= 1, 2, 3... t -1.

[0108] Step 7: Use the cross-frame frequency domain cross-attention mechanism to fuse the aligned frame feature maps of the aligned frame feature sequence, output the fused feature map, and use the fused feature map to obtain the crack segmentation result through the segmentation network to realize crack detection.

[0109] The cross-frame frequency domain cross-attention mechanism can further mine shared information in the frequency domain of each frame, and establish dynamic fusion weights with the current frame as the main focus. Specifically, it includes the following steps:

[0110] Step 7.1: Process all aligned frame feature maps in the aligned frame feature sequence. Perform Fourier transform to extract the amplitude map. .

[0111]

[0112] τ = 1, 2, 3...t.

[0113] Step 7.2: Build a query using the amplitude map of the current frame, and perform attention calculation using the amplitude maps of other frames as keys and values.

[0114]

[0115] α τ Represents the amplitude diagram of frame t. Features and amplitude diagram of frame τ Attention weight coefficients between features, with Softmax as the normalization function. W q To query the mapping matrix, W k The key mapping matrix, The normalization factor represents the feature dimension.

[0116] Step 7.3: Based on the attention calculation results, output the frequency domain fusion representation:

[0117]

[0118] This indicates the first [item] after attention-weighted fusion. t The increased amplitude spectrum characteristics of the frame α t Indicates the first t Frame amplitude map features and the first τ Attention weight coefficients between frame amplitude map features W v For value mapping matrix, For the first τ Amplitude diagram of the frame.

[0119] Step 7.4: Use the phase spectrum of the current frame amplitude map obtained in Step 7.1 The frequency domain features are reconstructed from the output frequency domain fusion representation and then subjected to inverse Fourier transform (IFFT) back to the spatial domain to obtain the fused feature map. .

[0120]

[0121] This indicates the first [item] after attention-weighted fusion. t The increased amplitude spectrum characteristics of the frame j The imaginary unit, Combined into the complex form of the phase term.

[0122] Step 7.5: Normalize the fused feature map, then perform a residual connection with the aligned frame feature map of the current frame in the aligned frame feature sequence, and then use the segmentation network to perform semantic segmentation to obtain the crack segmentation result, thus realizing crack detection. The obtained crack segmentation result is as follows: Figure 4 As shown, the method of obtaining segmentation results by using segmentation networks can be achieved using existing technologies, and will not be described in detail here.

[0123] Cracks identified using existing technology, such as Figure 5 As shown, through Figure 4 and Figure 5 The comparison shows that the method in this embodiment can detect cracks in the shaded area, while the prior art cannot detect cracks in the shaded area.

[0124] The detection method in this embodiment utilizes a spatial attention mechanism to enhance the details of the original input feature map, and simultaneously performs frequency domain enhancement processing on the enhanced feature map to obtain a frequency domain enhanced feature map. This effectively enhances the features of tiny cracks in each frame of the image, especially in low-light or structurally blurred scenes, accurately capturing the structural contours and details of cracks. It effectively strengthens the edge details and structural information of cracks, improving the robustness of target detection in complex scenes. This is beneficial for extracting complete crack features in complex scenes. Image acquisition can be done with a regular camera, without the need for a high-resolution industrial camera or a static mounting platform. It offers high flexibility in practical applications and utilizes a temporal stream as the detection data source, overcoming the shortcomings of using static data sources. Furthermore, it eliminates the need for high-cost sensors and complex calibration environments, achieving high-precision detection of crack targets and demonstrating strong applicability.

[0125] In this embodiment, considering the requirements of inter-frame stability for actual crack detection, a temporal consistency loss term is designed during training to supervise the consistency of prediction results in physical space between adjacent frames. Let the method model of this embodiment be in frame... t andt The predicted segmentation result on +1 is P t ,P t+1 ,right t The segmentation result of +1 frame is based on the pose transformation matrix. Perform an affine transformation on the current frame to obtain the transformed segmentation result. :

[0126]

[0127] Warp is an affine transformation algorithm. For the first t The prediction result of +1 frame and the first frame t The pose transformation matrix between the predicted results of the frames is obtained from the self-motion data.

[0128] Calculate timing consistency loss for:

[0129]

[0130] Total loss for:

[0131]

[0132] The loss is generated for semantic segmentation, where λ is a set hyperparameter.

[0133] Example 2

[0134] This embodiment provides a crack detection system based on time-series flow and frequency domain enhancement, including:

[0135] Image acquisition module: configured to acquire multiple frames of images based on time series, and acquire the self-motion data corresponding to each frame of image;

[0136] Feature extraction module: configured to extract features from each frame of the image to obtain the original input feature map;

[0137] The detail enhancement processing module is configured to process the original input feature map using a spatial attention mechanism to obtain a detail-enhanced feature map.

[0138] Frequency domain enhancement processing module: configured to perform frequency domain enhancement processing on the detail enhancement feature map to obtain the frequency domain enhanced feature map;

[0139] The residual connection module is configured to perform a residual connection between the frequency-domain enhanced feature map and the original input feature map to obtain an output feature map.

[0140] Feature alignment module: It is configured to align the output feature maps of historical frames in the time series to the coordinate system of the current frame output feature map by performing an affine transformation based on the corresponding pose transformation matrix, forming an aligned frame feature sequence. The pose transformation matrix is ​​obtained from the motion data.

[0141] Crack segmentation module: It is configured to use a cross-frame frequency domain cross-attention mechanism to fuse the aligned frame feature maps of the aligned frame feature sequence, output a fused feature map, and use the fused feature map to obtain crack segmentation results through a segmentation network to achieve crack detection.

[0142] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for detecting cracks in hydraulic concrete, characterized in that, The method comprises the following steps: Obtaining a plurality of frames of images of concrete based on a time sequence and self-motion data of an image acquisition device corresponding to each frame of image at the time of acquisition, the self-motion data comprising three-axis acceleration and angular velocity of the image acquisition device; Extracting features from each frame of image to obtain an original input feature map; Processing the original input feature map using a spatial attention mechanism to obtain a detail-enhanced feature map; Performing frequency domain enhancement processing on the detail-enhanced feature map to obtain a frequency domain enhanced feature map; Residual connection is performed on the frequency domain enhanced feature map and the original input feature map to obtain an output feature map; Performing affine transformation on the output feature map of the historical frame in the time sequence according to the corresponding pose transformation matrix to align the output feature map of the historical frame to the coordinate system of the current frame output feature map, forming an aligned frame feature sequence, and the pose transformation matrix is obtained from the self-motion data; Fusion processing is performed on the aligned frame feature map of the aligned frame feature sequence using a cross-frame frequency domain cross-attention mechanism, and a crack segmentation result is obtained from the fused feature map using a segmentation network to realize crack detection.

2. The method of claim 1, wherein the method comprises: The method of frequency domain enhancement processing is: performing Fourier transform on the detail-enhanced feature map, and re-weighting the frequency domain features by using a sliding window and a mask multiplication operation to complete the frequency domain processing; the result of the frequency domain processing is mapped back to the spatial domain through inverse Fourier transform to obtain the frequency domain enhanced feature map.

3. The method of claim 2, wherein the step of detecting the crack in the hydraulic engineering concrete is performed by using a crack detection device. After the result of the frequency domain processing is mapped back to the spatial domain through inverse Fourier transform to obtain the frequency domain processing feature map, 1X1 convolution operation is performed, and then the frequency domain enhanced feature map is obtained according to the convolution operation result and the detail-enhanced feature map.

4. The method of claim 1, wherein the method is a method of detecting cracks in hydraulic concrete. The method of processing using the spatial attention mechanism is: performing layer normalization and convolution processing on the original input feature map, obtaining a feature map weighted by spatial attention and a feature map weighted by channel and spatial joint according to the processed feature map, adding the feature map weighted by spatial attention and the feature map weighted by channel and spatial joint, and then performing convolution processing to obtain a weight matrix, multiplying the weight matrix with the original input feature map to obtain the detail-enhanced feature map.

5. The method of claim 4, wherein the step of detecting the crack in the hydraulic engineering concrete is performed by using a crack detection device. The method of obtaining the feature map weighted by spatial attention is: Performing global average pooling and global maximum pooling on the processed feature map to obtain an average pooling feature and a maximum pooling feature; Concatenating the average pooling feature and the maximum pooling feature along the channel dimension of the convolution-processed feature map to form a spliced feature map with fused context information; Performing convolution operation on the spliced feature map, and normalizing the convolution operation result to obtain a spatial attention weight matrix; Element-wise multiplying the processed feature map with the spatial attention weight matrix to obtain the feature map weighted by spatial attention.

6. The method of claim 4, wherein the step of detecting the crack in the hydraulic engineering concrete is characterized by, The method of obtaining the feature map weighted by channel and spatial joint is: Performing global average pooling on the processed feature map to obtain a global feature description of each channel, and performing convolution transformation on the global feature description to form a channel attention map; Performing convolution operation on the processed feature map to obtain a spatial attention map; Adding the channel attention map and the spatial attention map, and normalizing to obtain a joint attention weight matrix; The processed feature map is multiplied element by element with the joint attention weight matrix to obtain a channel and space joint weighted feature map.

7. The method of claim 1, wherein the method is a method of detecting cracks in hydraulic concrete. The method for processing the aligned frame feature sequence by using the cross-frame frequency domain cross attention mechanism is as follows: Perform Fourier transform on all the aligned frame feature maps in the aligned frame feature sequence to extract amplitude maps. Perform attention calculation according to the amplitude map of the current frame and the amplitude maps of the historical frames to output a frequency domain fusion representation. The phase spectrum of the current frame amplitude map is combined with the frequency domain fusion representation to reconstruct a frequency domain feature and inverse transform back to the spatial domain to output a fused feature map, and the processing of the cross-frame frequency domain cross attention mechanism is completed.

8. The method of claim 1, wherein the method is a method of detecting cracks in hydraulic concrete. According to the ego-motion data corresponding to each frame of image, a pose transformation matrix is obtained through pre-integration and state estimation.

9. The method of claim 8, wherein the step of detecting the crack in the hydraulic engineering concrete is performed by using a crack detection device. After filtering the ego-motion data, a pose transformation matrix is obtained through pre-integration and state estimation.

10. A hydraulic concrete crack detection system, characterized by, It comprises: An image acquisition module configured to acquire multiple frames of images of concrete based on a time sequence, and to acquire ego-motion data of an image acquisition device corresponding to each frame of image acquisition, the ego-motion data including three-axis acceleration and angular velocity of the image acquisition device; A feature extraction module configured to perform feature extraction on each frame of image to obtain an original input feature map; A detail enhancement processing module configured to process the original input feature map using a spatial attention mechanism to obtain a detail enhancement feature map; A frequency domain enhancement processing module configured to perform frequency domain enhancement processing on the detail enhancement feature map to obtain a frequency domain enhanced feature map; A residual connection module configured to perform residual connection on the frequency domain enhanced feature map and the original input feature map to obtain an output feature map; A feature alignment module configured to perform affine transformation on the output feature maps of the historical frames in the time sequence according to the corresponding pose transformation matrix to align the output feature maps of the historical frames to the coordinate system of the current frame output feature map, forming an aligned frame feature sequence, and the pose transformation matrix is obtained from the ego-motion data; A crack segmentation module configured to perform fusion processing on the aligned frame feature maps of the aligned frame feature sequence by using a cross-frame frequency domain cross attention mechanism to output a fused feature map, and to obtain a crack segmentation result by using a segmentation network on the fused feature map to realize crack detection.

Images