Optical image structure apparent crack identification method and system

Abstract

The invention belongs to the technical field of image intelligent interpretation and structure apparent disease recognition, and relates to an optical image structure apparent crack recognition method and system, and the system comprises a structure feature construction module and a spatial detail feature enhanced encoder. Low model calculation complexity is maintained while spatial information of the small linear cracks is reserved; a depth semantic feature coding module is pre-trained, a channel attention mechanism-based space detail feature and depth semantic feature adaptive fusion module is constructed in a decoding stage for enhancement, and the expression ability of multi-scale crack features is improved; a crack'point-line-surface 'multi-task decoding module; and a'point-line-surface' multi-level joint optimization module. Under the condition of maintaining relatively low model complexity, the identification result of the small crack is more accurate and complete; and the model is more robust to complex interference information in the environment.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent image interpretation and structure surface disease recognition, and relates to an optical image structure surface crack recognition method and system. Background Art

[0002] At present, in the field of intelligent identification of cracks on the surfaces of structures such as bridges, tunnels, roads, and water conservancy projects, there are a large number of models and identification methods built through deep learning. These existing crack detection methods based on intelligent image interpretation usually include first collecting crack image data of specific application scenarios (such as pavement, tunnel lining, etc.) and annotating samples, and then performing pixel-level annotation, cropping, enhancement and other processing on the collected image data to obtain crack segmentation training samples, and then training the constructed semantic segmentation network based on the annotated crack sample data set, and finally cropping, predicting and splicing the test data through the trained network model.

[0003] For example, the invention patent with the publication number CN118781053A, entitled Tunnel Surface Crack Identification Method, Device, Electronic Device and Storage Medium, improves the efficiency of tunnel surface image recognition and the recognition accuracy of each sliced ​​image by segmenting and dicing the tunnel surface image and inputting each sliced ​​image into a deep learning network model for recognition processing. However, these methods usually have the following two technical problems: 1. Since the classic semantic segmentation network encoder contains multiple pooling downsampling layers, the spatial resolution of the extracted multi-scale semantic features is reduced by n times (n>=4) relative to the original input image, and a large number of crack cross-sections in the input image are less than 3 pixels, resulting in serious loss of semantic information of linear cracks in the feature extraction process, making it difficult to obtain reliable crack segmentation results. 2. The existing image-based crack intelligent recognition technology is still mainly based on pixel-level semantic similarity constraints. Due to the complex and diverse interference information such as lighting, shadows, background noise, etc. on the surface of the structure, the extracted crack results usually have a large number of false warnings, and the reliability of the results is low. However, cracks have significant structural differences compared to other environmental interference information, such as crack width gradients, irregular boundaries, and edge burrs. The existing technology is seriously insufficient in mining the prior information of the above-mentioned significant crack morphology and structural characteristics. Summary of the invention

[0004] The purpose of the present invention is to provide a method and system for identifying apparent cracks in optical image structures, so as to solve the problems existing in the above-mentioned prior art.

[0005] The technical solution of the present invention is as follows:

[0006] A method for identifying apparent cracks in optical image structures comprises the following steps:

[0007] S1. Obtain the apparent crack image of the structure and annotate the crack information to generate a data set;

[0008] S2. Generate a crack multi-structure feature map for the crack image, and generate the corresponding center line and key points for the crack annotation;

[0009] S3. Use the constructed spatial detail feature enhanced encoder to extract the multi-level spatial detail features of the cracks from the generated crack multi-structure feature map, and use the encoder pre-trained with large-scale data to extract the multi-scale semantic features of the cracks from the crack image;

[0010] S4. Construct a multi-level crack segmentation network with multi-encoding feature fusion, including a multi-encoder that integrates spatial detail features and deep semantic features, a module that integrates multi-scale spatial detail features and deep semantic features, and a multi-level segmentation decoder for crack key points, center lines, and regional surfaces. Adaptively integrate the multi-level spatial detail features and multi-scale semantic features obtained in S3 through the channel attention mechanism.

[0011] S5. Based on the network model and data set constructed in S1-S4, iteratively train the crack recognition model for joint optimization of point-line-surface tasks, and then use the model to identify cracks;

[0012] S6. Perform crack identification post-processing optimization to obtain the final crack identification result.

[0013] Furthermore, step S1 specifically includes the following sub-steps:

[0014] S1.1. Collecting image data of the surface of the structure based on an optical sensor device, wherein the sensor type is a high-resolution optical camera;

[0015] S1.2. The apparent crack information is annotated in the form of closed boundaries of the crack area, which are visually identified and annotated by personnel with professional knowledge. After the crack recognition model training is completed, it is quickly semi-automatically annotated by combining intelligent recognition results with a small amount of manual correction;

[0016] S1.3. Perform image quality inspection, brightness enhancement, image normalization, and annotation quality calibration;

[0017] S1.4. Constructing the sample set includes: cropping of images and corresponding crack annotations, random transformation of samples, and linear lifting and enhancement processing.

[0018] Furthermore, step S2 specifically includes the following sub-steps:

[0019] S2.1. For the constructed sample library, the structural information of its images is generated by Sobel gradient, high-frequency features based on convolution kernel, high-frequency features based on Hough transform, Laplace, Gaussian filtering, regularization filtering, mean filtering, and adaptive filtering, which is represented by S I , where S I ∈w×h×N, N is the type of operator;

[0020] S2.2. For the annotated crack label Lseg, the center line Lctr of the crack is obtained through the skeleton generation algorithm provided by the opencv library. The connection of the crack pixel neighborhood is calculated based on the crack center line. When the number of non-adjacent pixel neighboring points is greater than or equal to 2 and the formed angle is less than the specified value of 150°, the point is marked as the key point Lkey of the crack.

[0021] Furthermore, the encoder for spatial detail feature enhancement described in step S3 includes multiple convolution layers, upsampling layers and maximum pooling downsampling layers with convolution kernels of 3×3 and 1×1 respectively. First, semantic features of higher spatial and channel dimensions are extracted from the input multi-structure information through the convolution layer and the upsampling layer, thereby retaining the spatial detail information of the crack. Subsequently, high-resolution features with spatial resolutions of 1 times and 1 / 2 times of the original image are extracted through layer-by-layer convolution and downsampling, and the number of feature channels is 32 and 64 respectively.

[0022] Furthermore, in step S4, the multi-level segmentation decoders of the crack key points, center lines, and regional surfaces are respectively composed of convolution layers with convolution kernels of 3×3 and 1×1. Based on the features after multi-scale feature fusion, binary images with a channel number of 1 and the same spatial resolution as the original image are extracted, where the pixel values ​​represent the probability of being a crack, a center line, and a key point, respectively.

[0023] Furthermore, step S5 specifically includes the following sub-steps:

[0024] S5.1 Iteratively train the multi-task crack segmentation model through supervised learning methods; further design sample enhancement strategies during the training process, including random rotation, scaling, and cropping, to improve the generalization of the model through more diverse sample data;

[0025] S5.2 For the extracted crack area surface, optimization is performed through binary cross entropy loss and Dice similarity loss weighting, where the binary cross entropy loss function alleviates the imbalance problem between cracks and background through the positive sample weighting strategy;

[0026] S5.3 For the extracted crack centerline, the evaluation is performed by calculating the average distance between each pixel in the centerline prediction result and the nearest point in the true label;

[0027] S5.4 For the extracted crack key points, firstly calculate and sort the average probability of the pixels in the neighborhood where the identified key points are located, and then select the sorted k pre-selected points according to the number k of real key points in the label, and the k predicted points closest to each real key point for accuracy evaluation, and the evaluation method is the average value of the distance between the two closest points;

[0028] S5.5 The final loss function is the weighted sum of the corresponding loss functions of the crack area surface, center line, and key points.

[0029] Furthermore, step S6 specifically includes the following sub-steps:

[0030] The crack area surface obtained by the S6.1 model segmentation is first normalized by the sigmoid function to obtain a probability map with a value range of [0, 1];

[0031] S6.2 Probability by specifying a threshold Figure 2 The crack semantic map is obtained by value conversion. On this basis, the morphological processing algorithm provided by the open source OpenCV library is used to combine the obtained probability map to perform connectivity analysis and obtain the instance-level object of the crack.

[0032] S6.3 further post-processes the extracted crack results through algorithms such as hole filling and noise removal to obtain the final optimized crack identification results.

[0033] An optical imaging structure apparent crack recognition system, which implements the above recognition method, comprises:

[0034] Encoders pre-trained on large-scale datasets;

[0035] The encoder with enhanced spatial detail features maintains low model computational complexity while retaining the spatial information of small linear cracks by designing a lightweight extraction module of multi-scale high-resolution spatial detail features.

[0036] Multi-scale semantic and detail feature fusion module: In the decoding stage, a spatial detail feature and deep semantic feature adaptive fusion module based on the channel attention mechanism is constructed to enhance the expression ability of multi-scale crack features;

[0037] The crack "point-line-surface" multi-task segmentation and decoding module designs segmentation networks for corresponding tasks according to the unique prior constraints of cracks at different levels of points, lines, and regional surfaces, so as to realize simultaneous extraction of multiple tasks; the crack "point-line-surface" supervised optimization module designs corresponding loss functions for crack key points, center lines, and regional surfaces, and performs joint optimization.

[0038] The beneficial technical effects of the present invention are as follows:

[0039] In order to solve the problem that the semantic information of small cracks is lost due to multiple pooling and downsampling operations in the encoding stage of the existing crack recognition technology based on the deep learning framework, the extraction of small cracks on the surface of optical image structures is incomplete and inaccurate. A convolution module for enhancing crack spatial detail features is designed. This module extracts multiple information that characterizes the prior of crack structure obtained by operators such as Sobel, Laplace, and wavelet transform, retains high-resolution crack spatial detail features, and fuses them with multi-scale semantic features extracted by the pre-trained classical encoding network to improve the completeness and accuracy of small crack recognition.

[0040] The existing intelligent crack recognition technology is only based on the pixel-level cross entropy loss function, lacks the mining of the unique structure and morphological characteristics of cracks, and is difficult to resist the interference of complex environmental noise, resulting in unreliable model recognition results. A multi-task model with "point-line-surface" multi-level structural consistency constraints is designed. The model simultaneously extracts the key points, center lines and regional surfaces that characterize the cracks, and constructs corresponding loss assessment functions respectively. The reliability of crack recognition results is improved through feature migration and collaborative optimization between multiple tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a schematic diagram of the network model architecture of this technology;

[0042] Figure 2 This is a comparison chart of crack identification results after introducing each module. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of the present application clearer, specific embodiments will be described below in conjunction with the accompanying drawings.

[0044] refer to Figure 1 As shown, a method for identifying apparent cracks in optical image structures includes:

[0045] S1. Obtain the apparent crack image of the structure and annotate the crack information to generate a data set;

[0046] S2. Generate a crack multi-structure feature map for the crack image, and generate the corresponding center line and key points for the crack annotation;

[0047] S3. Use the constructed spatial detail feature enhanced encoder to extract the multi-level spatial detail features of the cracks from the generated crack multi-structure feature map, and use the encoder pre-trained with large-scale data to extract the multi-scale semantic features of the cracks from the crack image;

[0048] S4. Construct a multi-level crack segmentation network with multi-encoding feature fusion, including a multi-encoder that fuses spatial detail features and deep semantic features, and a multi-level segmentation decoder for crack key points, center lines, and regional surfaces; adaptively fuse the multi-level spatial detail features and multi-scale semantic features obtained in S3 through the channel attention mechanism;

[0049] S5. Based on the network model and data set constructed in S1-S4, iteratively train the crack recognition model for joint optimization of point-line-surface tasks, and then use the model to identify cracks;

[0050] S6. Perform post-processing optimization of the crack identification structure to obtain the final crack identification result.

[0051] Example:

[0052] The optical imaging structure apparent crack recognition system includes a structural feature construction module, a high spatial resolution feature encoding module, a pre-trained deep semantic feature encoding module, a multi-scale spatial detail and deep semantic feature adaptive fusion module, a crack "point-line-surface" multi-task decoding module and a "point-line-surface" multi-level joint optimization module.

[0053] First, image data of the surface of structures including but not limited to bridges, tunnels, roads, and water conservancy projects are collected using sensor equipment; the sensor type is generally a high-resolution optical RGB camera, and the collected image dimensions are W×H×3, where W and H represent the number of pixels occupied by the width and height of the image, respectively; personnel with professional knowledge visually interpret the image to identify cracks and mark semantic labels with closed boundaries;

[0054] The label form is generally a closed vector map composed of crack boundary points, which is converted into a single-channel image consistent with the image resolution through rasterization, wherein the pixel value of the crack area is 1, and the pixel value of other non-crack areas is 0; the image is preprocessed by quality inspection, brightness enhancement, etc., and the quality inspection of the crack annotation results is further combined with the image to ensure the accuracy of the crack annotation information; the image and the corresponding annotated image are cropped, including two methods: sliding window cropping and random cropping of the crack area, to improve the sampling rate of the crack area and alleviate the imbalance problem of positive and negative samples; the sliding window sampling overlap range is 64 pixels, and the cropped sample width (w) and height (h) are 512 pixels respectively. The cropped image and label sample set are represented by I and L respectively; for the convenience of model training and optimization, the sample data set is randomly divided into two independent subsets, the training set and the test set, at a ratio of 8:2, wherein the training set is used for model iterative training and optimization, and the test set is used for model accuracy evaluation;

[0055] The apparent crack recognition model of optical imaging structures includes a structural feature construction module, a high spatial resolution feature encoding module, a pre-trained deep semantic feature encoding module, a multi-scale spatial detail and deep semantic feature adaptive fusion module, a crack "point-line-surface" multi-task decoding module and a "point-line-surface" multi-level joint optimization module.

[0056] The image structure feature construction module is extracted from the grayscale image corresponding to the image through N operators such as but not limited to Sobel gradient, high-frequency features based on convolution kernel, high-frequency features based on Hough transform, Laplace, Gaussian filtering, regularization filtering, mean filtering, and adaptive filtering, which is represented by SI, where SI∈w×h×N;

[0057] The high spatial resolution feature encoding module is composed of a combination of multiple convolution layers with convolution kernels of 3×3 and 1×1, upsampling layers, and maximum pooling downsampling layers. The module first extracts semantic features of higher spatial and channel dimensions from the input image multi-structure features SI through convolution layers and upsampling layers, and retains the spatial detail information of the cracks.

[0058] The high spatial resolution feature encoding module extracts high-resolution features with the same and half spatial resolution of the original image through layer-by-layer convolution and downsampling. The number of feature channels is 32 and 64, respectively, denoted as FH1 and FH2. While enhancing the spatial details and semantic richness of crack features, lightweight module parameters are maintained to support accurate identification of small cracks.

[0059] The pre-trained deep semantic feature encoding module obtains multi-scale features based on the RegNet008 encoder. This technology selects three scale features with resolutions of 1 / 4, 1 / 8 and 1 / 16 of the original image to improve the accuracy of the model's cross-scene multi-scale crack extraction;

[0060] The multi-scale spatial detail and deep semantic feature adaptive fusion module upsamples and fuses the high spatial resolution detail features extracted from the image structure features and the multi-scale semantic features extracted from the original image by the large-scale pre-trained encoding network layer by layer. The corresponding scale semantic features in the decoding stage of the feature fusion process are adaptively fused through the channel attention mechanism;

[0061] The crack "point-line-surface" multi-task decoding module constructs the crack area surface, center line and key point decoding segmentation head network respectively based on the optimized features after multi-scale feature fusion, which is composed of convolution layers with convolution kernels of 3×3 and 1×1, and extracts normalized feature maps with a channel number of 1 and the same spatial resolution as the original image. The pixel values ​​represent the probability of being a crack, a center line, and a key point respectively.

[0062] The "point-line-surface" multi-level joint optimization module designs corresponding loss evaluation functions for the crack area surface, crack center line and crack key point respectively;

[0063] The crack area loss function is obtained by weighting the binary cross entropy loss and the Dice similarity loss, wherein the binary cross entropy loss function alleviates the problem of imbalance between cracks and background proportions through a positive sample weighting strategy; the crack centerline loss function is evaluated by calculating the average distance between each pixel in the centerline prediction result and the nearest point in the true label;

[0064] The crack key point loss function first calculates the average probability of the 8-neighborhood pixels where the identified key point is located and sorts them. Then, the sorted k pre-selected points are selected according to the number of real key points (k) in the label, and the k predicted points closest to each real key point are evaluated for accuracy. The evaluation method is the average distance between the two closest points. The final loss function is the weighted sum of the loss functions corresponding to the crack area surface, center line, and key point, and the weight coefficients are 0.6, 0.2, and 0.2 respectively.

[0065] Before model training, the crack label data L is preprocessed. The annotated crack label is represented as Lseg. The centerline label Lctr of the crack is obtained through the skeleton generation algorithm provided by the opencv library. The 8-neighborhood connection of the crack pixel is calculated based on the crack centerline. When the number of adjacent points of non-adjacent pixels is greater than or equal to 2 and the angle is less than 150°, the point is marked as the key point of the crack, represented by Lkey.

[0066] The input data of the model training process are batched images, labels and corresponding multi-channel image structure features, which are represented as: I∈B×w×h×3, L∈B×w×h×3 and SI∈B×w×h×N;

[0067] During the model training process, each time the sample data is loaded, enhancement processing is performed. The image and label data enhancement methods mainly include random scaling, rotation, flipping, color change, brightness change, and linear upscaling to improve the diversity of samples;

[0068] After the model training is completed, the test data set can be inferred and predicted. The input data of the model in the inference stage is only the surface image data of the structure;

[0069] In the model inference stage, only the crack area surface obtained by model segmentation is used as the final crack prediction result. First, the probability map with a value range of [0, 1] is obtained by normalizing it through the sigmoid function; then the probability map is normalized by specifying a threshold (generally 0.5). Figure 2The semantic map of cracks is obtained by quantization. On this basis, the morphological processing algorithm provided by the opencv open source library is used to combine the probability map for connectivity analysis to obtain the instance-level objects of cracks. Finally, the extracted crack results are further post-processed through algorithms such as hole filling and noise removal to obtain the final optimized crack recognition results.

[0070] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for identifying apparent cracks in optical image structures, characterized in that: The steps include: S1. Obtain the apparent crack image of the structure and annotate the crack information to generate a data set; S2. Generate a crack multi-structure feature map for the crack image, and generate the corresponding center line and key points for the crack annotation; S3. Use the constructed spatial detail feature enhanced encoder to extract the multi-level spatial detail features of the cracks from the generated crack multi-structure feature map, and use the encoder pre-trained with large-scale data to extract the multi-scale semantic features of the cracks from the crack image; S4. Construct a multi-level crack segmentation network with multi-encoding feature fusion, including a multi-encoder that integrates spatial detail features and deep semantic features, a module that integrates multi-scale spatial detail features and deep semantic features, and a multi-level segmentation decoder for crack key points, center lines, and regional surfaces. Adaptively integrate the multi-level spatial detail features and multi-scale semantic features obtained in S3 through the channel attention mechanism. S5. Based on the network model and data set constructed in S1-S4, iteratively train the crack recognition model for joint optimization of point-line-surface tasks, and then use the model to identify cracks; S6. Perform crack identification post-processing optimization to obtain the final crack identification result.

2. The optical image structure apparent crack recognition method according to claim 1, characterized in that: Step S1 specifically includes the following sub-steps: S1.

1. Collecting image data of the surface of the structure based on an optical sensor device, wherein the sensor type is a high-resolution optical camera; S1.

2. The apparent crack information is annotated in the form of closed boundaries of the crack area, which are visually identified and annotated by personnel with professional knowledge. After the crack recognition model training is completed, it is quickly semi-automatically annotated by combining intelligent recognition results with a small amount of manual correction; S1.

3. Perform image quality inspection, brightness enhancement, image normalization, and annotation quality calibration; S1.

4. Constructing the sample set includes: cropping of images and corresponding crack annotations, random transformation of samples, and linear lifting and enhancement processing.

3. The optical image structure apparent crack recognition method according to claim 1, characterized in that: Step S2 specifically includes the following sub-steps: S2.

1. For the constructed sample library, the structural information of its images is generated by Sobel gradient, high-frequency features based on convolution kernel, high-frequency features based on Hough transform, Laplace, Gaussian filtering, regularization filtering, mean filtering, and adaptive filtering, which is represented by S I , where S I ∈w×h×N, N is the type of operator; S2.

2. For the annotated crack label Lseg, the center line Lctr of the crack is obtained through the skeleton generation algorithm provided by the opencv library. The connection of the crack pixel neighborhood is calculated based on the crack center line. When the number of non-adjacent pixel neighboring points is greater than or equal to 2 and the formed angle is less than the specified value of 150°, the point is marked as the key point Lkey of the crack.

4. The optical image structure apparent crack recognition method according to claim 1, characterized in that: The encoder for spatial detail feature enhancement described in step S3 includes multiple convolution layers, upsampling layers, and maximum pooling downsampling layers with convolution kernels of 3×3 and 1×1, respectively. First, semantic features of higher spatial and channel dimensions are extracted from the input multi-structure information through the convolution layer and the upsampling layer, thereby retaining the spatial detail information of the crack. Subsequently, high-resolution features with spatial resolutions of 1 times and 1 / 2 times of the original image are extracted through layer-by-layer convolution and downsampling, and the number of feature channels is 32 and 64, respectively.

5. The optical image structure apparent crack recognition method according to claim 1, characterized in that: In step S4, the multi-level segmentation decoders of crack key points, center lines, and regional surfaces are respectively composed of convolution layers with convolution kernels of 3×3 and 1×1. Based on the features after multi-scale feature fusion, binary images with a channel number of 1 and the same spatial resolution as the original image are extracted, where the pixel values ​​represent the probability of being a crack, center line, and key point, respectively.

6. The optical image structure apparent crack recognition method according to claim 1, characterized in that: Step S5 specifically includes the following sub-steps: S5.1 Iteratively train the multi-task crack segmentation model through supervised learning methods; further design sample enhancement strategies during the training process, including random rotation, scaling, and cropping, to improve the generalization of the model through more diverse sample data; S5.2 For the extracted crack area surface, optimization is performed through binary cross entropy loss and Dice similarity loss weighting, where the binary cross entropy loss function alleviates the imbalance problem between cracks and background through the positive sample weighting strategy; S5.3 For the extracted crack centerline, the evaluation is performed by calculating the average distance between each pixel in the centerline prediction result and the nearest point in the true label; S5.4 For the extracted crack key points, firstly calculate and sort the average probability of the pixels in the neighborhood where the identified key points are located, and then select the sorted k pre-selected points according to the number k of real key points in the label, and the k predicted points closest to each real key point for accuracy evaluation, and the evaluation method is the average value of the distance between the two closest points; S5.5 The final loss function is the weighted sum of the corresponding loss functions of the crack area surface, center line, and key points.

7. The optical image structure apparent crack recognition method according to claim 1, characterized in that: Step S6 specifically includes the following sub-steps: The crack area surface obtained by the S6.1 model segmentation is first normalized by the sigmoid function to obtain a probability map with a value range of [0, 1]; S6.2 Binarize the probability map by specifying a threshold to obtain a crack semantic map. On this basis, the morphological processing algorithm provided by the open source opencv library is combined with the obtained probability map to perform connectivity analysis to obtain instance-level objects of cracks. S6.3 further post-processes the extracted crack results through algorithms such as hole filling and noise removal to obtain the final optimized crack identification results.

8. An optical imaging structure apparent crack recognition system, the system implements the method as claimed in claim 1, characterized in that: include: Encoders pre-trained on large-scale datasets; The encoder with enhanced spatial detail features maintains low model computational complexity while retaining the spatial information of small linear cracks by designing a lightweight extraction module of multi-scale high-resolution spatial detail features. Multi-scale semantic and detail feature fusion module: In the decoding stage, a spatial detail feature and deep semantic feature adaptive fusion module based on the channel attention mechanism is constructed to enhance the expression ability of multi-scale crack features; The crack "point-line-surface" multi-task segmentation and decoding module designs segmentation networks for corresponding tasks according to the unique prior constraints of cracks at different levels of points, lines, and regional surfaces, and realizes simultaneous extraction of multiple tasks; the crack "point-line-surface" supervised optimization module designs corresponding loss functions for crack key points, center lines, and regional surfaces, and performs joint optimization.

Images