A crack detection method based on cross-modal neural network with optical flow alignment

Abstract

The present invention proposes a crack detection method based on a cross-mode neural network with optical flow alignment. The method is based on a cross-mode semantic segmentation neural network FAC-Net with optical flow alignment. By fusing feature information of RGB images and depth images, the method can perform cross-modal crack detection. The method comprises the following steps: step S0, constructing a data set, training a cross-modal feature neural network FAC-Net, step S1, obtaining an RGB image and a depth image of a target to be detected; step S2, passing the RGB image and the depth image of the same target to be detected through a pre-trained cross-modal feature neural network FAC-Net to obtain a classification result image, wherein the classification result includes crack semantic pixels and background semantic pixels. The present invention can realize the fusion of feature information of RGB images and depth images, and perform cross-modal crack detection.

Description

Technical Field

[0001] The invention relates to the technical field of concrete structure crack recognition, and in particular to a crack detection method based on a cross-mode neural network with optical flow alignment. Background Art

[0002] Every year, non-human disasters such as floods, earthquakes, and typhoons occur from time to time. These disasters not only pose a great threat to human life, but also cause irreversible damage to concrete structures such as roads, bridges, and buildings. They may turn a good house into a dangerous house with a severely damaged structure, which will expose it to unpredictable risks. For example, a very important project in the identification of dangerous houses is the observation of building cracks. Currently, the exploration of dangerous houses is mostly manual exploration, which poses a huge risk to the life safety of the explorers. Therefore, in the process of finding and determining the location of concrete cracks, it is important to use a machine to collect images of the house site, find the cracks in the background, and mark the approximate crack locations in the images to ensure effective detection of concrete cracks under safe conditions.

[0003] With the rapid development of deep learning, especially the development of image processing, target detection and computer vision technology, image-based nondestructive testing technology has become a research hotspot for defect detection at home and abroad. Most detection methods use digital image processing technology and machine learning algorithms to detect some simple types of structural damage.

[0004] However, the existing technical solutions still have some problems that cannot be effectively solved in actual scenarios, especially in the case of complex environmental conditions, such as water stains and human smears that are easily confused with cracks. This is because the above methods all ignore the unique physical characteristics of cracks that are different from the background environment. The essential difference in their physical characteristics is that cracks are deeper, change faster, and have a larger amplitude than other surrounding environments.

[0005] Optical flow is an important method for motion image analysis. Its concept was first proposed by James J. Gibson in the 1940s. It refers to the speed of pattern motion in time-varying images. Because when an object is moving, the brightness pattern of its corresponding points on the image is also moving. The apparent motion of this image brightness pattern is the optical flow. Optical flow expresses the changes in the image. Since it contains information about the motion of the target, it can be used by the observer to determine the motion of the target. The definition of optical flow can be extended to the optical flow field, which refers to a two-dimensional (2D) instantaneous velocity field composed of all pixels in the image, in which the two-dimensional velocity vector is the projection of the three-dimensional velocity vector of the visible point in the scene on the imaging surface. Therefore, the optical flow not only contains the motion information of the observed object, but also contains rich information about the three-dimensional structure of the scene. Summary of the invention

[0006] The present invention proposes a crack detection method based on a cross-modal neural network with optical flow alignment, which can fuse RGB image and depth image feature information and perform cross-modal crack detection.

[0007] The present invention adopts the following technical solutions.

[0008] A crack detection method based on a cross-modal neural network with optical flow alignment, the method is based on a cross-modal semantic segmentation neural network FAC-Net with optical flow alignment, and can perform cross-modal crack detection by fusing RGB image and depth image feature information, and includes the following steps:

[0009] Step S0: construct a data set and train a cross-modal feature neural network FAC-Net.

[0010] Step S1, obtaining an RGB image and a depth image of a target to be detected;

[0011] Step S2, the RGB image and the depth image of the same target to be detected are passed through a pre-trained cross-modal feature neural network FAC-Net to obtain a classification result image, where the classification result includes crack semantic pixels and background semantic pixels;

[0012] The crack area can be identified by the crack semantic pixels and background semantic pixels.

[0013] The specific method of step S0 is as follows:

[0014] Step S01: photographing a concrete object using a camera having depth perception technology and an RGB sensor to obtain original image data of the concrete object to form a data set;

[0015] Step S02: classify and mark the data set into crack pixel categories and background pixel categories;

[0016] Step S03, divide the data set into a training set and a test set according to the proportion;

[0017] Step S04: training the FAC-Net neural network using the data set;

[0018] Step S05: deploy the trained neural network model on the back-end device for crack detection.

[0019] In step S01, the concrete objects include the concrete structure surfaces of roads, bridges and buildings;

[0020] In step S02, the categories of crack pixels and background pixels can be expanded by data enhancement;

[0021] In step S03, the image data set includes RGB image data, depth image data and corresponding label data.

[0022] In step S1, a camera having depth perception technology and an RGB sensor is used to obtain an RGB image and a depth image of the target to be detected. The shooting method is not limited when obtaining the image, and the obtained image is clear and can be regarded as a qualified standard.

[0023] In step S2, the captured RGB image and depth image are transmitted to the back-end device through the network or wired mode. The back-end device inputs the RGB image and depth image into the trained FAC-Net neural network to obtain the corresponding concrete crack detection result.

[0024] The FAC-Net network structure is an encoder-decoder structure; the encoder has two branches and one fusion area, and the branches are RGB branch and Depth branch respectively;

[0025] After each branch unit of the FAC-Net network, the RGB features and Depth features are input into the FAA fusion module to obtain the fused features, and then the fused features are added to the original branch features; after 4 units, the highest-level fused features are input into the feature pyramid PPM to further extract features;

[0026] The decoder uses the FA module to convert the low-resolution high-semantic feature image to the high-resolution low-semantic feature image, and obtains the output image of the classification result through three modules;

[0027] All branches of the encoder use the classic resnet50 network as the backbone network;

[0028] In the encoder backbone network structure, the first unit has a convolutional layer, a maximum pooling layer, and a comprehensive convolutional layer in sequence; the second unit has a three-layer structure of a comprehensive convolutional layer; and the third unit has a three-layer structure of a comprehensive convolutional layer.

[0029] In the encoder backbone network structure, the convolution layer of the first unit includes 64 convolution kernels, the convolution kernel size is 7×7, the step size is 2, and the padding is 3; the pooling kernel size of the maximum pooling layer of the first unit is 3×3, and the step size is 2; the comprehensive convolution layer of the first unit is a three-layer structure, wherein the first convolution layer includes 64 convolution kernels, the convolution kernel size is 1×1, the second convolution layer includes 64 convolution kernels, the convolution kernel size is 3×3, and the padding is 1, and the third convolution layer includes 256 convolution kernels, and the convolution kernel size is 1×1;

[0030] The comprehensive convolution layer of the second unit has a three-layer structure. The first convolution layer includes 128 convolution kernels with a convolution kernel size of 1×1, the second convolution layer includes 128 convolution kernels with a convolution kernel size of 3×3 and a padding of 1, and the third convolution layer includes 512 convolution kernels with a convolution kernel size of 1×1.

[0031] The comprehensive convolution layer of the third unit has a three-layer structure. The first convolution layer includes 256 convolution kernels with a convolution kernel size of 1×1, the second convolution layer includes 256 convolution kernels with a convolution kernel size of 3×3 and a padding of 1, and the third convolution layer includes 1024 convolution kernels with a convolution kernel size of 1×1.

[0032] The comprehensive convolution layer of the fourth unit has a three-layer structure. The first convolution layer includes 512 convolution kernels with a convolution kernel size of 1×1, the second convolution layer includes 512 convolution kernels with a convolution kernel size of 3×3 and a padding of 1, and the third convolution layer includes 2048 convolution kernels with a convolution kernel size of 1×1.

[0033] In the detection method, the image information of the depth image and the RGB image are aligned and then fused using the optical flow characteristics of the FAA fusion module;

[0034] The FAA fusion module includes an RGB branch and a Depth branch;

[0035] When the RGB branch is working, the RGB features of the image are expressed in the form of H×W×C, that is, height×width×number of channels. The RGB features first pass through the first convolutional layer of the RGB branch. The convolution kernel size of the first convolutional layer is 1×1, the step size is 1, and the padding is 0. After being processed by the first convolutional layer, the size of the image feature map becomes H×W×1;

[0036] When the Depth branch is working, the representation of the Depth feature of the image is H×W×C, that is, height×width×number of channels. The Depth feature passes through the first convolution layer of the Depth branch, and the two features after convolution are spliced ​​in the channel direction to become H×W×2. Then, after passing through the fusion convolution layer with a convolution kernel size of 3×3, a step size of 1, and a padding of 1, a fusion optical flow map of H×W×4 is obtained;

[0037] The fused optical flow map has two layers of RGB branch optical flow maps and the other two layers of Depth branch optical flow maps. The fused optical flow map is then split into two H×W×2 optical flow maps. The original H×W×C RGB features and the original H×W×C Depth features are firstly feature-corrected through the corresponding optical flow maps. The corrected H×W×C RGB features and H×W×C Depth features are then input into the relevant mechanism of the spatial attention module to further extract the fused features. The specific method is that the two features are first averaged and max-pooled along the channel direction to generate four features of size H×W×1, and then the four features are concatenated into a H×W×4 feature along the channel direction, and then a convolution layer with a convolution kernel size of 3×3, a step size of 1, and a padding of 1 and a Sigmoid activation function are passed through to obtain a H×W×2 spatial attention weight matrix.

[0038] Finally, the H×W×2 spatial attention weight matrix is ​​split into two H×W×1 spatial attention weight matrices; the original H×W×C RGB features and the original H×W×C Depth features are multiplied with the corresponding weight matrices, and then the two reweighted features are added. The summed features are finally output through the ReLU activation function to obtain the H×W×C FAA fusion features; the input RGB features, Depth features and output FAA fusion features are all of the same size.

[0039] In the detection method, the FA module uses the jump connection of the encoder and the decoder and uses the optical flow correction method to predict the flow field to flow the low-resolution high-semantic feature image to the high-resolution low-semantic feature image, so as to retain the detailed features of the crack as much as possible. The specific method is: the number of channels of the low-resolution high-semantic feature image and the high-resolution low-semantic feature image are compressed to 2 through the first convolution layer with a convolution kernel size of 1×1, a step size of 1, and a padding of 0, and then the low-resolution high-semantic feature image of H / 2×W / 2×2 is bilinearly interpolated to make its size the same as the high-resolution low-semantic feature image of H×W×2, and then the two are spliced ​​in the channel direction;

[0040] After splicing, it passes through two fused convolution layers with a convolution kernel size of 3×3, a step size of 1, and a padding of 1 to obtain an H×W×2 optical flow map and an H×W×1 spatial attention weight matrix; the low-semantic feature image is multiplied by the spatial attention weight matrix, the high-semantic image is feature corrected through the optical flow map, and then the two are added to obtain the merged feature.

[0041] The optical flow correction method is used to predict the flow field to convert the low-resolution high-semantic feature image to the high-resolution low-semantic feature image. In the prediction process of the neural network, the feature size of the image changes as follows: an RGB image with a size of 512×512×3 and a Depth image with a size of 512×512×1 are input for detection. After the first unit fusion, an image fusion feature with a size of 128×128×256 is obtained. After the fusion feature is added to the original feature, a new RGB feature with a size of 128×128×256 and a new Depth; After the second unit, a fusion feature with a size of 64×64×512 is obtained. After the fusion feature is added to the original feature, a new RGB feature and a new Depth feature with a size of 64×64×512 are obtained; After the third unit, a fusion feature with a size of 32×32×1024 is obtained. After the fusion feature is added to the original feature, a new RGB feature and a new Depth feature with a size of 32×32×1024 are obtained; After the fourth unit, a fusion feature with a size of 16×16×2048 is obtained;

[0042] The fused features with a size of 16×16×2048 are further extracted through the feature pyramid to obtain new fused features with a size of 16×16×2048; the new fused features with a size of 16×16×2048 and the fused features with a size of 32×32×1024 are input into the FA module for optical flow correction to obtain optical flow aligned fused features with a size of 32×32×1024; the optical flow aligned fused features with a size of 32×32×1024 and the fused features with a size of 64×64×512 are input into the FA module for optical flow correction to obtain new optical flow aligned fused features with a size of 64×64×512; The optical flow alignment fusion features with a size of 64×64×512 and the fusion features with a size of 128×128×256 are input into the FA module for optical flow correction to obtain new optical flow alignment fusion features with a size of 128×128×256; the optical flow alignment fusion features with a size of 128×128×256 are bilinearly interpolated to enlarge their size to 512×512×256; the optical flow alignment fusion features with a size of 512×512×256 are passed through a convolution layer with a convolution kernel size of 1×1 and a stride of 1 to obtain the final predicted image of 512×512×1. The pixel semantic information of the predicted image is divided into two categories: crack pixels and background pixels.

[0043] In step S0, a pre-prepared crack image dataset is sent to a cross-modal feature neural network FAC-Net for training. The crack image dataset includes N crack RGB images and N corresponding crack depth images. The dataset is divided into a test set and a training set in a ratio of 2:8. The role of the training set is to train FAC-Net, and the role of the test set is to observe the accuracy of the neural network model in identifying cracks during the training process.

[0044] Three indicators are set as the comparison benchmarks for the observed neural network model, namely precision, recall, and F1 score; the indicator calculation formula is:

[0045] The parameter TP, i.e. the number of occurrences that are considered to be true and are indeed true;

[0046] Parameter FP, which is the number of occurrences that are considered true but are actually false;

[0047] Parameter FN, the number of occurrences that are considered false but are actually true;

[0048] Parameter TN, the number of occurrences that are considered false and are indeed true;

[0049] Then precision = TP / (TP+FP)

[0050] Recall = TP / (TP+FN)

[0051] F1 score = 2 × precision × recall / (precision + recall).

[0052] In the above scheme, FAA fusion module is the Chinese abbreviation of Flow Alignment Attention Module; Feature Pyramid PPM is the Chinese abbreviation of Pyramid Pooling Module; FA module is the Chinese abbreviation of Flow Alignment Module.

[0053] Compared with the prior art, the technical advantages of the present invention are:

[0054] 1. Use RGB images and depth images as data to achieve cross-modal crack detection.

[0055] Since there are different physical properties between cracks and confusing backgrounds, the essential difference is that cracks are deeper, have faster depth changes, and have larger amplitudes than other surrounding environments. The present invention uses a cross-modal detection method that fuses RGB images and depth images. Due to the complementarity of RGB images and depth images, the combination of the color semantic information of RGB images and the depth semantic information of depth images can better identify the difference between cracks and easily confusing background environments than using only the color semantic information of RGB images.

[0056] 2. Propose a method to use optical flow alignment to align RGB information and depth information.

[0057] Because when taking depth images and RGB images, the depth images may have objective influences such as blurred edges and misalignment with RGB images, resulting in inconsistent information areas expressed by the depth images and RGB images. If it is simply added and fused, it may mislead the neural network for feature extraction of crack edge information. The present invention uses the characteristics of optical flow to design the FAA fusion module structure, aiming to align the information areas expressed by the depth image and the RGB image by optical flow alignment, so that better detail feature information can be obtained when the two are fused.

[0058] The present invention can perform real-time crack marking and detection on walls and floors of factories, houses, commercial buildings and other buildings, and is mainly used in dangerous building exploration, urban planning and other fields.

[0059] The present invention realizes a real-time crack semantic segmentation algorithm based on target detection. The algorithm adds a real-time feature extraction module on the basis of the YOLOv5 target detection algorithm. When the size of the region of interest predicted by the target detection is less than the threshold, the Ostu algorithm is used to quickly detect the small region of interest. When the size of the region of interest is greater than the threshold, the real-time semantic segmentation network FANet network is used to perform real-time slice fusion detection on the large region of interest. The features detected in each region of interest are mapped to the original image position to form the final crack feature map, so as to achieve the purpose of detecting the specific position of the crack on the basis of ensuring real-time and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0061] Attached Figure 1 It is a schematic diagram of the structure of the FAC-Net network;

[0062] Attached Figure 2 It is a schematic diagram of the structure of the FAA fusion module;

[0063] Attached Figure 3 It is a schematic diagram of the structure of the spatial attention module;

[0064] Attached Figure 4 It is a schematic diagram of the structure of the FA module;

[0065] Attached Figure 5 It is a schematic diagram of the process of the present invention;

[0066] Attached Figure 6 This is an example image of the crack image dataset;

[0067] Attached Figure 7 It is a schematic diagram of the comparison between the target image to be detected and the detection result. DETAILED DESCRIPTION

[0068] As shown in the figure, a crack detection method based on a cross-mode neural network with optical flow alignment is provided. The method is based on a cross-mode semantic segmentation neural network FAC-Net with optical flow alignment. By fusing RGB image and depth image feature information, crack detection can be performed across modalities. The method includes the following steps:

[0069] Step S0: construct a data set and train a cross-modal feature neural network FAC-Net.

[0070] Step S1, obtaining an RGB image and a depth image of a target to be detected;

[0071] Step S2, the RGB image and the depth image of the same target to be detected are passed through a pre-trained cross-modal feature neural network FAC-Net to obtain a classification result image, where the classification result includes crack semantic pixels and background semantic pixels;

[0072] The specific method of step S0 is as follows:

[0073] Step S01: photographing a concrete object using a camera having depth perception technology and an RGB sensor to obtain original image data of the concrete object to form a data set;

[0074] Step S02: classify and mark the data set into crack pixel categories and background pixel categories;

[0075] Step S03, divide the data set into a training set and a test set according to the proportion;

[0076] Step S04: training the FAC-Net neural network using the data set;

[0077] Step S05: deploy the trained neural network model on the back-end device for crack detection.

[0078] In step S01, the concrete objects include the concrete structure surfaces of roads, bridges and buildings;

[0079] In step S02, the categories of crack pixels and background pixels can be expanded by data enhancement;

[0080] In step S03, the image data set includes RGB image data, depth image data and corresponding label data.

[0081] In step S1, a camera having depth perception technology and an RGB sensor is used to obtain an RGB image and a depth image of the target to be detected. The shooting method is not limited when obtaining the image, and the obtained image is clear and can be regarded as a qualified standard.

[0082] In step S2, the captured RGB image and depth image are transmitted to the back-end device through the network or wired mode. The back-end device inputs the RGB image and depth image into the trained FAC-Net neural network to obtain the corresponding concrete crack detection result.

[0083] like Figure 1 As shown, the FAC-Net network structure is an encoder-decoder structure; the encoder has two branches and one fusion area, and the branches are RGB branch and Depth branch respectively;

[0084] After each branch unit of the FAC-Net network, the RGB features and Depth features are input into the FAA fusion module to obtain the fused features, and then the fused features are added to the original branch features; after 4 units, the highest-level fused features are input into the feature pyramid PPM to further extract features;

[0085] The decoder uses the FA module to convert the low-resolution high-semantic feature image to the high-resolution low-semantic feature image, and obtains the output image of the classification result through three modules;

[0086] All branches of the encoder use the classic resnet50 network as the backbone network;

[0087] In the encoder backbone network structure, the first unit has a convolutional layer, a maximum pooling layer, and a comprehensive convolutional layer in sequence; the second unit has a three-layer structure of a comprehensive convolutional layer; and the third unit has a three-layer structure of a comprehensive convolutional layer.

[0088] In the encoder backbone network structure, the convolution layer of the first unit includes 64 convolution kernels, the convolution kernel size is 7×7, the step size is 2, and the padding is 3; the pooling kernel size of the maximum pooling layer of the first unit is 3×3, and the step size is 2; the comprehensive convolution layer of the first unit is a three-layer structure, wherein the first convolution layer includes 64 convolution kernels, the convolution kernel size is 1×1, the second convolution layer includes 64 convolution kernels, the convolution kernel size is 3×3, and the padding is 1, and the third convolution layer includes 256 convolution kernels, and the convolution kernel size is 1×1;

[0089] The comprehensive convolution layer of the second unit has a three-layer structure. The first convolution layer includes 128 convolution kernels with a convolution kernel size of 1×1, the second convolution layer includes 128 convolution kernels with a convolution kernel size of 3×3 and a padding of 1, and the third convolution layer includes 512 convolution kernels with a convolution kernel size of 1×1.

[0090] The comprehensive convolution layer of the third unit has a three-layer structure. The first convolution layer includes 256 convolution kernels with a convolution kernel size of 1×1, the second convolution layer includes 256 convolution kernels with a convolution kernel size of 3×3 and a padding of 1, and the third convolution layer includes 1024 convolution kernels with a convolution kernel size of 1×1.

[0091] The comprehensive convolution layer of the fourth unit has a three-layer structure. The first convolution layer includes 512 convolution kernels with a convolution kernel size of 1×1, the second convolution layer includes 512 convolution kernels with a convolution kernel size of 3×3 and a padding of 1, and the third convolution layer includes 2048 convolution kernels with a convolution kernel size of 1×1.

[0092] like Figure 2 As shown, in the detection method, the image information of the depth image and the RGB image are aligned and then fused using the optical flow characteristics of the FAA fusion module;

[0093] The FAA fusion module includes an RGB branch and a Depth branch;

[0094] When the RGB branch is working, the RGB features of the image are expressed in the form of H×W×C, that is, height×width×number of channels. The RGB features first pass through the first convolutional layer of the RGB branch. The convolution kernel size of the first convolutional layer is 1×1, the step size is 1, and the padding is 0. After being processed by the first convolutional layer, the size of the image feature map becomes H×W×1;

[0095] When the Depth branch is working, the representation of the Depth feature of the image is H×W×C, that is, height×width×number of channels. The Depth feature passes through the first convolution layer of the Depth branch, and the two features after convolution are spliced ​​in the channel direction to become H×W×2. Then, after passing through the fusion convolution layer with a convolution kernel size of 3×3, a step size of 1, and a padding of 1, a fusion optical flow map of H×W×4 is obtained;

[0096] The fused optical flow map has two layers of RGB branch optical flow maps and the other two layers of Depth branch optical flow maps. Figure 3As shown in the figure, the fused optical flow map is split into two H×W×2 optical flow maps, and the original H×W×C RGB features and the original H×W×C Depth features are first corrected through the corresponding optical flow maps; then the corrected H×W×C RGB features and H×W×C Depth features are input into the relevant mechanism of the spatial attention module to further extract the fused features. The specific method is that the two features are first averaged and max-pooled along the channel direction to generate four features of size H×W×1, and then the four features are concatenated into a H×W×4 feature along the channel direction, and then the convolution layer with a convolution kernel size of 3×3, a step size of 1, and a padding of 1 and a Sigmoid activation function are passed to obtain a H×W×2 spatial attention weight matrix;

[0097] Finally, the H×W×2 spatial attention weight matrix is ​​split into two H×W×1 spatial attention weight matrices; the original H×W×C RGB features and the original H×W×C Depth features are multiplied with the corresponding weight matrices, and then the two reweighted features are added. The summed features are finally output through the ReLU activation function to obtain the H×W×C FAA fusion features; the input RGB features, Depth features and output FAA fusion features are all of the same size.

[0098] like Figure 4 As shown, in the detection method, the FA module uses the jump connection of the encoder and the decoder, and uses the optical flow correction method to predict the flow field to flow the low-resolution high-semantic feature image to the high-resolution low-semantic feature image, so as to retain the detailed features of the crack as much as possible. The specific method is: the number of channels of the low-resolution high-semantic feature image and the high-resolution low-semantic feature image are compressed to 2 through the first convolution layer with a convolution kernel size of 1×1, a step size of 1, and a padding of 0, and then the low-resolution high-semantic feature image of H / 2×W / 2×2 is bilinearly interpolated to make its size the same as the high-resolution low-semantic feature image of H×W×2, and then the two are spliced ​​in the channel direction;

[0099] After splicing, it passes through two fused convolution layers with a convolution kernel size of 3×3, a step size of 1, and a padding of 1 to obtain an H×W×2 optical flow map and an H×W×1 spatial attention weight matrix; the low-semantic feature image is multiplied by the spatial attention weight matrix, the high-semantic image is feature corrected through the optical flow map, and then the two are added to obtain the merged feature.

[0100] The optical flow correction method is used to predict the flow field to convert the low-resolution high-semantic feature image to the high-resolution low-semantic feature image. In the prediction process of the neural network, the feature size of the image changes as follows: an RGB image with a size of 512×512×3 and a Depth image with a size of 512×512×1 are input for detection. After the first unit fusion, an image fusion feature with a size of 128×128×256 is obtained. After the fusion feature is added to the original feature, a new RGB feature with a size of 128×128×256 and a new Depth; After the second unit, a fusion feature with a size of 64×64×512 is obtained. After the fusion feature is added to the original feature, a new RGB feature and a new Depth feature with a size of 64×64×512 are obtained; After the third unit, a fusion feature with a size of 32×32×1024 is obtained. After the fusion feature is added to the original feature, a new RGB feature and a new Depth feature with a size of 32×32×1024 are obtained; After the fourth unit, a fusion feature with a size of 16×16×2048 is obtained;

[0101] The fused features with a size of 16×16×2048 are further extracted through the feature pyramid to obtain new fused features with a size of 16×16×2048; the new fused features with a size of 16×16×2048 and the fused features with a size of 32×32×1024 are input into the FA module for optical flow correction to obtain optical flow aligned fused features with a size of 32×32×1024; the optical flow aligned fused features with a size of 32×32×1024 and the fused features with a size of 64×64×512 are input into the FA module for optical flow correction to obtain new optical flow aligned fused features with a size of 64×64×512; The optical flow alignment fusion features with a size of 64×64×512 and the fusion features with a size of 128×128×256 are input into the FA module for optical flow correction to obtain new optical flow alignment fusion features with a size of 128×128×256; the optical flow alignment fusion features with a size of 128×128×256 are bilinearly interpolated to enlarge their size to 512×512×256; the optical flow alignment fusion features with a size of 512×512×256 are passed through a convolution layer with a convolution kernel size of 1×1 and a stride of 1 to obtain the final predicted image of 512×512×1. The pixel semantic information of the predicted image is divided into two categories: crack pixels and background pixels.

[0102] In step S0, a pre-prepared crack image dataset is sent to a cross-modal feature neural network FAC-Net for training. The crack image dataset includes N crack RGB images and N corresponding crack depth images. The dataset is divided into a test set and a training set in a ratio of 2:8. The role of the training set is to train FAC-Net, and the role of the test set is to observe the accuracy of the neural network model in identifying cracks during the training process.

[0103] Three indicators are set as the comparison benchmarks for the observed neural network model, namely precision, recall, and F1 score; the indicator calculation formula is:

[0104] The parameter TP, i.e. the number of occurrences that are considered to be true and are indeed true;

[0105] Parameter FP, which is the number of occurrences that are considered true but are actually false;

[0106] Parameter FN, the number of occurrences that are considered false but are actually true;

[0107] Parameter TN, the number of occurrences that are considered false and are indeed true;

[0108] Then precision = TP / (TP+FP)

[0109] Recall = TP / (TP+FN)

[0110] F1 score = 2 × precision × recall / (precision + recall).

[0111] The following table shows the comparison results of crack detection indicators of the model FAC-Net described in the present invention and the classic semantic segmentation network U-Net under different environmental backgrounds:

[0112]

[0113] It can be seen from the data in the table that the new cross-modal neural network designed in the present invention, the cross-modal semantic segmentation neural network FAC-Net based on optical flow alignment, can better identify the feature differences between cracks and easily confused backgrounds under different environmental backgrounds, thereby achieving a better crack detection effect and having strong robustness.

[0114] Figure 7 There are pixel areas identified as cracks in the middle of the left and right parts of the image. It can be seen that when the background color is complex, FAC-Net can still eliminate various confusing information and accurately detect the location of the cracks.

[0115] In the above content, FAA fusion module is the Chinese abbreviation of Flow Alignment Attention Module; Feature Pyramid PPM is the Chinese abbreviation of Pyramid Pooling Module; FA module is the Chinese abbreviation of Flow Alignment Module.

Claims

1. A crack detection method based on a cross-mode neural network with optical flow alignment, characterized in that: The method is based on a cross-modal semantic segmentation neural network FAC-Net with optical flow alignment, which can perform cross-modal crack detection by fusing RGB image and depth image feature information, and includes the following steps: Step S0: construct a data set and train a cross-modal feature neural network FAC-Net. Step S1, obtaining an RGB image and a depth image of a target to be detected; Step S2, the RGB image and the depth image of the same target to be detected are passed through a pre-trained cross-modal feature neural network FAC-Net to obtain a classification result image, where the classification result includes crack semantic pixels and background semantic pixels; The specific method of step S0 is as follows: Step S01: photographing a concrete object using a camera having depth perception technology and an RGB sensor to obtain original image data of the concrete object to form a data set; Step S02: classify and mark the data set into crack pixel categories and background pixel categories; Step S03, divide the data set into a training set and a test set according to the proportion; Step S04, using the training set to train the FAC-Net neural network; Step S05: deploying the trained neural network model on the back-end device for crack detection; The FAC-Net network structure is an encoder-decoder structure; the encoder has two branches and one fusion area, and the branches are RGB branch and Depth branch respectively; After each branch unit of the FAC-Net network, the RGB feature and the Depth feature are input into the FAA fusion module to obtain the fused feature, and then the fused feature is added to the original branch feature; After 4 units, the highest-level fusion features are input into the feature pyramid PPM to further extract features; The decoder uses the FA module to convert the low-resolution high-semantic feature image to the high-resolution low-semantic feature image, and obtains the output image of the classification result through three modules; All branches of the encoder use the classic resnet50 network as the backbone network; In the encoder backbone network structure, the first unit has a convolutional layer, a maximum pooling layer, and a comprehensive convolutional layer in sequence; the second unit is a three-layer structure of a comprehensive convolutional layer; The third unit is a three-layer structure of comprehensive convolutional layers; In the detection method, the image information of the depth image and the RGB image are aligned and then fused using the optical flow characteristics of the FAA fusion module; The FAA fusion module includes an RGB branch and a Depth branch; When the RGB branch is working, the RGB features of the image are expressed in the form of H×W×C, that is, height×width×number of channels. The RGB features first pass through the first convolutional layer of the RGB branch. The convolution kernel size of the first convolutional layer is 1×1, the step size is 1, and the padding is 0. After being processed by the first convolutional layer, the size of the image feature map becomes H×W×1; When the Depth branch is working, the representation of the Depth feature of the image is H×W×C, that is, height×width×number of channels. The Depth feature passes through the first convolution layer of the Depth branch, and the two features after convolution are spliced ​​in the channel direction to become H×W×2. Then, after passing through the fusion convolution layer with a convolution kernel size of 3×3, a step size of 1, and a padding of 1, a fusion optical flow map of H×W×4 is obtained; The fused optical flow map has two layers of RGB branch optical flow maps and the other two layers of Depth branch optical flow maps. The fused optical flow map is then split into two H×W×2 optical flow maps. The original H×W×C RGB features and the original H×W×C Depth features are firstly rectified through the corresponding optical flow maps respectively. Then, the rectified H×W×C RGB features and H×W×C Depth features are input into the relevant mechanism of the spatial attention module to further extract the fusion features. The specific method is that the two features are first averaged and max-pooled along the channel direction to generate four features of size H×W×1, and then the four features are concatenated into a H×W×4 feature along the channel direction, and then passed through a convolution layer with a convolution kernel size of 3×3, a stride of 1, and a padding of 1 and a Sigmoid activation function to obtain a H×W×2 spatial attention weight matrix; Finally, the H×W×2 spatial attention weight matrix is ​​split into 2 H×W×1 spatial attention weight matrices; After multiplying the original H×W×C RGB features and the original H×W×C Depth features with the corresponding weight matrix, the two reweighted features are added together, and the resulting summed features are finally outputted as H×W×C FAA fusion features after the ReLU activation function. Make the input RGB features, Depth features and output FAA fusion features all of the same size.

2. The crack detection method based on cross-mode neural network with optical flow alignment according to claim 1 is characterized in that: In step S01, the concrete objects include the concrete structure surfaces of roads, bridges and buildings; In step S02, the categories of crack pixels and background pixels can be expanded by data enhancement; In step S03, the image data set includes RGB image data, depth image data and corresponding label data.

3. The crack detection method based on cross-mode neural network with optical flow alignment according to claim 1 is characterized in that: In step S1, a camera having depth perception technology and an RGB sensor is used to obtain an RGB image and a depth image of the target to be detected. The image is obtained in any shooting method, and the obtained image is clear and can be regarded as a qualified standard.

4. The crack detection method based on cross-mode neural network with optical flow alignment according to claim 3 is characterized by: In step S2, the captured RGB image and depth image are transmitted to the back-end device through the network or wired mode. The back-end device inputs the RGB image and depth image into the trained FAC-Net neural network to obtain the corresponding concrete crack detection result.

5. The crack detection method based on cross-mode neural network with optical flow alignment according to claim 1 is characterized in that: In the encoder backbone network structure, the convolution layer of the first unit includes 64 convolution kernels, the convolution kernel size is 7×7, the step size is 2, and the padding is 3; the pooling kernel size of the maximum pooling layer of the first unit is 3×3, and the step size is 2; the comprehensive convolution layer of the first unit is a three-layer structure, wherein the first convolution layer includes 64 convolution kernels, the convolution kernel size is 1×1, the second convolution layer includes 64 convolution kernels, the convolution kernel size is 3×3, and the padding is 1, and the third convolution layer includes 256 convolution kernels, and the convolution kernel size is 1×1; The comprehensive convolution layer of the second unit has a three-layer structure. The first convolution layer includes 128 convolution kernels with a convolution kernel size of 1×1, the second convolution layer includes 128 convolution kernels with a convolution kernel size of 3×3 and a padding of 1, and the third convolution layer includes 512 convolution kernels with a convolution kernel size of 1×1. The comprehensive convolution layer of the third unit has a three-layer structure. The first convolution layer includes 256 convolution kernels with a convolution kernel size of 1×1, the second convolution layer includes 256 convolution kernels with a convolution kernel size of 3×3 and a padding of 1, and the third convolution layer includes 1024 convolution kernels with a convolution kernel size of 1×1. The comprehensive convolution layer of the fourth unit has a three-layer structure. The first convolution layer includes 512 convolution kernels with a convolution kernel size of 1×1, the second convolution layer includes 512 convolution kernels with a convolution kernel size of 3×3 and a padding of 1, and the third convolution layer includes 2048 convolution kernels with a convolution kernel size of 1×1.

6. The crack detection method based on cross-mode neural network with optical flow alignment according to claim 1 is characterized by: In the detection method, the FA module uses the jump connection of the encoder and the decoder and uses the optical flow correction method to predict the flow field to flow the low-resolution high-semantic feature image to the high-resolution low-semantic feature image, so as to retain the detailed features of the crack as much as possible. The specific method is: the number of channels of the low-resolution high-semantic feature image and the high-resolution low-semantic feature image are compressed to 2 through the first convolution layer with a convolution kernel size of 1×1, a step size of 1, and a padding of 0, and then the low-resolution high-semantic feature image of H / 2×W / 2×2 is bilinearly interpolated to make its size the same as the high-resolution low-semantic feature image of H×W×2, and then the two are spliced ​​in the channel direction; After splicing, two fused convolution layers with a convolution kernel size of 3×3, a step size of 1, and a padding of 1 are respectively used to obtain an H×W×2 optical flow map and an H×W×1 spatial attention weight matrix; The low-semantic feature image is multiplied by the spatial attention weight matrix, the high-semantic image is feature corrected through the optical flow map, and then the two are added together to obtain the merged feature.

7. The crack detection method based on cross-mode neural network with optical flow alignment according to claim 6 is characterized by: The optical flow correction method is used to predict the flow field to convert the low-resolution high-semantic feature image to the high-resolution low-semantic feature image. In the prediction process of the neural network, the feature size of the image changes as follows: an RGB image with a size of 512×512×3 and a Depth image with a size of 512×512×1 are input for detection, and an image fusion feature with a size of 128×128×256 is obtained after the first unit is fused. After the fusion feature is added to the original feature, a new RGB feature and a new Depth with a size of 128×128×256 are obtained. After the second unit, a fusion feature with a size of 64×64×512 is obtained. After the fusion feature is added to the original feature, a new RGB feature and a new Depth feature with a size of 64×64×512 are obtained. After the third unit, a fusion feature with a size of 32×32×1024 is obtained. After the fusion feature is added to the original feature, a new RGB feature and a new Depth feature with a size of 32×32×1024 are obtained. After the fourth unit, the fusion feature with a size of 16×16×2048 is obtained; The fused features with a size of 16×16×2048 are further extracted through the feature pyramid to obtain new fused features with a size of 16×16×2048; the new fused features with a size of 16×16×2048 and the fused features with a size of 32×32×1024 are input into the FA module for optical flow correction to obtain optical flow aligned fused features with a size of 32×32×1024; The optical flow alignment fusion features with a size of 32×32×1024 and the fusion features with a size of 64×64×512 are input into the FA module for optical flow correction to obtain new optical flow alignment fusion features with a size of 64×64×512; The optical flow alignment fusion features with a size of 64×64×512 and the fusion features with a size of 128×128×256 are input into the FA module for optical flow correction to obtain new optical flow alignment fusion features with a size of 128×128×256; The optical flow alignment fusion features with a size of 128×128×256 are bilinearly interpolated to enlarge their size to 512×512×256; the optical flow alignment fusion features with a size of 512×512×256 are passed through a convolution layer with a convolution kernel size of 1×1 and a stride of 1 to obtain the final predicted image of 512×512×1. The pixel semantic information of the predicted image is divided into two categories: crack pixels and background pixels.

8. The crack detection method based on cross-mode neural network with optical flow alignment according to claim 1 is characterized by: In step S0, a pre-prepared crack image dataset is sent to a cross-modal feature neural network FAC-Net for training. The crack image dataset includes N crack RGB images and N corresponding crack depth images. The dataset is divided into a test set and a training set in a ratio of 2:

8. The role of the training set is to train FAC-Net, and the role of the test set is to observe the accuracy of the neural network model in identifying cracks during the training process. Three indicators are set as the comparison benchmarks for the observed neural network model, namely precision, recall, and F1 score; the indicator calculation formula is: The parameter TP, i.e. the number of occurrences that are considered to be true and are indeed true; Parameter FP, which is the number of occurrences that are considered true but are actually false; Parameter FN, the number of occurrences that are considered false but are actually true; Parameter TN, the number of occurrences that are considered false and are indeed true; Then precision = TP / (TP+FP) Recall = TP / (TP+FN) F1 score = 2 × precision × recall / (precision + recall).

Images