Visible light and infrared light image fusion method and device for road crack detection

Through multi-scale feature extraction and multi-channel attention generation adversarial network, the problem of unclear edges and poor details in road crack detection in the prior art is solved, and clearer and more detailed road fusion images are generated, improving detection accuracy .

CN119067867AInactive Publication Date: 2024-12-03GANSU XINWANGTONG TECH INFORMATION CO LTD +1

Patent Information

Application Number
CN202411154736.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-22
Publication Date
2024-12-03
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing methods of visible and infrared image fusion lead to unclear edges and inexplicable details in road crack detection, resulting in inaccurate detection of detection results.

Method used

Using a multi-channel attention generation adversarial network based on multi-scale feature extraction, a multi-scale feature extraction sub-network, a multi-attention feature fusion layer and a dual discriminator to generate the network layer, to construct infrared intensity paths and visible light gradient paths, and combine channel attention. Force mechanisms extract and fuse image features to generate road fusion images with clear edges and rich details.

Benefits of technology

The problem of unclear cracks caused by single-scale features is overcome, the characteristics of infrared and visible light images are retained, and different lighting conditions are adapted to the accuracy of road crack detection and the retention of detailed information are improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119067867A_ABST
    Figure CN119067867A_ABST
Patent Text Reader

Abstract

The invention discloses a visible light and infrared light image fusion method and device for road crack detection, and relates to the technical field of image fusion. The method comprises the following steps: inputting a visible light image of a target road and an infrared light image of the target road into an image fusion network to obtain a fusion image of the target road, wherein the image fusion network is obtained by training a multi-path attention generative adversarial network based on multi-scale feature extraction; the network comprises a multi-scale feature extraction sub-network, a multi-attention feature fusion layer, a double-discriminator adversarial generation network layer and a splicing layer. The multi-scale feature extraction sub-network comprises a Nest connection-based multi-scale feature extraction layer and a pyramid feature fusion module which are connected in sequence; the multi-attention feature fusion layer comprises a first cascade operation, an intensity path, a multi-scale fusion path and a gradient path, and the double-discriminator adversarial generative network layer comprises two discriminators and a decoder. According to the invention, a road fusion image with clear edges and rich details can be obtained.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image fusion, and in particular to a visible light and infrared light image fusion method and device for road crack detection. Background Art

[0002] Traditional pavement detection methods are often based on single-modal input data. However, single-modal data often has defects under certain conditions. Changes in image acquisition conditions, such as camera calibration, lighting conditions, and image quality, often make it impossible to obtain clear crack information when dealing with complex environmental conditions. Since visible light and infrared images have almost all inherent features of objects, they can be fused to produce highly robust and information-rich images. However, current fusion rules often ignore the differences in image features, resulting in artifacts in the results and loss of details in the source images.

[0003] In recent years, researchers tend to use deep learning algorithms to achieve the fusion of visible light and infrared images. An existing multi-source image fusion algorithm extracts visible light and infrared images of the fusion target area in the non-subsampled contourlet transform (NSCT) domain, which improves the quality of the fused image. However, when the network depth reaches a certain level, the effect will decrease. A method for image fusion that fuses static multi-exposures (Deep unsupervised approach for Fusion, DeepFuse) first converts the input image to different color spaces, and extracts low-level features (such as texture and edges) and high-level features (such as semantics and structure) in different color spaces through two parallel neural network branches. Then the corresponding weights are fused, and the two feature images are weighted averaged to obtain the output image, but the final fused image also has the problem of low contrast with the environment. A generative adversarial network for infrared and visible image fusion (FusionGAN) defines the fusion algorithm as an adversarial game that maintains infrared thermal radiation information and visible light appearance texture information. However, since the discriminator only focuses on the visible light image during the adversarial generation process, it is difficult to obtain local information of the fused image. A new generator fusion architecture, called Generative Adversarial Network with Multi-classification Constraints (GANMcC), is proposed. The global generator of this framework generates a rough fused image by encoding the global information of the input image, and the local generator generates a detail-corrected image by encoding the local information of the input image. However, the fusion result is closer to the pixel distribution of the infrared image, the image is too smooth, and some discontinuous edges will be generated.

[0004] In summary, the road fusion image obtained by the existing visible light and infrared image fusion method has unclear edges and not rich details, which leads to inaccurate results of the final road crack detection. Summary of the invention

[0005] The purpose of the present invention is to provide a visible light and infrared light image fusion method and device for road crack detection, which can obtain a road fusion image with clear edges and rich details.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] A visible light and infrared light image fusion method for road crack detection, comprising:

[0008] Acquire a visible light image of a target road and an infrared light image of a target road;

[0009] The visible light image of the target road and the infrared light image of the target road are input into an image fusion network to obtain a fused image of the target road, and the fused image of the target road is used for road crack detection. The image fusion network is obtained by training a multi-path attention generation adversarial network based on multi-scale feature extraction; the multi-path attention generation adversarial network based on multi-scale feature extraction includes a multi-scale feature extraction subnetwork, a multi-attention feature fusion layer, a dual discriminator adversarial generation network layer and a splicing layer; the multi-scale feature extraction subnetwork includes a multi-scale feature extraction layer and a pyramid feature fusion module based on Nest connection connected in sequence; the multi-scale feature extraction layer based on Nest connection includes multiple convolutional layers; the multi-attention feature fusion layer includes a first cascade operation, an intensity path, a multi-scale fusion path and a gradient path, the intensity path includes a second cascade operation, a first convolution block, a second convolution block, an intensity attention mechanism and a first cross multiplication operation connected in sequence, and the output end of the second convolution block is also connected to the input end of the first cross multiplication operation; the multi-scale fusion path includes a third convolution block, a channel attention mechanism and a second cross multiplication operation connected in sequence, and the output end of the third convolution block is also connected to the second cross multiplication operation The input end of the operation is connected; the gradient path includes a third cascade operation, a fourth convolution block, a fifth convolution block, a texture attention mechanism and a third cross multiplication operation connected in sequence, and the output end of the fifth convolution block is also connected to the input end of the third cross multiplication operation; the input end of the second cascade operation, the input end of the third convolution block and the input end of the third cascade operation are all connected to the output end of the pyramid feature fusion module; the output end of the first cross multiplication operation, the output end of the second cross multiplication operation and the output end of the third cross multiplication operation are all connected to the input end of the first cascade operation, and the dual discriminator is anti-genetic The network layer includes an infrared light identifier, a visible light identifier and a decoder, the output end of the first cascade operation is connected to the input end of the decoder, the first input end of the infrared light identifier is used to input the infrared light image of the target road, the first input end of the visible light identifier is used to input the visible light image of the target road, the second input end of the infrared light identifier and the second input end of the visible light identifier are both connected to the output end of the decoder, the output end of the infrared light identifier and the output end of the visible light identifier are both connected to the splicing layer, and the splicing layer is used to output a fused image.

[0010] A computer device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-mentioned visible light and infrared light image fusion method for road crack detection.

[0011] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0012] The multi-path attention generation adversarial network based on multi-scale feature extraction provided by the present invention comprises a multi-scale feature extraction subnetwork, a multi-attention feature fusion layer, a dual discriminator adversarial generation network layer and a splicing layer; the multi-scale feature extraction subnetwork comprises a multi-scale feature extraction layer and a pyramid feature fusion module based on Nest connection connected in sequence; the multi-attention feature fusion layer comprises a first cascade operation, an intensity path, a multi-scale fusion path and a gradient path, the intensity path comprises a second cascade operation, a first convolution block, a second convolution block, an intensity attention mechanism and a first cross multiplication operation connected in sequence, and the output end of the second convolution block is also connected to the input end of the first cross multiplication operation; the multi-scale fusion path comprises a third convolution block, a channel attention mechanism and a second cross multiplication operation connected in sequence, And the output end of the third convolution block is also connected to the input end of the second cross multiplication operation; the gradient path includes a third cascade operation, a fourth convolution block, a fifth convolution block, a texture attention mechanism and a third cross multiplication operation connected in sequence, and the output end of the fifth convolution block is also connected to the input end of the third cross multiplication operation; the dual discriminator adversarial generation network layer includes an infrared light discriminator, a visible light discriminator and a decoder. The multi-path attention generation adversarial network structure based on multi-scale feature extraction provided by the present invention mainly divides the fusion path into an infrared intensity path and a visible light gradient path. Through the multi-scale feature extraction subnetwork, the features output by each layer are used to construct a connection as the input of the above two paths, so that the features extracted by the network can be used to the maximum extent, overcoming the problem of unclear cracks of single-scale features. Not only that, the present invention also constructs a channel attention mechanism on the multi-scale fusion path, and performs three-way feature fusion with different fusion feature distributions. Under this mechanism, the features of the infrared or visible light fusion image will be retained, and more important information of the current environment will be paid attention to, so as to better adapt to different lighting conditions during the processing process, and a road fusion image with clear edges and rich details can be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0014] Figure 1 A diagram of a multi-way attention generative adversarial network structure based on multi-scale feature extraction provided by an embodiment of the present invention;

[0015] Figure 2A diagram of the multi-scale feature extraction subnetwork structure provided by an embodiment of the present invention;

[0016] Figure 3 A schematic diagram of a multi-attention fusion process of a multi-attention feature fusion layer provided in an embodiment of the present invention;

[0017] Figure 4 Schematic diagram of the attention mechanism structure in the multi-attention feature fusion layer;

[0018] Figure 5 Schematic diagram of the multi-attention fusion process;

[0019] Figure 6 This is the comparison result diagram of the fusion results of the RoadScene group;

[0020] Figure 7 This is a comparison chart of the fusion results of some network models in the CracksFusion dataset;

[0021] Figure 8 This is a comparison chart of the fusion results of another part of the network models in the CracksFusion dataset;

[0022] Fig. 9 This is a graph showing the results of the six objective indicators of the comparative test;

[0023] Fig.10 This is the result of the ablation experiment;

[0024] Fig.11 This is a diagram of the internal structure of a computer device. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0027] The road crack image appears as a linear or curved fracture, which is usually caused by the shrinkage, stretching, deformation and other reasons of the road surface. The road surface texture in the area will change accordingly, such as color, glossiness, texture, etc. Visible light images contain rich texture information, while infrared images have high contrast. Fusion images can avoid the shortcomings of visible light images being sensitive to lighting conditions and infrared images being noisy and having low resolution. Therefore, this paper proposes a new M2GAN image fusion framework, such as Figure 1 As shown. The main part of the framework includes a visible light and infrared multi-scale feature extraction sub-network, a multi-attention feature fusion layer, and a dual discriminator adversarial generation network layer. These modules work together to form the overall structure of the entire network, realizing the extraction and fusion functions of crack features. Based on this, an embodiment of the present invention provides a visible light and infrared light image fusion method for road crack detection, including:

[0028] A visible light image of a target road and an infrared light image of a target road are acquired.

[0029] The visible light image of the target road and the infrared light image of the target road are input into an image fusion network to obtain a fused image of the target road, and the fused image of the target road is used for road crack detection. The image fusion network is obtained by training a multi-path attention generative adversarial network based on multi-scale feature extraction; the multi-path attention generative adversarial network based on multi-scale feature extraction includes a multi-scale feature extraction subnetwork, a multi-attention feature fusion layer, a dual discriminator adversarial generative network layer and a splicing layer; the multi-scale feature extraction subnetwork includes a multi-scale feature extraction layer and a pyramid feature fusion module based on Nest connection connected in sequence; the multi-path attention generative adversarial network based on Nest connection is trained by the image fusion network. The multi-scale feature extraction layer connected by t includes multiple convolution layers; the multi-attention feature fusion layer includes a first cascade operation, an intensity path, a multi-scale fusion path and a gradient path, the intensity path includes a second cascade operation, a first convolution block, a second convolution block, an intensity attention mechanism and a first cross multiplication operation connected in sequence, and the output end of the second convolution block is also connected to the input end of the first cross multiplication operation; the multi-scale fusion path includes a third convolution block, a channel attention mechanism and a second cross multiplication operation connected in sequence, and the output end of the third convolution block is also connected to the input end of the second cross multiplication operation; the gradient path includes a third cascade operation, a fourth convolution block, a fifth convolution block connected in sequence The output end of the fifth convolution block is also connected to the input end of the third cross product operation; the input end of the second cascade operation, the input end of the third convolution block and the input end of the third cascade operation are all connected to the output end of the pyramid feature fusion module; the input end of the second cascade operation is also used to input the infrared image of the target road, and the input end of the third cascade operation is also used to input the visible light image of the target road; the output end of the first cross product operation, the output end of the second cross product operation and the output end of the third cross product operation are all connected to the input end of the first cascade operation, and the dual discriminator is used to counteract the generation of The network layer includes an infrared light discriminator, a visible light discriminator and a decoder. The output end of the first cascade operation is connected to the input end of the decoder. The first input end of the infrared light discriminator is used to input the infrared light image of the target road. The first input end of the visible light discriminator is used to input the visible light image of the target road. The second input end of the infrared light discriminator and the second input end of the visible light discriminator are both connected to the output end of the decoder. The output end of the infrared light discriminator and the output end of the visible light discriminator are both connected to the splicing layer. The splicing layer is used to output a fused image. The final fused image is generated after confronting the dual discriminators.

[0030] In practical applications, the multi-scale feature extraction layer based on Nest connection includes 6 convolution layers; the input end of the first convolution layer, the input end of the fourth convolution layer, and the input end of the sixth convolution layer are all used to input the visible light image of the target road and the infrared light image of the target road, the output end of the first convolution layer, the output end of the third convolution layer, the output end of the fourth convolution layer, the output end of the fifth convolution layer, and the output end of the sixth convolution layer are all connected to the input end of the pyramid feature fusion module, the output end of the first convolution layer and the output end of the fourth convolution layer are also connected to the input end of the second convolution layer, the output end of the second convolution layer and the output end of the fifth convolution layer are also connected to the input end of the third convolution layer, the output end of the fourth convolution layer and the output end of the sixth convolution layer are also connected to the input end of the fifth convolution layer, and the convolution kernels of the first convolution layer, the fourth convolution layer, and the sixth convolution layer are all different.

[0031] In practical applications, the pyramid feature fusion module includes a first convolution layer, a second convolution layer, a third convolution layer, a fourth convolution layer and a fifth convolution layer.

[0032] The output end of the first convolutional layer is connected to the input end of the second convolutional layer and the input end of the fourth convolutional layer respectively; the output end of the second convolutional layer is connected to the input end of the third convolutional layer; the output end of the third convolutional layer is connected to the input end of the fifth convolutional layer; the output end of the fifth convolutional layer is connected to the input end of the fourth convolutional layer, and the output end of the fourth convolutional layer is respectively connected to the input end of the second cascade operation, the input end of the third convolutional layer neural network and the input end of the third cascade operation.

[0033] An output end of the first convolution layer and an output end of the third convolution layer in the multi-scale feature extraction layer are both connected to an input end of the first convolution layer in the pyramid feature fusion module.

[0034] The output end of the fourth convolution layer and the output end of the fifth convolution layer in the multi-scale feature extraction layer are both connected to the input end of the second convolution layer in the pyramid feature fusion module.

[0035] An output end of the sixth convolutional layer in the multi-scale feature extraction layer is connected to an input end of the third convolutional layer in the pyramid feature fusion module.

[0036] In order to extract more accurate image features about the target crack, it is necessary to make full use of the multi-scale regional information containing the current pixel. Generally speaking, single-scale convolution is usually used to capture local features of fixed size in image information extraction, and cannot obtain contextual information of a larger range. Therefore, the multi-scale feature extraction (MFE) sub-network designed in the present invention can make full use of the deep and shallow information provided by the backbone network for efficient fusion. The structure is as follows: Figure 2 As shown. This module adopts a multi-scale convolution kernel design to extract feature information of different scales. By integrating the information extracted at different scales using a pyramid feature fusion layer, global and local features can be captured, allowing the network to extract richer and more complete feature information from the fused image. The nodes in the figure are both the names of the fused features and represent the fusion operation. In order to obtain the complete feature set of the input visible light and infrared images at different scales, three convolution layer kernels with scales of 5×5, 3×3 and 1×1 are first used to extract the feature information of the input visible light and infrared images at different dimensions, thereby obtaining the features of the input MFE {P1, P2, P3}. Starting from the deepest feature of the input, a top-down fusion is performed to obtain In order to make full use of multi-scale features, the low-resolution, high-semantic information crack top-level features are fused with the high-resolution, low-semantic information crack bottom-level features through top-down side connections. The original input features are fused layer by layer to obtain This makes the crack features at all scales have rich feature information; finally, a high-resolution feature map P is obtained by adding layer by layer through a bottom-up path. out , thereby capturing the edge, texture, color and other detailed information of small cracks at different scales and improving the visual expression of the network.

[0037] In practical applications, such as Figure 4 As shown in part (b), the texture attention mechanism includes: a second maximum pooling layer, a second average pooling layer, a seventh convolutional layer, an addition operation, a fourth cascade operation, a second sigmoid function, a Soble operation, a Laplace operation, a first SimAM attention operation, and a second SimAM attention operation.

[0038] The input end of the Soble operation, the input end of the Laplacian operation, the input end of the second maximum pooling layer and the input end of the second average pooling layer are all connected to the output end of the fifth convolutional block; the output end of the Laplacian operation is connected to the input end of the first SimAM attention operation; the output end of the Soble operation is connected to the input end of the second SimAM attention operation; the output end of the second maximum pooling layer and the output end of the second average pooling layer are both connected to the input end of the seventh convolutional layer, the output end of the second SimAM attention operation and the output end of the seventh convolutional layer are both connected to the input end of the addition operation, the output end of the first SimAM attention operation and the output end of the addition operation are both connected to the input end of the fourth cascade operation, the output end of the fourth cascade operation is connected to the input end of the second sigmoid function, and the output end of the second sigmoid function is connected to the input end of the third cross product operation. In the texture attention mechanism, two gradient residuals with SimAM attention are connected with Sobel and Laplacian operators to compensate for the fine-grained representation of features. The Sobel operator is used to calculate the grayscale gradient of each pixel in the image to maintain the strong texture features of the image. The Laplacian operator is used to detect the edges and details of the image, which provides a basis for further extracting weak texture features. Stronger texture details are retained by splicing channels with strong texture features. Finally, a more complete texture feature representation is obtained by combining the two gradient residuals.

[0039] The sigmoid function is used to compress the range of the eigenvalues ​​and obtain the final weighted value graph. It can be expressed as the following formula (1):

[0040]

[0041] Here, IF is the input feature map of the attention module, here is the map in the input texture attention mechanism, TA represents the texture attention operation, MP represents the maximum pooling operation, and AP represents the average pooling operation. AM represents the SimAM attention operation, and represents Sobel and Laplace operators, Con 1×1 is a 1×1 convolutional layer, and S represents the sigmoid function. The texture attention module allows the model to focus on the texture area of ​​the input feature map. It represents the add operation, the feature maps are added, the number of channels remains unchanged, and F represents the input feature map Feature.

[0042] In practical applications, such as Figure 4As shown in part (c), the intensity attention mechanism includes: a high-pass filter and an intensity attention module; the intensity attention module includes a first maximum pooling layer, a first average pooling layer, a sixth convolutional layer and a first sigmoid function; the input ends of the first maximum pooling layer and the first average pooling layer are respectively connected to the output end of the high-pass filter, the output ends of the first maximum pooling layer and the first average pooling layer are respectively connected to the input end of the sixth convolutional layer, the output end of the sixth convolutional layer is connected to the input end of the first sigmoid function, the input end of the high-pass filter is connected to the output end of the second convolutional block; the output end of the first sigmoid function is connected to the input end of the first cross product operation. The construction of the intensity attention module is intended to make the model pay attention to areas with high pixel intensity. First, the maximum pooling and average pooling operations are performed on the input channel dimension to obtain the corresponding two weight maps. Then the weight maps are connected and input into the 1×1 convolutional layer to reduce the dimension of the weight map. First, a high-pass filter layer is designed to store the intensity information of the input feature map to obtain an intensity feature map. Then, through the structure of an intensity attention module, including a maximum pooling layer, an average pooling layer, a 1×1 convolution layer and a sigmoid function. The operation formula is shown in formula (2):

[0043] IA(IF)=S(Con 1×1 (concat(AP(HP(IF)),MP(HP(IF))))), (2)

[0044] Among them, IA intensity represents the attention operation, HP represents the high-pass filter operation. For photos with weak light intensity, the image may appear blurry or unclear due to insufficient light. The high-pass filter makes the image clearer by enhancing the details of the high-frequency area. IF is the input feature map of the attention module, which is the map in the input intensity attention mechanism. Concat represents the cascade operation.

[0045] In practical applications, such as Figure 4 As shown in part (a), the channel attention mechanism replaces the SharedMLP in the channel attention module with the channel attention mechanism of the fully connected layer. The channel attention mechanism performs global maximum pooling and global average pooling operations, and then uses two fully connected layers, maximum pooling operations and sigmoid functions to obtain the final weight map. As shown in the following formula (3):

[0046] CA(IF)=S(concat(FC(AP(IF)),FC(MP(IF)))), (3)

[0047] CA represents the channel attention operation, IF is the input feature map of the attention module, here is the map in the input channel attention mechanism, and FC represents the fully connected layer.

[0048] Specifically, even if different types of source images contain different main information, there is still some secondary information that needs to be retained in the final fused image to facilitate the identification and judgment of the background and the target. Most existing methods find it difficult to achieve a good balance between visible light and infrared inherent information. For example, the fusion results of some methods tend to be visible light images. Although they contain rich texture details, they do not have significant contrast and cannot clearly distinguish the target from the background when the light intensity is low. Conversely, although some methods may make the results closer to infrared images, the texture may not be rich enough. This imbalance may bring challenges to subsequent recognition tasks, especially when less visible light information is captured during the day. The multi-attention feature fusion layer proposed in this embodiment, such as Figure 3 As shown, a fused image is generated according to the infrared target contour and visible light texture information, and a dual discriminator is constructed in the dual discriminator adversarial generation network layer to solve the contrast and detail problems caused by light intensity changes, thereby improving the performance of the fused image. In the multi-attention feature fusion layer, a multi-attention feature fusion module is formed by a multi-scale fusion path, and a multi-attention feature fusion module (MEFM) is mainly constructed to realize feature extraction and fusion of the source image. The multi-attention feature fusion layer includes three paths: a gradient path, an intensity path, and a multi-scale fusion path. The intensity and gradient paths focus on the salient target area of ​​the infrared image and the texture information of the visible light image respectively. In order to enhance the features extracted by the two paths, the multi-scale feature extraction image and the visible light image are spliced ​​as the input of the gradient path, and are spliced ​​with the infrared image as the input of the intensity path. With the goal of improving the extraction efficiency of the two features, this embodiment simultaneously constructs two attention mechanisms (intensity attention mechanism and texture attention mechanism). The multi-scale fusion path further emphasizes and fuses the regional features of the image by adding a channel attention mechanism, and pre-fuses the shallow features of the two paths.

[0049] In practical applications, the decoder includes a convolution block module, a ninth convolution layer and a tanh activation function connected in sequence; the convolution block module includes a plurality of sixth convolution blocks connected in sequence.

[0050] In practical applications, the first convolution block, the second convolution block, the third convolution block, the fourth convolution block, the fifth convolution block and the sixth convolution block all include an eighth convolution layer, a BN layer and a LeakyReLU activation function connected in sequence.

[0051] In practical applications, the decoder is mainly responsible for feature extraction of the fused image information output by the encoder (multi-scale feature extraction subnetwork and multi-attention feature fusion layer), and improves the decoder's ability to learn and extract information through training. The decoder consists of four layers of convolutional blocks. From the first layer to the third layer of the convolutional network, a 3×3 filter is used, the step size is fixed to 2, and a batch normalization layer is used from the first layer to the third layer. In addition, in terms of activation functions, the LeakyReLU activation function is used in the first three layers. The last layer is Tanh, the main purpose of which is to retain the detailed information of the image and ensure that the fused image can transition more smoothly without introducing too much noise. The specific parameters of the convolutional layer of the decoder structure are shown in Table 1.

[0052] Table 1 Decoder network structure layer

[0053] layer nuclear Step Length Input Channels Output Channel Activation Function Conv1 3×3 2 1 32 LeakyReLU Conv2 3×3 2 32 64 LeakyReLU Conv3 3×3 2 64 128 LeakyReLU Conv4 3×3 2 128 256 Tanh

[0054] The present invention sets up two discriminators: one is used to distinguish the fused image from the infrared image (Discriminator-IR), and the other is used to distinguish the fused image from the visible light image (Discriminator-VIS). These two discriminators enable the fused image to have the high-resolution details of the visible light image and the highlighted areas of the infrared image. Since the visible light and infrared images are captured by different sensors, the important features of concern are also different. The different network structures enable Dis-IR to pay more attention to the intensity distribution of the input image, Dis-VIS focuses on the texture information of the input image, and the decoder generates the fusion result by using the texture information of the visible light image and the pixel intensity distribution of the infrared image through continuous adversarial learning.

[0055] When the intensity attention module is in effect, the network focuses on the high pixel intensity area by compressing the input features in the channel direction. On the other hand, the texture attention mechanism compresses the input feature gradient map in the channel direction and assigns weights to the input features to make the network focus on the texture area of ​​the input feature map. Figure 5 As shown, Figure 5 Part (a) shows the working process of the visible light discriminator. Figure 5 Part (b) shows the working process of the infrared light discriminator. The discriminator consists of three convolutional layers and a tanh activation function. Batch normalization is introduced to solve the problem of network gradient disappearance. In order to improve the nonlinear expression ability of the neural network and avoid numerical overflow, the tanh function can be used to optimize the training and prediction effects of the model.

[0056] In the training phase, the source images are stacked and then input to the decoder to obtain a preliminary fusion result. The fused output is passed to the two discriminators respectively, and the quality of the fused image is evaluated and fed back through the loss function. By alternately performing the above two steps, adversarial training of the decoder and the discriminator can be performed to obtain a generated image with an ideal effect, achieving the purpose of image fusion.

[0057] Not only the main infrared intensity loss is constructed between the fused image and the infrared image, but also the auxiliary gradient loss is introduced. Similarly, a loss function is designed to consider the gradient difference between the fused image and the visible light image, and it is still feasible when the visible light image is overexposed or the gradient changes greatly. A specific content loss is designed to constrain the decoder, and the main and auxiliary ideas are introduced into the extraction of gradient and intensity information, so as to have rich texture details and significant contrast. First, convolution kernels of different sizes are used to extract the feature information of visible light and infrared images to obtain more dimensional detail features. After multi-scale feature extraction, the extracted multi-scale fusion features are sent to the multi-scale fusion path for channel attention mechanism, which is realized by a single convolution operation.

[0058] Among them, L con Represents a convolutional block, and I0 is the input of a convolutional block. 3×3 , LRelu and BN represent 3×3 convolutional layer, LeakyRelu activation function and BatchNormalization respectively. f Represents the image of multi-scale feature fusion, I vis and I in Represent visible light image and infrared image respectively. concat is a cascade operation. VTAP2 is the output of the second layer of the visible light image (VisibleTextureAttention-path), IIAP2 is the output of the second layer of the infrared image (Infrared IntensityAttention-path), and TFF1 (Three attention Features Fusion) is the result of the three-way attention feature fusion. As shown in equations (4)-(8).

[0059] L con =LRelu(BN(Con 3×3 ((I0))) (4)

[0060] VTAP2=L con (L con (concat(I vis , I f ))) (5)

[0061] IIAP2=Lcon (L con (concat(I in , I f ))) (6)

[0062] CAP2=L con (L con (I f ))) (7)

[0063] TFF1=L con (concat(VTAP2,IIAP2,CAP2)) (8)

[0064] In order to make the fused image contain the most important information of the source image, a joint loss function is designed to train the multi-attention feature fusion layer. This function consists of three parts: intensity loss, gradient loss, structural similarity loss and pixel loss, which are defined as follows (9):

[0065] L total =αL int +βL grad +δL ssim (9)

[0066] Among them, L total Represents the total loss of the decoder, L int represents the salient target strength loss, L grad represents the gradient loss, L ssim Represents the structural similarity loss. α, β and δ are weight coefficients used to properly control the weight ratio of each factor.

[0067] 1) Loss of salient target strength

[0068] In the fused image, the traditional intensity loss function is directly calculated by interpolation. Although this calculation method can obtain some clear information of the infrared image, it also directly retains a lot of noise. In order to solve this problem, this embodiment proposes a salient target intensity loss function. The infrared image is processed using the OtsuThresholding method, and the binary image is multiplied with the fused image and the infrared image respectively to obtain two salient images to constrain the training of the decoder. The purpose of the intensity loss function is to enable the generated fused image to retain more information about the salient areas of the infrared image while ignoring the information in other infrared images. H and W represent the length and width of the image, Fused_I represents the intensity of the fused image, and I in Represents the input infrared image intensity, I B Represents the input background image intensity.

[0069]

[0070] 2) Gradient loss

[0071] An excellent fusion result needs to preserve both the pixel intensity of the salient target in the infrared image and the texture information of the visible light image. To achieve this goal, the gradient loss of the fused image and the visible light image can be used so that the fused image contains more texture information of the visible light image. represents the gradient of the fused image, Represents the gradient of the visible light image, and the gradient loss is defined as follows:

[0072]

[0073] 3) SSIM loss

[0074] SSIM is an indicator used to measure the structural similarity of two images. In order to ensure that the fused image is more similar to the two source images in terms of structural properties, make the fused image more natural and smooth, and avoid artifacts, the specific definition is as follows.

[0075] L ssim =1-SSIM(Fused_I,I vis )+1-SSIM((Fused_I,I in ) (12)

[0076]

[0077] Among them, I vis Represents the fused image and visible light image input respectively, μ Y , represent the average values ​​of x and y, σ Y , representing the standard deviations σ of x and y respectively XY Represents the covariance of x and y, C1, C2, C3 represent constants to avoid the denominator being 0, SSIM represents the structural similarity between the input and output images, and is mainly used to construct the loss optimization of SSIM.

[0078] 4) Discriminator loss function

[0079] The dual discriminator structure is used in the fusion algorithm, so the network discriminator loss is defined as follows

[0080]

[0081] In this formula: They represent the discriminator loss with infrared and fused images as input and the discriminator loss with visible light and fused images as input respectively; P ir / vis It shows the distribution of visible light and infrared image data. represents the discriminator loss with infrared image or visible light image as input, represents the discriminator loss with the fused image as input.

[0082] The principle of the multi-way attention generation adversarial network based on multi-scale feature extraction provided by the present invention is that the shallow features extracted by the convolutional neural network have high resolution, which contain more details and position information of small-sized cracks. However, as the depth of the network increases, the representation of small target information is gradually weakened. By cleverly distinguishing the differences between multiple attentions, significant contrast and rich texture details are obtained, making the fusion result more balanced. To this end, the present invention designs a multi-scale feature extraction subnetwork to increase the semantic information of shallow high-resolution features through lateral connections during the fusion process. When the model works, first extract the multi-way image feature information in the network model, and input this information through the multi-attention feature fusion layer. Then, at the end of the multi-scale feature extraction subnetwork, it is fused with the visible light and infrared images respectively, focusing the information from the two aspects of texture and intensity, and fusion of the extracted three-way features. Finally, the confrontation between the decoder and the discriminator is used to improve the network performance. The multi-way attention generation adversarial network based on multi-scale feature extraction provided by the present invention is a multi-way attention generation adversarial network with multi-scale connections, which can better fuse the significant information of visible light and infrared images. The network structure mainly divides the decoder fusion path into an infrared intensity path and a visible light gradient path. The visible light and infrared multi-scale feature extraction sub-network uses the features output by each layer to build a connection as the input of the above two paths, so that the features extracted by the network can be used to the maximum extent, overcoming the problem of unclear cracks in single-scale features. In addition, the present invention also constructs a channel attention mechanism on the multi-scale fusion path, and performs three-way feature fusion with different fusion feature distributions. Under this mechanism, the features of the infrared or visible light fusion image will be retained, and more important information of the current environment will be paid attention to, so as to better adapt to different lighting conditions during the processing process.

[0083] The present invention also provides a specific embodiment to explain the above method in detail, and the specific steps are as follows:

[0084] Step 2.1: Experimental configuration and dataset construction

[0085] The experiment uses the deep learning platform Pytorch1.7, and the hardware is Intel(R) Xeon(R) CPU E5-2678v3. This embodiment uses the public dataset RoadScene and the self-made dataset CracksFusion for experiments. The RoadScene dataset is a dataset established in the literature for the field of autonomous driving and intelligent transportation systems. It contains 221 pairs of precisely aligned infrared images and visible light images, which are used to improve the perception and accuracy of the autonomous driving system. The CracksFusion dataset is a self-made dataset that was taken by drones around the North Ring Road in Lanzhou City, Gansu Province, and includes 300 pairs of grayscale processed visible light and infrared images.

[0086] In the experiment, typical fusion methods were selected for evaluation on RoadScene and CracksFusion datasets, including Discrete Wavelet Transform (DWT), Fast Unified Image Fusion Network (PMGI) based on Gradient and Intensity Ratio Maintenance, Squeeze Decomposition Network (SDNet), Generative Image Fusion Adversarial Network (FusionGAN), Densely Connected Convolutional Network (DenseFuse), Progressive Image Fusion Based on Illumination Perception (PIAFusion), Pixel-Level Two-Dimensional Image Fusion (U2Fusion) and Generative Adversarial Network with Multi-Classification Constraint (GANMcC), a total of 8 fusion methods, which were evaluated and analyzed by subjective and objective evaluation indicators.

[0087] Step 2.2: Subjective evaluation of fusion images

[0088] In this example, the sample "FLIR_00455" in the RoadScene dataset was selected for experimental analysis. Figure 6 As shown, Figure 6 Part (a) shows the visible light image. Figure 6 Part (b) shows the infrared image. Figure 6 Part (c) shows the processing result using DWT. Figure 6 Part (d) shows the processing results using PMGI. Figure 6 Part (e) shows the processing result using SDNet. Figure 6 Part (f) shows the processing result using Densefuse. Figure 6 Part (g) shows the processing result using FusionGAN. Figure 6 Part (h) shows the processing result using GANMcC. Figure 6 Part (i) shows the processing result using PIAfusion. Figure 6 Part (j) shows the processing result using U2Fusion. Figure 6Part (k) shows the processing result of the visible light and infrared light image fusion method for road crack detection provided by the present invention. The DWT algorithm and the PMGI algorithm will have background blur and noise artifact problems during the image fusion process. In comparison, the SDNet, U2Fusion and DenseFuse algorithms can improve the clarity of the fused image, but are relatively lacking in the contrast between the target crack and the overall scene, and also lack the edge and detail information of the visible light and infrared images.

[0089] PIAFusion has high accuracy in identifying cracks after fusion, but it has high requirements for the selection of suitable scenes and image quality. If the image quality is poor or there are serious noise, blur and other problems, it may affect the detection effect of the algorithm. The fusion results of FusionGAN and GANMcC are closer to the pixel distribution of infrared images, but lack texture detail information. In the algorithm experiment proposed in this embodiment, the texture structure characteristics of the cracks and the overall environment show more obvious differences, and the edge detail information is also effectively preserved. In addition, the contrast between the cracks and the cars in the environment is large, making the cracks easier to detect.

[0090] In order to further verify the reliability of the proposed method for small crack identification, this example selected five groups of images with different illumination intensities from the CracksFusion dataset. Figure 7 and Figure 8 As shown, Figure 7 Part (a) shows the visible light image. Figure 7 Part (b) shows the infrared image. Figure 7 Part (c) shows the processing result using DWT. Figure 7 Part (d) shows the processing results using PMGI. Figure 7 Part (e) shows the processing result using SDNet. Figure 7 Part (f) shows the processing result using Densefuse. Figure 8 Part (a) shows the processing result using FusionGAN. Figure 8 Part (b) shows the processing result using GANMcC. Figure 8 Part (c) shows the processing result using Densefuse. Figure 8 Part (d) shows the processing result using U2Fusion. Figure 8Part (e) shows the processing results of the visible light and infrared light image fusion method for road crack detection provided by the present invention. For the convenience of intuitive comparison, the results of the fusion of the same image using different methods are partially enlarged at the same position, as shown in the red box. The crack widths of the first group of images vary greatly. The fusion methods of the SDNet model, the Densefuse model, and the FusionGAN model cannot detect small cracks.

[0091] PMGI considers spatial gradient information in network design, thereby retaining the detail information of the source image to a certain extent, but it also faces the problem of texture detail loss. The model provided in this embodiment uses different attention mechanisms to effectively improve the global information and detail information, and strengthens the weight of the crack area. Therefore, for images with large differences in crack width, the model can still achieve good results.

[0092] The traditional method DWT retains more background detail information and improves the phenomenon of excessive noise, but there will still be artifact interference. For example, in the second group of images of the DWT algorithm fusion results, the outline of the small cracks that are close to each other in the middle area is not clear, the contrast is not obvious, and they will be mistakenly identified as a crack, and will be affected by the rutting marks. The model provided in this embodiment utilizes the multi-scale feature extraction subnetwork of the image and the feature pyramid at the same time, so the recognition ability of the cracks is stronger and the overall structure of the cracks can be fully displayed. The cracks in the upper half of the third group of images have low contrast with the background and are relatively simple vertical cracks. The recognition effect of each model is good. The cracks in the lower half are unevenly distributed and complex, and background noise such as scratches and black spots appear, resulting in the PMGI and SDNet models to misidentify the crack joints.

[0093] The contour information of the PIAFusion fusion result is clearer, but there are still some details that are not accurately reflected, as shown in the red box in the figure. For the fourth row of pictures, the environment is complex, and a crack appears in the model's detection image that is not recognized. The fusion result of the U2Fusion model has clear cracks, but it will miss small cracks and information at the intersection of multiple cracks, and the detection of crack pixels is not sufficient. For example, the contrast of the fifth column picture is low and the cracks are covered by shadows. GANMcC effectively balances infrared significant information and visible light background information by constructing dual-channel input. However, a white halo is generated at the crack in the upper left corner, which is not conducive to subsequent processing. In contrast, the results of this embodiment not only highlight the thermal target and texture details, but also improve the contrast of the fused image, and have the best subjective fusion effect.

[0094] Step 2.3: Objective evaluation of fused images

[0095] In order to objectively evaluate the performance of the fused image, this embodiment uses the following six evaluation indicators: structural similarity (SSIM), information entropy (EN), visual fidelity (VIF), mutual information (MI), difference correlation coefficient (SCD) and edge preservation (Q ab / f ). Among them, EN and MI are evaluation indicators based on information theory; SSIM and VIF are evaluation indicators based on human visual perception; SCD and Q ab / f It is an evaluation indicator based on the source image and the fused image.

[0096] The fusion performance is quantitatively evaluated through experimental analysis of "FLIR_00455" in the RoadScene dataset. The evaluation results are shown in Table 2, with the optimal and suboptimal values ​​marked in red and black bold, respectively. The algorithm in this embodiment has obvious advantages in most evaluation indicators, especially compared with the DenseFuse algorithm, the SF indicator is improved by about 77.22%, and the edge and texture information appear richer and clearer. This shows that by introducing the corresponding attention mechanism in each channel, the fusion result is more beneficial for crack identification.

[0097] Table 2 Objective evaluation of the “FLIR” group

[0098]

[0099]

[0100] We further selected five groups of visible light and infrared photos from the RoadScene dataset for fusion quantitative data analysis and visualized them in the form of line graphs. Fig. 9 The objective evaluation data of different methods are shown. Fig. 9 Part (a) is SSIM. Fig. 9 Part (b) is EN, Fig. 9 Part (c) is VIF, Fig. 9 Part (d) is SCD, Fig. 9 The middle (e) part is MI, Fig. 9 The part (f) is Q ab / f By comparing various technical indicators, the method provided by the embodiment of the present invention not only improves the quality of the overall fused image, but also retains good visual signal fidelity, which is consistent with the subjective judgment conclusion. The embodiment of the present invention conducts comparative experiments in the CracksFusion dataset to further expand the scope of objective evaluation experiments. Table 3 shows the relevant results.

[0101] Table 3 Objective evaluation of the “CracksFusion” group

[0102]

[0103]

[0104] The algorithm provided by the embodiment of the present invention has achieved the best results in the three indicators of structural similarity, difference correlation coefficient and mutual information of the two data sets. In particular, the two indicators of structural similarity and mutual information have obvious advantages over other algorithms, which is basically consistent with the subjective evaluation. Compared with the Densefuse algorithm, the structural similarity is improved by 39.35%, and the mutual information is improved by 22.14%. PIAFusion is more biased towards visible light images in the fusion results, which is more in line with the visual system of the human eye, resulting in edge preservation that is only slightly lower than PIAFusion. Finally, in terms of information entropy and edge preservation indicators, the algorithms of this embodiment have been improved to varying degrees.

[0105] Step 2.4: Ablation experiment

[0106] The algorithm proposed in the embodiment of the present invention mainly includes a multi-scale feature extraction (MFE) subnetwork and a multi-attention feature fusion layer. In this embodiment, 4 groups of ablation experiments and 2 evaluation indicators are designed to prove the effectiveness of these two modules. When only the MFE module is used, the module output is cascaded with the source image features of the two modalities, and the three-way feature modules are fused after the number of channels is reduced by the convolution block, and finally sent out of the network. When only the MEFM module is used, the MFE module is deleted, and the deep features are directly convolved with 1×1, and the number of channels is reduced before being sent to the MEFM module. A group of images are randomly selected from the RoadScene dataset, and the experimental results are shown in the figure. Fig.10 As shown, Fig.10 Part (a) is a visible light image. Fig.10 Part (b) is the infrared image. Fig.10 Part (c) is the image processed by NO_Both (representing the original network). Fig.10 Part (d) is the image processed by NO_MEFM (representing only multi-attention feature extraction network). Fig.10 Part (e) is the image processed by NO_MFE (representing only multi-scale feature fusion network). Fig.10 Part (f) is an image obtained by using the visible light and infrared light image fusion method for road crack detection provided by the present invention. The NO_Both model loses most of the background detail information of the visible light, while the NO_MEFM retains more source image detail texture features and can capture the image information more comprehensively. However, the contrast is slightly insufficient, which may lead to misidentification of road cracks. The fusion result of NO_MFE improves the above shortcomings, but there are artifacts in the red frame cracks.

[0107] In general, the multi-scale feature extraction subnetwork and multi-attention feature fusion layer proposed in this embodiment successfully retain the details of infrared and visible light images and effectively eliminate artifacts. The objective evaluation results of the ablation experiment are shown in Table 4. Compared with the original fusion network, the multi-scale feature extraction subnetwork improves the structural similarity and edge preservation by 8.68% and 14.80% respectively, indicating that the fusion result is more detailed. With the introduction of the multi-attention feature fusion layer, the edge preservation is significantly improved by 21.43%, indicating that the collaborative training of different modules enables the key features of the attention model to be better strengthened and retained.

[0108] Table 4 Objective evaluation of the ablation experimental group

[0109]

[0110]

[0111] Finally, the algorithm of this embodiment achieves the optimal values ​​in the two indicators of structural similarity and edge preservation by fusing two design modules, and increases by 10.19% and 23.72% respectively, verifying that the deep semantic information fusion module fuses the deep semantic features from the two encoders, which is conducive to improving the feature representation ability of the network. The shallow-middle layer information fusion module that integrates channel attention and spatial attention promotes the effective fusion of shallow-middle layer features based on the modal characteristics of infrared images and visible light images. The proposed module combination is indeed helpful for image fusion.

[0112] Step 2.5: Run efficiency analysis

[0113] The model running efficiency is compared and analyzed, with the average time required to fuse ten groups of images as the standard. Densefuse and FusionGAN use convolution to encode and decode local features, and the computational complexity is low. As the backbone network in image fusion, GANMcC has good visual effects but is relatively time-consuming. The method of this embodiment has the shortest running time compared to other fusion networks. In summary, considering the objective indicators and the running time comparison in Table 5, it can be seen that the algorithm of this embodiment has obtained better evaluation in terms of overall indicators and performance.

[0114] Table 5 Comparison of running time of different methods on the dataset

[0115]

[0116] Step 2.6: Conclusion:

[0117] The present invention provides a multi-path attention generative adversarial network (Multiscalefeature extraction-multiscale attention GAN, M2GAN) based on multi-scale feature extraction. By designing a multi-scale feature extraction subnetwork, the problems of insignificant small crack detection and loss of texture details are solved. At the same time, a multi-attention feature fusion layer is proposed, which can make the extracted features more comprehensive and rich. The corresponding attention modules are introduced into the discriminator to enhance the dependency between different channel features. The proposed method effectively inherits more image features from visible light and infrared images, generates better fused images, and has the best fusion effect. It is effective for crack detection in fused images of different roads, can improve the accuracy of segmentation results, and reduce the recognition error rate. In terms of subjective visual fusion, it can significantly reduce artifacts, make the target edge information clearer, and the crack detection capability is also significantly improved. Experimental results show that the method of this embodiment achieves good visual effects and higher evaluation indicators on the RoadScene and CracksFusion datasets, indicating that this method is superior to the other 6 fusion methods. On the RoadScene public dataset, compared with the experimental results of various mainstream image fusion algorithms, the results of six evaluation indicators were significantly improved, among which the structural similarity and edge preservation indicators were improved by an average of 16.56% and 22.45% respectively. M2GAN has better visual effects and structural similarity, and is superior to the comparison method in objective evaluation.

[0118] In one embodiment, a computer device is provided, the computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the computer program to implement the visible light and infrared light image fusion method for road crack detection described in the above embodiment. The computer device may be a database, and its internal structure diagram may be as follows: Fig.11 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store pending transactions. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data processing method is implemented.

[0119] In order to solve the common problems of small cracks being difficult to identify in the fusion of visible light and infrared images, loss of texture details and introduction of edge artifacts caused by weakening light intensity, etc., the present invention proposes an image fusion method based on a multi-way attention generative adversarial network for multi-scale feature extraction. First, M2GAN proposes a multi-scale feature extraction subnetwork. This module uses registered visible light and infrared images to extract information of different scales in visible light and infrared images, and through side connections, the crack details and semantic information in the fusion process are retained at the same time, making the crack features richer. Then a multi-way attention mechanism is proposed to splice the multi-scale fused image with the infrared source image and the visible light source image respectively, and construct the infrared intensity path and the visible light gradient path to save more target information and background information.

[0120] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0121] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A visible light and infrared light image fusion method for road crack detection, characterized in that: include: Acquire a visible light image of a target road and an infrared light image of a target road; The visible light image of the target road and the infrared light image of the target road are input into an image fusion network to obtain a fused image of the target road, and the fused image of the target road is used for road crack detection. The image fusion network is obtained by training a multi-path attention generation adversarial network based on multi-scale feature extraction; the multi-path attention generation adversarial network based on multi-scale feature extraction includes a multi-scale feature extraction subnetwork, a multi-attention feature fusion layer, a dual discriminator adversarial generation network layer and a splicing layer; the multi-scale feature extraction subnetwork includes a multi-scale feature extraction layer and a pyramid feature fusion module based on Nest connection connected in sequence; the multi-scale feature extraction layer based on Nest connection includes multiple convolutional layers; the multi-attention feature fusion layer includes a first cascade operation, an intensity path, a multi-scale fusion path and a gradient path, the intensity path includes a second cascade operation, a first convolution block, a second convolution block, an intensity attention mechanism and a first cross multiplication operation connected in sequence, and the output end of the second convolution block is also connected to the input end of the first cross multiplication operation; the multi-scale fusion path includes a third convolution block, a channel attention mechanism and a second cross multiplication operation connected in sequence, and the output end of the third convolution block is also connected to the second cross multiplication operation The input end of the operation is connected; the gradient path includes a third cascade operation, a fourth convolution block, a fifth convolution block, a texture attention mechanism and a third cross multiplication operation connected in sequence, and the output end of the fifth convolution block is also connected to the input end of the third cross multiplication operation; the input end of the second cascade operation, the input end of the third convolution block and the input end of the third cascade operation are all connected to the output end of the pyramid feature fusion module; the output end of the first cross multiplication operation, the output end of the second cross multiplication operation and the output end of the third cross multiplication operation are all connected to the input end of the first cascade operation, and the dual discriminator is anti-genetic The network layer includes an infrared light identifier, a visible light identifier and a decoder, the output end of the first cascade operation is connected to the input end of the decoder, the first input end of the infrared light identifier is used to input the infrared light image of the target road, the first input end of the visible light identifier is used to input the visible light image of the target road, the second input end of the infrared light identifier and the second input end of the visible light identifier are both connected to the output end of the decoder, the output end of the infrared light identifier and the output end of the visible light identifier are both connected to the splicing layer, and the splicing layer is used to output a fused image.

2. The visible light and infrared light image fusion method for road crack detection according to claim 1, characterized in that: The multi-scale feature extraction layer based on Nest connection includes 6 convolution layers; the input end of the first convolution layer, the input end of the fourth convolution layer and the input end of the sixth convolution layer are all used to input the visible light image of the target road and the infrared light image of the target road, the output end of the first convolution layer, the output end of the third convolution layer, the output end of the fourth convolution layer, the output end of the fifth convolution layer and the output end of the sixth convolution layer are all connected to the input end of the pyramid feature fusion module, the output end of the first convolution layer and the output end of the fourth convolution layer are also connected to the input end of the second convolution layer, the output end of the second convolution layer and the output end of the fifth convolution layer are also connected to the input end of the third convolution layer, the output end of the fourth convolution layer and the output end of the sixth convolution layer are also connected to the input end of the fifth convolution layer, and the convolution kernels of the first convolution layer, the fourth convolution layer and the sixth convolution layer are all different.

3. The visible light and infrared light image fusion method for road crack detection according to claim 2 is characterized in that: The pyramid feature fusion module includes a first convolution layer, a second convolution layer, a third convolution layer, a fourth convolution layer and a fifth convolution layer; The output end of the first convolutional layer is connected to the input end of the second convolutional layer and the input end of the fourth convolutional layer respectively; the output end of the second convolutional layer is connected to the input end of the third convolutional layer; the output end of the third convolutional layer is connected to the input end of the fifth convolutional layer; the output end of the fifth convolutional layer is connected to the input end of the fourth convolutional layer, and the output end of the fourth convolutional layer is connected to the input end of the second cascade operation, the input end of the third convolutional layer neural network, and the input end of the third cascade operation respectively; The output end of the first convolution layer and the output end of the third convolution layer in the multi-scale feature extraction layer are both connected to the input end of the first convolution layer in the pyramid feature fusion module; The output end of the fourth convolution layer and the output end of the fifth convolution layer in the multi-scale feature extraction layer are both connected to the input end of the second convolution layer in the pyramid feature fusion module; An output end of the sixth convolutional layer in the multi-scale feature extraction layer is connected to an input end of the third convolutional layer in the pyramid feature fusion module.

4. The visible light and infrared light image fusion method for road crack detection according to claim 1, characterized in that: The intensity attention mechanism includes: a high-pass filter and an intensity attention module; the intensity attention module includes a first maximum pooling layer, a first average pooling layer, a sixth convolutional layer and a first sigmoid function; the input ends of the first maximum pooling layer and the first average pooling layer are respectively connected to the output end of the high-pass filter, the output ends of the first maximum pooling layer and the first average pooling layer are respectively connected to the input end of the sixth convolutional layer, the output end of the sixth convolutional layer is connected to the input end of the first sigmoid function, and the input end of the high-pass filter is connected to the output end of the second convolution block; the output end of the first sigmoid function is connected to the input end of the first cross product operation.

5. The visible light and infrared light image fusion method for road crack detection according to claim 1, characterized in that: The channel attention mechanism is to replace the Shared MLP in the channel attention module with the channel attention mechanism of the fully connected layer.

6. The visible light and infrared light image fusion method for road crack detection according to claim 1, characterized in that: The texture attention mechanism includes: a second maximum pooling layer, a second average pooling layer, a seventh convolution layer, an addition operation, a fourth cascade operation, a second sigmoid function, a Soble operation, a Laplace operation, a first SimAM attention operation, and a second SimAM attention operation; The input end of the Soble operation, the input end of the Laplace operation, the input end of the second maximum pooling layer and the input end of the second average pooling layer are all connected to the output end of the fifth convolution block; the output end of the Laplace operation is connected to the input end of the first SimAM attention operation; the output end of the Soble operation is connected to the input end of the second SimAM attention operation; the output end of the second maximum pooling layer and the output end of the second average pooling layer are both connected to the input end of the seventh convolution layer, the output end of the second SimAM attention operation and the output end of the seventh convolution layer are both connected to the input end of the addition operation, the output end of the first SimAM attention operation and the output end of the addition operation are both connected to the input end of the fourth cascade operation, the output end of the fourth cascade operation is connected to the input end of the second sigmoid function, and the output end of the second sigmoid function is connected to the input end of the third cross product operation.

7. The visible light and infrared light image fusion method for road crack detection according to claim 1, characterized in that: The decoder includes a convolution block module, a ninth convolution layer and a tanh activation function connected in sequence; the convolution block module includes a plurality of sixth convolution blocks connected in sequence.

8. The visible light and infrared light image fusion method for road crack detection according to claim 7, characterized in that: The first convolution block, the second convolution block, the third convolution block, the fourth convolution block, the fifth convolution block and the sixth convolution block each include an eighth convolution layer, a BN layer and a Leaky ReLU activation function connected in sequence.

9. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the visible light and infrared light image fusion method for road crack detection described in any one of claims 1 to 8.

Citation Information

Patent Citations

  • Local missing face recognition method, chip and terminal

    CN118196874A

  • Infrared and visible light image fusion algorithm based on multi-scale decomposition

    CN118396864A

  • AGV Integrated Control Method

    KR1020240141987A

  • Pavement crack detection method, medium, and system

    WO2024092968A1

Cited By

  • Infrared and visible light fusion method based on spectrum attention generative adversarial network

    CN119762927A

  • Visible light-infrared image structure adaptive fusion crack detection method and system

    CN120047432A

  • Road apparent disease real-time identification method based on visible light and infrared image multi-modal fusion

    CN120147709A

  • A real-time road surface disease recognition method based on visible light and infrared image multi-modal fusion

    CN120147709B

  • Highway pavement disease detection method and system

    CN120182625A