Semantic segmentation method of RGB-D dam defect images based on bidirectional cross-modal attention mechanism

The semantic segmentation method for RGB-D dam defect images using a bidirectional cross-modal attention mechanism solves the problem of insufficient feature fusion in RGB-D image models for dam defect detection, achieving efficient and accurate defect identification and segmentation, and improving inspection efficiency and safety.

CN118334330BActive Publication Date: 2024-11-08HOHAI UNIV +2
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410358422.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-03-27
Publication Date
2024-11-08
Estimated Expiration
2044-03-27

AI Technical Summary

Technical Problem

Existing RGB-D image models struggle to fully extract and fuse features from RGB and depth images, resulting in poor semantic segmentation of dam defect images and impacting inspection accuracy and efficiency.

Method used

A semantic segmentation method for RGB-D dam defect images based on a bidirectional cross-modal attention mechanism is adopted. Multi-scale feature maps are extracted through a hierarchical Transformer network, and feature calibration and fusion are performed using channel attention mechanism and cross-modal attention mechanism to filter noise and enhance complementary features.

Benefits of technology

This improves the accuracy and efficiency of semantic segmentation of dam defect images, reduces labor costs, and ensures the safe operation of dams.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118334330B_ABST
    Figure CN118334330B_ABST
Patent Text Reader

Abstract

The present invention discloses a semantic segmentation method for RGB-D dam defect images based on a bidirectional cross-modal attention mechanism, which extracts features from an input RGB-D dam image, and uses two parallel hierarchical Transformer networks to extract RGB image feature maps and depth image feature maps of various resolution sizes in stages; the two feature maps extracted in each stage are sent together to a feature calibration module CA-FCM, and the features from the two modalities are mutually calibrated using a channel attention mechanism to filter out each other's noise information; the features calibrated at the same level are sent to a feature fusion module CM-FFM, and the RGB feature map and the depth feature map are further improved based on the cross-modal attention mechanism, and fused into a single feature map; the fused feature map is decoded by a decoder composed of a multi-layer perceptron and converted into a semantic segmentation map. The present invention solves the problem of insufficient feature extraction of RGB-D dam defect images and the inability to effectively fuse RGB feature maps and depth feature maps, thereby improving the accuracy of semantic segmentation of RGB-D dam defect images.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to an RGB-D dam defect image semantic segmentation method based on a bidirectional cross-modal attention mechanism, and belongs to the technical field of RGB-D dam defect image semantic segmentation. Background Art

[0002] RGB-D semantic segmentation is an advanced computer vision technology that uses images that contain both color (RGB) and depth (D) information to perform high-precision scene understanding and object recognition. This technology can more accurately determine the shape, size, and relative position of objects in the scene by analyzing the color and distance information of each pixel in the image, thereby achieving a deep understanding of the scene. Efficient RGB-D semantic segmentation models are currently widely used in many fields, including autonomous driving, robot navigation, augmented reality, and intelligent monitoring, providing these applications with real-time and accurate environmental perception capabilities.

[0003] Depth information plays a crucial role in the semantic segmentation of RGB-D images. Compared with traditional semantic segmentation based on RGB images, depth information provides additional spatial geometric clues for segmentation algorithms, helping to accurately segment objects under uneven lighting conditions or objects with similar visual features but different spatial positions. This capability is particularly suitable for the analysis of complex scenes, where relying solely on color information may not be able to accurately complete the task.

[0004] In the maintenance of construction projects and infrastructure, the use of RGB-D images for defect detection has gradually become an effective method. However, RGB images and depth images belong to two different modalities, making it difficult for the model to fully exploit the characteristics of both, resulting in poor semantic segmentation results for defect images. Summary of the invention

[0005] Purpose of the invention: To address the problems in the prior art, it is necessary to fully extract the feature information of RGB-D images, effectively fuse the two modalities, and design an efficient RGB-D semantic segmentation method, so as to help the engineering system automatically analyze RGB-D images and identify defects, and provide the maintenance team with detailed damage reports and repair suggestions, which will not only help to repair potential safety hazards in a timely manner, but also significantly reduce further damage and repair costs that may be caused by delayed maintenance.

[0006] Various defects may occur in the long-term operation of the dam. With the development of modern monitoring technology, equipment such as drones, mobile cameras and depth cameras can efficiently collect a large amount of high-quality image data. However, the collected dam defect images include two modes: RGB images and depth images. The RGB images describe the appearance features such as the shape of the dam defects, and the depth images record the spatial position information of the dam defects. It is not easy for the model to fully extract and utilize the features of these two images, resulting in poor semantic segmentation results for the defects. In order to overcome this problem, the present invention provides an RGB-D dam defect image semantic segmentation method based on a bidirectional cross-modal attention mechanism. By extracting multi-scale feature maps of RGB-D images and effectively fusing the complementary features of RGB-D images, the method can accurately locate and identify different types of dam defects, such as cracks, concrete spalling, etc. This not only greatly improves the accuracy and efficiency of inspections, but also reduces labor costs, which is of great significance for ensuring the safe operation of dams.

[0007] Technical solution: A semantic segmentation method for RGB-D dam defect images based on a bidirectional cross-modal attention mechanism, comprising the following steps:

[0008] (1) Feature extraction is performed on the input RGB-D dam defect image, and two parallel hierarchical Transformer networks are used to obtain multi-scale RGB feature maps and depth feature maps in stages; RGB-D image is the abbreviation of RGB image and depth image, which are two modal representations of dam defect images. RGB images can be collected by devices such as cameras, and depth images are collected by depth cameras.

[0009] (2) Extract the features of the two modes Figure 1 The features from the two modalities are then sent to the feature calibration module CA-FCM, which uses the channel attention mechanism to calibrate the features from the two modalities, filter out the noise information from each other, and make the defect information more obvious;

[0010] (3) The calibrated features are sent to the feature fusion module CM-FFM, and the RGB feature map and the depth feature map are further improved based on the cross-modal attention mechanism, so as to integrate the complementary features of the two modalities into a single feature map.

[0011] (4) The fused feature map is decoded by a decoder composed of a multi-layer perceptron to complete the semantic segmentation of the dam defect image.

[0012] The feature calibration module CA-FCM uses the channel attention mechanism to calibrate the feature maps of the two modalities. The specific steps are as follows:

[0013] (2.1) First, the RGB feature map and the depth feature map of the dam defect are concatenated, and then the global information of the two modalities is compressed through the global average pooling operation to obtain a cross-modal attention vector I:

[0014]

[0015] where F rgb represents the RGB feature map, F depth represents the deep feature map, || represents the concatenation operation of the two-modal feature map, Represents a global average pooling operation.

[0016] (2.2) Then, the attention vector for filtering noise information is obtained under the action of the multi-layer perceptron and the sigmoid function, and the attention vector is split into and And the weight value range is between 0 and 1:

[0017]

[0018] Where σ represents the sigmoid function, represents a multi-layer perceptron, Represents the splitting process, represents the attention vector for filtering the noise of the RGB feature map, Represents the attention vector for filtering the deep feature map.

[0019] (2.3) Finally, the RGB feature map and the depth feature map are respectively and Cross-multiply to obtain the feature map after removing noise and

[0020]

[0021]

[0022] in Represents the product of channel dimensions.

[0023] The specific steps of further improving and fusing the RGB feature map and the depth feature map through the feature fusion module CM-FFM based on the cross-modal attention mechanism are as follows:

[0024] (3.1) Calibrated RGB feature map It is sent to a 1×1 convolutional layer to generate three new feature maps and Similarly, on the deep branch Three new feature maps will also be generated and

[0025] (3.2) Through and Perform matrix multiplication to obtain the self-attention weights for the RGB image, and then use the softmax function to limit the weight range to (0, 1). Then, subtract the self-attention weight matrix from the identity matrix E to obtain the cross-modal attention vector The same operation is performed on the deep branch:

[0026]

[0027]

[0028] in Express and Transpose, represents the cross-modal attention vector on the deep branch, Represents matrix multiplication.

[0029] (3.3) The cross-modal attention vector With feature map Multiply it with the original feature map after filtering the noise Add together to get the feature map F′ after complementary information enhancement rgb , the same goes for deep branches:

[0030]

[0031]

[0032] The feature extraction network in the RGB-D semantic segmentation method is composed of a hierarchical Transformer, and the self-attention mechanism therein is replaced by a linear self-attention mechanism, which fully combines the advantages of CNN and Transformer, perceives image detail feature information while capturing long-distance dependencies, and the linear complexity also avoids computational redundancy.

[0033] The residual block used for the feature fusion module CM-FFM in the RGB-D semantic segmentation method consists of a 1×1 convolution layer, a 3×3 convolution layer, a RELU layer and a 1×1 convolution layer. Finally, a jump connection is realized through a 1×1 convolution layer, which can reduce the dimension of the feature map and facilitate subsequent decoding.

[0034] The feature fusion module CM-FFM in the RGB-D semantic segmentation method can fully extract complementary features and suppress the interference of redundant features in the multimodal fusion process. For example, when the self-attention weight at a certain RGB spatial position is 0.05, it means that the RGB feature at this position is not enough to complete accurate segmentation. Therefore, the network can use deep features to supplement the corresponding position. The weight of the deep feature is set to 0.95 and then combined with the feature map. Multiply as complementary features, and then add them to the original features to enrich the feature information; similarly, when the self-attention weight is high, the features of the other modality supplemented by the network are relatively few, thereby avoiding the redundancy of multimodal information. In this method, when the self-attention weight exceeds 0.5, it indicates that the feature map information of the modality is relatively rich, so the extraction of complementary features will be relatively small. The self-attention weight of the modal feature map is lower than 0.5, which indicates that it is relatively small.

[0035] The decoder in the RGB-D semantic segmentation method is composed only of an efficient and lightweight multi-layer perceptron, which directly concatenates and aggregates the received multi-level feature maps after unifying the dimensions, and finally converts them into semantic segmentation maps.

[0036] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the RGB-D dam defect image semantic segmentation method based on the bidirectional cross-modal attention mechanism as described above is implemented.

[0037] A computer-readable storage medium stores a computer program for executing the RGB-D dam defect image semantic segmentation method based on the bidirectional cross-modal attention mechanism as described above.

[0038] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0039] 1. In the feature extraction part, the hierarchical Transformer network based on the linear self-attention mechanism fully combines the advantages of CNN and Transformer, perceives the detailed feature information of the image while capturing long-distance dependencies, and the linear complexity also avoids computational redundancy, so that the model can better capture the characteristics of RGB-D dam defect images.

[0040] 2. The feature calibration module CA-FCM based on the channel attention mechanism can filter out the potential noise of the input RGB-D dam defect image and recalibrate the feature representation of RGB and depth, and perform cross-modal interaction during the feature propagation process to refine the features.

[0041] 3. The feature fusion module CM-FFM based on the cross-modal attention mechanism can effectively utilize the complementary features of RGB defect images and depth defect images for multimodal fusion, and suppress the redundant information in the feature map, so that the fused feature map has rich and clear multimodal information, which is conducive to accurate semantic segmentation of RGB-D dam defect images. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a general framework diagram of semantic segmentation of RGB-D dam defect image in a specific embodiment;

[0043] Figure 2 It is a structural diagram of a feature calibration module CA-FCM based on a channel attention mechanism in a specific embodiment;

[0044] Figure 3 It is a schematic diagram of the structure of the feature fusion module CM-FFM based on the cross-modal attention mechanism in a specific embodiment. DETAILED DESCRIPTION

[0045] The present invention is further explained below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0046] It is known that the inspection of a power station dam project is carried out by taking RGB-D defect images by image acquisition devices such as drones, mobile cameras and depth cameras. The images may contain four types of defects, namely cracks, alkaline precipitation, water seepage and concrete spalling. It is necessary to extract RGB-D image features through the model, distinguish the foreground and background, and accurately segment the defect positions in the image, thereby reducing the time of manual defect search and standardizing the inspection process.

[0047] Figure 1 The overall workflow of the RGB-D dam defect image semantic segmentation method based on the bidirectional cross-modal attention mechanism is given, and the specific implementation is as follows:

[0048] (1) Feature extraction is performed on the input RGB-D dam defect image. Two parallel hierarchical Transformer networks are used to extract RGB feature maps and depth feature maps of multiple resolutions in stages.

[0049] (2) The two feature maps obtained in the feature extraction stage are sent to the feature calibration module CA-FCM together, and the channel attention mechanism is used to calibrate the features from the two modalities and filter out the noise information from each other. Figure 2 shown.

[0050] (2.1) First, concatenate the two, and then compress the global information of the two modalities through the global average pooling operation to obtain a cross-modal attention vector I:

[0051]

[0052] where F rgb represents the RGB feature map, F depth represents the deep feature map, || represents the concatenation operation of the two-modal feature map, Represents a global average pooling operation.

[0053] (2.2) Then, the attention vector for filtering noise information is obtained under the action of the multi-layer perceptron and the sigmoid function, and it is split into and And the weight value range is between 0 and 1:

[0054]

[0055] Where σ represents the sigmoid function, represents a multi-layer perceptron, Represents the splitting process.

[0056] (2.3) Finally, the RGB feature map and the depth feature map are respectively and Cross-multiply to obtain the feature map after removing noise and

[0057]

[0058]

[0059] in Represents the product of channel dimensions.

[0060] (3) The features calibrated at the same level are then sent to the feature fusion module CM-FFM, which further improves the RGB feature map and the depth feature map based on the cross-modal attention mechanism and fuses them into a single feature map, such as Figure 3 shown.

[0061] (3.1) Calibrated RGB feature map It is sent to a 1×1 convolutional layer to generate three new feature maps and Similarly, on the deep branch Three new feature maps will also be generated and

[0062] (3.2) Through and Perform matrix multiplication to obtain the self-attention weights for the RGB image, and then use the softmax function to limit the weight range to (0, 1). Then, subtract the self-attention weight matrix from the identity matrix E to obtain the cross-modal attention vector The same operation is performed on the deep branch:

[0063]

[0064]

[0065] in Express and Transpose, represents the cross-modal attention vector on the deep branch, Represents matrix multiplication. Matrix subtraction can fully extract complementary features in the multimodal fusion process and suppress the interference of redundant features. For example, when the self-attention weight at a certain RGB spatial position is 0.05, it means that the RGB feature at this position is not enough to complete accurate segmentation. Therefore, the network can use deep features to supplement the corresponding position. The weight of the deep feature is set to 0.95 and then compared with the feature map. Multiplication is used as a complementary feature and then added to the original feature to enrich the feature information; similarly, when the self-attention weight is high, the features of the other modality supplemented by the network are relatively small, thereby avoiding the redundancy of multimodal information.

[0066] (3.3) The cross-modal attention vector With feature map Multiply it with the original feature map after filtering the noise Add together to get the feature map F′ after complementary information enhancement rgb , the same goes for deep branches:

[0067]

[0068]

[0069] (3.4) After enhancing the complementary information of the feature map, CM-FFM directly converts F′ rgb and F′ depth Concatenate to get the new feature map F′ fuse Finally, the new feature map is input into the residual block to obtain the fused feature map F fuse The residual block consists of a 1×1 convolutional layer, a 3×3 convolutional layer, a RELU layer, and a 1×1 convolutional layer. Finally, a 1×1 convolutional layer is used to implement a jump connection, which can reduce the dimension of the feature map and facilitate subsequent decoding.

[0070] (4) The feature maps after fusion at each stage will be decoded by a decoder composed of a multi-layer perceptron, and after unifying the dimensions, they will be directly converted into a semantic segmentation map.

[0071] Obviously, those skilled in the art should understand that the various steps of the RGB-D dam defect image semantic segmentation method based on the bidirectional cross-modal attention mechanism of the above-mentioned embodiment of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented with executable program codes of the computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be performed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present invention are not limited to any specific combination of hardware and software.

Claims

1. A semantic segmentation method for RGB-D dam defect images based on a bidirectional cross-modal attention mechanism, characterized in that: The method processes two modalities, RGB image and depth image of dam defects, and includes the following steps: (1) Feature extraction is performed on the input RGB-D dam defect image. Two parallel hierarchical Transformer networks are used to extract multi-scale RGB image feature maps and depth image feature maps in stages. (2) The feature maps of the two modalities extracted at each stage are sent to the feature calibration module CA-FCM, and the channel attention mechanism is used to calibrate the features from the two modalities and filter out the noise information from each other; (3) The calibrated features at the same level are sent to the feature fusion module CM-FFM, which further improves the RGB feature map and the depth feature map based on the cross-modal attention mechanism and fuses them into a single feature map; (4) The fused feature map is decoded by a decoder composed of a multi-layer perceptron to complete the semantic segmentation of the dam defect image. In (3), the RGB feature map and the depth feature map of the dam defect image are further improved based on the cross-modal attention mechanism and fused into a single feature map. The specific steps are as follows: (3.1) Calibrated RGB feature map It is sent to a 1×1 convolutional layer to generate three new feature maps and Similarly, on the deep branch Three new feature maps will also be generated and (3.2) Through and Perform matrix multiplication to obtain the self-attention weights for the RGB image, and then use the softmax function to limit the weight range to (0, 1). Then, subtract the self-attention weight matrix from the identity matrix E to obtain the cross-modal attention vector The same operation is performed on the deep branch: in Express and Transpose, represents the cross-modal attention vector on the deep branch, Represents matrix multiplication; (3.3) The cross-modal attention vector With feature map Multiply it with the original feature map after filtering the noise Add together to get the feature map F′ after complementary information enhancement rgb , the same goes for deep branches: (3.4) After enhancing the complementary information of the feature map, CM-FFM directly converts F′ rgb and F′ depth Concatenate to get the new feature map F′ fuse ; Finally, the new feature map is input into the residual block to obtain the fused feature map F fuse .

2. The RGB-D dam defect image semantic segmentation method based on bidirectional cross-modal attention mechanism according to claim 1 is characterized in that: In (2), the channel attention mechanism is used to calibrate the RGB feature map and the depth feature map of the dam defect image to each other and filter out the noise information of each other. The specific steps are as follows: (2.1) First, the RGB feature map and the depth feature map of the dam defect are concatenated, and then the global information of the two modalities is compressed through the global average pooling operation to obtain a cross-modal attention vector I: Among them, F rgb represents the RGB feature map, F depth represents the deep feature map, || represents the concatenation operation of the two-modal feature map, represents the global average pooling operation; (2.2) Then, the attention vector for filtering noise information is obtained under the action of the multi-layer perceptron and the sigmoid function, and the attention vector is split into and And the weight value range is between 0 and 1: Where σ represents the sigmoid function, represents a multi-layer perceptron, Indicates the splitting process; (2.3) Finally, the RGB feature map and the depth feature map are respectively and Cross-multiply to obtain the feature map after removing noise and in Represents the product of channel dimensions.

3. The RGB-D dam defect image semantic segmentation method based on bidirectional cross-modal attention mechanism according to claim 1 is characterized in that: The self-attention mechanism in the hierarchical Transformer of (1) is replaced by a linear self-attention mechanism.

4. The RGB-D dam defect image semantic segmentation method based on bidirectional cross-modal attention mechanism according to claim 1 is characterized in that: The residual block used for CM-FFM in (3) is composed of a 1×1 convolution layer, a 3×3 convolution layer, a RELU layer and a 1×1 convolution layer, and finally a 1×1 convolution layer is used to realize a jump connection to reduce the dimension of the feature map.

5. The RGB-D dam defect image semantic segmentation method based on bidirectional cross-modal attention mechanism according to claim 1 is characterized in that: The CM-FFM in (3) extracts complementary features and suppresses the interference of redundant features in the multimodal fusion process; When the self-attention weight at a certain RGB spatial position is 0.05, it means that the RGB feature at that position is not enough to complete accurate segmentation. Therefore, the network uses deep features to supplement the corresponding position. The weight of the deep feature is set to 0.95 and then compared with the feature map. Multiply as complementary features, and then add them to the original features to enrich the feature information.

6. The RGB-D dam defect image semantic segmentation method based on bidirectional cross-modal attention mechanism according to claim 1 is characterized in that: The decoder of (4) is composed only of a multi-layer perceptron, which directly concatenates and aggregates the received multi-level features after unifying the dimensions, and finally converts them into a semantic segmentation map.

7. A computer device, characterized in that: The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the RGB-D dam defect image semantic segmentation method based on a bidirectional cross-modal attention mechanism as described in any one of claims 1 to 6 is implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the RGB-D dam defect image semantic segmentation method based on a bidirectional cross-modal attention mechanism as described in any one of claims 1-6.