Dam defect real-time semantic segmentation method and device for inspecting unmanned vehicle

Abstract

The invention provides a dam defect real-time semantic segmentation method and device for an inspection unmanned vehicle, and the method comprises the steps: carrying out the feature extraction of an RGB-D defect image of a dam monitored by the inspection unmanned vehicle, generating a multi-stage RGB image and a depth image based on a ConvNeXt-T network module, and inputting the RGB image and the depth image to a pooling attention mechanism for feature enhancement; the feature maps after feature enhancement in the same stage are sent to a feature fusion module, optimization is carried out based on a cross-modal attention mechanism, and fusion is carried out to obtain a multi-level fusion feature map; sending the last level of fusion feature map into a cavity pyramid pooling module to generate a multi-scale semantic map; inputting adjacent three levels of fusion feature maps into an up-sampling recovery module, and performing multiple convolution and pooling operations based on different representation and context information of each fusion feature map to obtain a semantic segmentation map; and inputting an upper sampling layer to generate a dam defect real-time semantic segmentation result. Therefore, real-time semantic segmentation with double feature enhancement is carried out, and the reasoning speed and segmentation precision of real-time semantic segmentation are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of real-time semantic segmentation, and in particular to a method and device for real-time semantic segmentation of dam defects for inspection of unmanned vehicles. Background Art

[0002] As an extremely important image understanding task in the field of computer vision, semantic segmentation has received widespread attention in recent years. Through this technology, computers can accurately assign each pixel in an image to a predefined semantic category, thereby achieving a deep understanding of the image scene. This refined classification not only helps computers identify objects and their boundaries in the image, but also better understand complex scenes and backgrounds in the image. This is of great significance to many practical application fields such as autonomous driving, medical image analysis, and remote sensing image processing.

[0003] However, relying solely on a single RGB image is not enough to accurately segment complex environments. RGB-D semantic segmentation enhances the model's ability to understand the geometric structure of the scene by introducing depth images. However, the increase in depth images often requires additional structures in the network to process depth information, which greatly increases the time cost of semantic segmentation.

[0004] Edge devices such as dam inspection robots are extremely sensitive to speed requirements in defect segmentation and need to be processed online in real time on edge devices. However, edge devices have limited computing power and memory, and low computing power. These precise networks require a lot of computer processing resources, so they are not suitable for edge devices. Existing model structure streamlining solutions often lead to limited model feature extraction capabilities, loss of detailed information such as the edges and overlapping parts of objects, and reduced model segmentation accuracy, making it impossible to accurately complete real-time semantic segmentation of dam defects. Summary of the invention

[0005] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0006] To this end, the first purpose of the present invention is to propose a real-time semantic segmentation method for dam defects for unmanned vehicle inspection, perform real-time semantic segmentation with dual feature enhancement, and improve the reasoning speed and segmentation accuracy of real-time semantic segmentation.

[0007] The second object of the present invention is to provide a real-time semantic segmentation device for dam defects used for unmanned vehicle inspection.

[0008] A third objective of the present invention is to provide an electronic device.

[0009] A fourth object of the present invention is to provide a non-transitory computer-readable storage medium storing computer instructions.

[0010] To achieve the above-mentioned purpose, the first embodiment of the present invention proposes a real-time semantic segmentation method for dam defects for unmanned vehicle inspection, wherein the method processes two modalities of RGB image and depth image of dam defects, and comprises the following steps:

[0011] Obtain the RGB-D defect images that need to be processed when the unmanned vehicle is used to monitor the dam, and extract the feature maps of the two modalities of multi-scale RGB image and depth image in stages through the ConvNeXt-T network module. Through the feature enhancement module based on the pooling attention mechanism, the feature maps of the two modalities at different stages are enhanced to generate the RGB enhanced feature maps and depth enhanced feature maps at each stage.

[0012] The RGB enhanced feature maps and depth enhanced feature maps of each stage are sent to the feature fusion module CM-FFM for fusion. The RGB enhanced feature maps and depth enhanced feature maps of each stage are optimized based on the cross-modal attention mechanism and fused into a single fused feature map of each stage to obtain multi-level fused feature maps F1, F2, F3, and F4.

[0013] The last level fusion feature map F4 is input into the residual link-based dilated spatial pyramid pooling module to generate a multi-scale semantic map;

[0014] The adjacent three-level fusion feature maps F1, F2, and F3 are input into the semantic-aware upsampling feature recovery module. Based on the different representations and context information of F1, F2, and F3, multiple convolution and pooling operations are performed on the multi-scale semantic map to obtain a semantic segmentation map with added complementary feature information.

[0015] The semantic segmentation map is input into the upsampling layer to generate a real-time semantic segmentation result of the dam defect.

[0016] To achieve the above-mentioned purpose, the second embodiment of the present invention proposes a real-time semantic segmentation device for dam defects for unmanned vehicle inspection, wherein the device processes two modalities of RGB image and depth image of dam defects, and comprises the following steps:

[0017] The first feature enhancement module is used to obtain the RGB-D defect images that need to be processed when the unmanned vehicle monitors the dam, and extract the feature maps of the two modalities of multi-scale RGB image and depth image in stages through the ConvNeXt-T network module. The feature enhancement module based on the pooling attention mechanism is used to enhance the feature maps of the two modalities at different stages to generate RGB enhanced feature maps and depth enhanced feature maps at each stage;

[0018] The fusion module is used to send the RGB enhanced feature map and the depth enhanced feature map of each stage to the feature fusion module CM-FFM for fusion, optimize the RGB enhanced feature map and the depth enhanced feature map of each stage based on the cross-modal attention mechanism, and fuse them into a single fused feature map of each stage to obtain multi-level fused feature maps F1, F2, F3, and F4;

[0019] The first generation module is used to input the last level fusion feature map F4 into the residual link-based hollow space pyramid pooling module to generate a multi-scale semantic map;

[0020] The second feature enhancement module is used to input the adjacent three-level fusion feature maps F1, F2, and F3 into the semantic-aware upsampling feature recovery module, and perform multiple convolution and pooling operations on the multi-scale semantic map based on the different representations and context information of F1, F2, and F3 to obtain a semantic segmentation map with added complementary feature information;

[0021] The second generation module is used to input the semantic segmentation map into the upsampling layer to generate a real-time semantic segmentation result of the dam defect.

[0022] To achieve the above-mentioned purpose, the third aspect of the present invention proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the first aspect.

[0023] In order to achieve the above-mentioned purpose, the fourth aspect of the present invention proposes a non-transitory computer-readable storage medium storing computer instructions, where the computer instructions are used to enable the computer to execute the method described in the first aspect.

[0024] The real-time semantic segmentation method, device, electronic device and storage medium for dam defects for inspection unmanned vehicles provided by the embodiment of the present invention extract features from the RGB-D defect image of the dam monitored by the inspection unmanned vehicle, generate multi-stage RGB images and depth images based on the ConvNeXt-T network module, and then input them into the pooling attention mechanism for feature enhancement; the feature map after feature enhancement in the same stage is sent to the feature fusion module, optimized based on the cross-modal attention mechanism, and fused to obtain a multi-level fusion feature map; the last level of fusion feature map is sent to the hollow pyramid pooling module to generate a multi-scale semantic map; the adjacent three-level fusion feature map is input into the upsampling recovery module, and multiple convolution and pooling operations are performed based on the different representations and context information of each fusion feature map to obtain a semantic segmentation map; and then the upsampling layer is input to generate the real-time semantic segmentation result of the dam defect. Thus, real-time semantic segmentation with dual feature enhancement is performed, which improves the reasoning speed and segmentation accuracy of real-time semantic segmentation.

[0025] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0027] Figure 1 A flow chart of a real-time semantic segmentation method for dam defects for unmanned vehicle inspection provided by an embodiment of the present invention;

[0028] Figure 2 A schematic diagram of the structure of a feature enhancement module based on a pooling attention mechanism provided by an embodiment of the present invention;

[0029] Figure 3 A schematic diagram of the structure of a dilute pyramid pooling module based on residual linking provided by an embodiment of the present invention;

[0030] Figure 4 A schematic diagram of the structure of an upsampling feature recovery module based on semantic perception provided by an embodiment of the present invention;

[0031] Figure 5 An overall flow chart of a real-time semantic segmentation method for dam defects provided by an embodiment of the present invention;

[0032] Figure 6 A schematic diagram of the structure of a real-time semantic segmentation device for dam defects for inspecting unmanned vehicles provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0033] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0034] It should be noted that the acquisition, storage, use, and processing of data in the technical solution of the present invention are in compliance with the relevant provisions of national laws and regulations.

[0035] The following describes a method and apparatus for real-time semantic segmentation of dam defects for unmanned vehicle inspection according to an embodiment of the present invention with reference to the accompanying drawings.

[0036] Figure 1 A flowchart of a real-time semantic segmentation method for dam defects for unmanned vehicle inspection provided by an embodiment of the present invention is provided.

[0037] like Figure 1 As shown, the method processes two modalities, RGB image and depth image of dam defects, and the method includes the following steps:

[0038] Step 101, obtain the RGB-D defect image to be processed of the dam monitored by the unmanned inspection vehicle, and extract the feature maps of the two modalities of multi-scale RGB image and depth image in stages through the ConvNeXt-T network module, and enhance the feature maps of the two modalities in different stages through the feature enhancement module based on the pooling attention mechanism to generate the RGB enhanced feature map and the depth enhanced feature map of each stage.

[0039] In some embodiments, an RGB-D defect image to be processed of a dam monitored by an unmanned patrol vehicle is obtained, and feature maps of two modalities, multi-scale RGB image and depth image, are extracted in stages through a ConvNeXt-T network module. Feature enhancement module PA-FEM based on pooling attention mechanism is used to enhance the feature maps of the two modalities at different stages to generate RGB enhanced feature maps and depth enhanced feature maps at each stage. An implementation method may be to cut the inspection picture of the dam monitored by an unmanned patrol vehicle into several image blocks of 4×4 size as RGB-D defect images to be processed, and as input of the ConvNeXt-T network module, wherein there is a certain amount of image overlap in the image blocks; the image blocks are input into the ConvNeXt-T network module, and after passing through the ConvNeXt-T network module After each depth separation convolution operation, a first RGB image and a first depth image are generated respectively; based on the adaptive average pooling operation in the feature enhancement module of the pooling attention mechanism, the first RGB image and the first depth image are reduced in dimension, and the second RGB image and the second depth image after dimension reduction are subjected to the maximum pooling operation to obtain a third RGB image and a third depth image; a 1×1 convolution operation is performed on the third RGB image and the third depth image, and the sigmoid function is used to activate the attention weight value, and the attention weight value is scaled to between 0 and 1 to obtain the pooling attention vector of the RGB image and the depth image; the first RGB image and the pooling attention vector of the RGB image are weightedly summed to obtain the RGB enhanced feature map of each feature stage; the first depth image and the pooling attention vector of the depth image are weighted summed to obtain the depth enhanced feature map of each stage.

[0040] Specifically, Figure 2As shown in the figure, the RGB image is a color image composed of three color channels: red, green and blue, and the depth image is a grayscale image. The images in the dataset (RGB-D defect images) are enhanced by random horizontal flipping and random scale scaling (0.5-1.75) to expand the dataset. For the original inspection image, the network first divides the inspection image into several 4×4 image blocks with a certain image overlap as the input of the ConvNeXt-T network module.

[0041] The obtained image block is input into the first ConvNeXt-T network block. After each depth-separable convolution operation in the ConvNeXt-T network module, the two branches generate the first RGB image F rgb and the first depth image F depth .

[0042] The ConvNeXt-T network block consists of a depth-separable convolutional layer and two 1*1 convolutions. The height, width, and number of channels of the input image are recorded as h×ω×dim. First, a depth-separable convolutional layer with a convolution kernel size of 7, a stride of 1, and a padding of 1 is passed, and then the initial features with unchanged size are obtained through linear normalization. Then, a 1*1 convolutional layer with a convolution kernel size of 1 and a stride of 1 and a GELU activation function are used to convert the feature map size to h×ω×dim. Finally, a 1*1 convolution is used to restore the feature map to h×ω×dim and then add it to the original input to obtain the feature map.

[0043] Adaptive average pooling operation in feature enhancement module based on pooling attention mechanism (F ap ), for the first RGB image (F rgb ∈R C×H×W ) and the first depth image (F depth ∈R C×H×W ) is used for dimensionality reduction, and the second RGB image after dimensionality reduction is and the second depth image Perform the maximum pooling operation (F mp ), get the third RGB image and the third depth image As shown in formula (1) and formula (2):

[0044]

[0045]

[0046] Will and Perform 1×1 convolution (Conv 1×1)) operation, and use the sigmoid function activation to scale the attention weight value to between 0 and 1, thereby obtaining the pooled attention vector of the RGB image and the pooled attention vector of the depth image As shown in formulas (3) and (4):

[0047]

[0048]

[0049] The first RGB image F rgb Pooled attention vector with RGB image Perform weighted summation (matrix weighted summation) to obtain the RGB enhanced feature map at each stage of the feature The first depth image F depth Pooled attention vector with depth image Perform weighted summation to obtain the depth enhancement feature map of each stage As shown in formulas (5) and (6):

[0050]

[0051]

[0052] In step 102, the RGB enhanced feature map and the depth enhanced feature map of each stage are sent to the feature fusion module CM-FFM for fusion, and the RGB enhanced feature map and the depth enhanced feature map of each stage are optimized based on the cross-modal attention mechanism, and fused into a single fused feature map of each stage to obtain multi-level fused feature maps F1, F2, F3, and F4.

[0053] In some embodiments, the RGB enhanced feature map and the depth enhanced feature map of each stage are sent to the feature fusion module CM-FFM for fusion, and the RGB enhanced feature map and the depth enhanced feature map of each stage are optimized based on the cross-modal attention mechanism, and fused into a single fused feature map of each stage to obtain a multi-level fused feature map F1, F2, F3, F4. An implementation method can be that when the RGB enhanced feature map and the depth enhanced feature map of each stage are sent to the feature fusion module CM-FFM for fusion, the RGB enhanced feature map and the depth enhanced feature map of each stage are input into a 1×1 convolution layer to generate a fourth RGB image and the fourth depth image Among them, N = H × W; by and Perform matrix multiplication to obtain the self-attention weight corresponding to the RGB image, and use the softmax function to limit the weight range to (0, 1). Subtract the matrix of self-attention weights from the identity matrix E to obtain the cross-modal attention vector corresponding to the RGB image. Similarly, according to the fourth depth image and Determine the cross-modal attention vector corresponding to the depth image The cross-modal attention vector Multiply and add to the RGB enhanced feature map to obtain the fifth RGB image F' after complementary information enhancement rgb ; The cross-modal attention vector Multiply and add the depth enhancement feature map to obtain the five depth images F' after complementary information enhancement depth ; The fifth RGB image and the fifth depth image are spliced ​​to generate a single fused feature map of each stage to obtain multi-level fused feature maps F1, F2, F3, and F4, wherein the single fused feature map is restored to the original size of the RGB-D defect image through a residual connection.

[0054] Specifically, when the RGB enhanced feature maps and depth enhanced feature maps of each stage are sent to the feature fusion module CM-FFM for fusion, the RGB enhanced feature maps of each stage are and depth-enhanced feature maps Input to the 1×1 convolution layer to generate the fourth RGB image and the fourth depth image Among them, N = H × W.

[0055] By and Perform matrix multiplication to obtain the self-attention weight corresponding to the RGB image, and use the softmax function to limit the weight range to (0, 1). Subtract the matrix of self-attention weights from the identity matrix E to obtain the cross-modal attention vector corresponding to the RGB image. Similarly, according to the fourth depth image and Determine the cross-modal attention vector corresponding to the depth image As shown in formulas (7) and (8):

[0056]

[0057]

[0058] The cross-modal attention vector Multiply and sum Add them together to get the fifth RGB image F' after complementary information enhancement rgb; The cross-modal attention vector Multiply and sum Add them together to get the five depth images F' after complementary information enhancement depth , as shown in formulas (9) and (10):

[0059]

[0060]

[0061] The fifth RGB image F' rgb and the fifth depth image F' depth Splice to generate a single fusion feature map F' at each stage fuse ∈R 2C×H×W , where F' fuse ∈R 2C×H×W It is input into the residual block of the residual connection, which is composed of a 1×1 convolution layer, a 3×3 convolution layer, a RELU layer and a 1×1 convolution layer, and finally a convolution layer is used to realize jump connection to obtain a fused feature map. The first fused feature map obtained is called feature map F1, and then multi-level fused feature maps F1, F2, F3, and F4 are obtained.

[0062] Step 103, input the last level fusion feature map F4 into the residual link based hollow space pyramid pooling module to generate a multi-scale semantic map.

[0063] In some embodiments, Figure 3 As shown in the figure, the residual link-based atrous spatial pyramid pooling module RASPP contains five branches, among which the first branch performs global average pooling on the input feature map; the second branch is a 1×1 convolution layer; the third, fourth, and fifth branches are atrous convolutions (3×3) with residual links (1×1 convolutions) and the atrous rates of 6, 12, and 18, respectively.

[0064] Specifically, when the last-level fusion feature map F4 is input into the dilute spatial pyramid pooling module based on residual links, it is processed sequentially by the five branches of the dilute spatial pyramid pooling module based on residual links, and finally the feature maps output by the five branches are spliced ​​to form the final multi-scale semantic map F5.

[0065] The second branch is a 1×1 convolution layer, which is used to reduce the dimension of F4 and promote the fusion of feature information. The residual connection uses the convolution operation to smooth the input F4 to alleviate the information discontinuity problem caused by the use of dilated convolution. Subsequently, the convolution-processed F4 is fused with the feature map obtained by dilated convolution to further enrich the spatial and edge information of F5.

[0066] Step 104, input the adjacent three-level fusion feature maps F1, F2, and F3 into the semantic-aware upsampling feature recovery module, and perform multiple convolution and pooling operations on the multi-scale semantic map based on the different representations and context information of F1, F2, and F3 to obtain a semantic segmentation map with added complementary feature information.

[0067] In some embodiments, the adjacent three-level fusion feature maps F1, F2, and F3 are input into the semantic-aware upsampling feature recovery module SU-FRM, and based on the different representations and context information of F1, F2, and F3, multiple convolution and pooling operations are performed on the multi-scale semantic map to obtain a semantic segmentation map with added complementary feature information. An implementation method may be that after the adjacent three-level fusion feature maps F1, F2, and F3 are input into the semantic-aware upsampling feature recovery module. F 2-i It is processed by 3×3 convolution, normalization and ReLu activation function in the sampling feature recovery module to keep the number of channels and size of the feature map unchanged. 3-i Through the global pooling, convolution and sigmoid activation functions in the sampling feature recovery module, a first global context vector is obtained to express different representations and context information of the fused feature map, where i∈(0,1,2); the processed F 2-i Multiply it with the global context vector to get the weighted low-level feature map In the high-level feature map F 3-i and Add the first stage result; multiply the first stage result with the second global context vector to obtain the second stage intermediate feature. 3-i After upsampling, it is added to the intermediate features of the second stage to obtain a semantic segmentation map with added complementary feature information, where the second global context vector is composed of the feature image F i 'Get. When i = 0, F i ' represents the multi-scale semantic map obtained by the dilated spatial pyramid pooling module based on residual links; when i = 1, 2, F i ' represents the semantic segmentation map output by the i-th upsampling feature recovery module.

[0068] Specifically, after the adjacent fused feature maps F1, F2, F3, and F4 are input into the semantic-aware upsampling feature recovery module. 2-i The 3×3 convolution, normalization, and ReLu activation functions in the sampling feature recovery module are sequentially processed to keep the number of channels and size of the fused feature map unchanged. 3-i Through the global pooling (global average pooling), 1×1 convolution and sigmoid activation function in the sampling feature recovery module, a fusion feature map (F 2-i) The first global context vector with different representations and context information Among them, i∈(0,1,2).

[0069] After processing F 2-i With the global context vector Multiply them to get the weighted low-level feature map In the high-level feature map F 3-i and Add the first stage result and the second global context vector Multiply to obtain the second stage intermediate features, and then 3-i After upsampling, it is added to the intermediate features of the second stage to obtain the semantic segmentation map (F6, F7, F8) with added complementary feature information. Among them, the second global context vector is composed of the feature image F i 'Get. When i = 0, F i ' represents the multi-scale semantic map obtained by the dilated spatial pyramid pooling module based on residual links; when i = 1, 2, F i ' represents the semantic segmentation map output by the i-th upsampling feature recovery module.

[0070] Furthermore, if Figure 4 As shown in the figure, when processing adjacent F3 and F4, first, F3 will be processed by 3×3 convolution, normalization (batch normalization) and ReLu activation function in sequence, keeping the number of channels and size of F3 unchanged. At the same time, F4 will obtain a global context vector through global pooling, 1×1 convolution and sigmoid activation function in sequence. The processed F3 will be combined with the global context vector Multiply them to get the weighted low-level feature map Finally, F4 is upsampled and Add together to get the result F of the first stage stage1 .

[0071] Similarly, the high-level feature map F5 is sequentially passed through global pooling, 1×1 convolution and sigmoid activation function to obtain a global context vector Then the first stage result F stage1 With the global context variable I3 1 Multiply to obtain the second stage intermediate features Finally, F5 is upsampled and Add together to get the final restored feature map F6.

[0072] Step 105, inputting the semantic segmentation map into the upsampling layer to generate a real-time semantic segmentation result of the dam defect.

[0073] In some embodiments, the defective area of ​​the dam can be determined based on the real-time semantic segmentation results, and then a defect repair strategy is generated for the defective area for repair.

[0074] The real-time semantic segmentation method for dam defects for inspection unmanned vehicles of the embodiment of the present invention extracts features from the RGB-D defect images of the dam monitored by the inspection unmanned vehicle, generates multi-stage RGB images and depth images based on the ConvNeXt-T network module, and then inputs them into the pooling attention mechanism for feature enhancement; the feature map after feature enhancement in the same stage is sent to the feature fusion module, optimized based on the cross-modal attention mechanism, and fused to obtain a multi-level fusion feature map; the last level of fusion feature map is sent to the hollow pyramid pooling module to generate a multi-scale semantic map; the adjacent three-level fusion feature map is input into the upsampling recovery module, and multiple convolution and pooling operations are performed based on the different representations and context information of each fusion feature map to obtain a semantic segmentation map; and then the upsampling layer is input to generate the real-time semantic segmentation result of the dam defect. Thus, real-time semantic segmentation with dual feature enhancement is performed, which improves the reasoning speed and segmentation accuracy of real-time semantic segmentation.

[0075] In addition, in some other embodiments, the real-time semantic segmentation method of dam defects used for patrol unmanned vehicles is migrated to the patrol unmanned vehicle system, and the patrol unmanned vehicle system includes: a data acquisition module, used to acquire and store dam images within the shooting range; an image segmentation module, used to receive the dam images taken in real time by the patrol unmanned vehicle, and based on the real-time semantic segmentation results of dam defects analyzed by the real-time semantic segmentation method of dam defects used for patrol unmanned vehicles, identify and segment the defective areas of the dam; a data transmission module, used to transmit the image information and mask information of the defective area; an inspection subsystem, used to receive image information and mask information, and identify the defect location, generate an inspection task list based on the defect location, and send the inspection task list to the control subsystem; the control subsystem, used to control the patrol unmanned vehicle to shoot, identify, and transmit the defective area to be identified according to the inspection task list.

[0076] Specifically, the data acquisition module usually uses a high-resolution camera installed in a fixed position or on an unmanned inspection vehicle to ensure coverage of key areas of the dam. The data of these dam images will be stored in real time on a local storage device or uploaded to a cloud database.

[0077] The data transmission module transmits the image information and mask information of the identified defective area to the inspection subsystem via a wireless network or other communication methods.

[0078] The control subsystem receives the inspection task list from the inspection system. The inspection task list includes inspection tasks and moving routes. According to the inspection task list, the inspection unmanned vehicle is controlled to complete the inspection task to ensure that the inspection unmanned vehicle moves according to the planned route, accurately photographs the key areas of the dam, monitors the movement status and data collection of the inspection unmanned vehicle, handles abnormal situations in a timely manner, and ensures the smooth completion of the inspection task.

[0079] In order to clearly explain the above embodiment, Figure 5 The overall flow chart of a real-time semantic segmentation method for dam defects provided by an embodiment of the present invention. Specifically, the feature maps of the two modalities of multi-scale RGB image and depth image are extracted in stages (ConvNeXt1, ConvNeXt2, ConvNeXt3, ConvNeXt4) through the ConvNeXt-T network module, and the feature enhancement module based on the pooling attention mechanism is used to enhance the feature maps of the two modalities at different stages to generate RGB enhanced feature maps and depth enhanced feature maps at each stage; the RGB enhanced feature maps and depth enhanced feature maps at each stage are sent to the feature fusion module CM-FFM for fusion (feature fusion), and the RGB enhanced feature maps and depth enhanced feature maps at each stage are optimized based on the cross-modal attention mechanism. The feature maps are deeply enhanced and fused into single fused feature maps at each stage to obtain multi-level fused feature maps F1, F2, F3, and F4; the last level fused feature map F4 is input into the void space pyramid pooling module (pyramid pooling) based on residual link to generate a multi-scale semantic map F5; the adjacent three-level fused feature maps F1, F2, and F3 are input into the semantic-aware upsampling feature recovery module, and based on the different representations and contextual information of F1, F2, and F3, multiple convolution and pooling operations F6, F7, and F8 are performed on the multi-scale semantic map, and the semantic segmentation map F8 with added complementary feature information is input into the upsampling layer to generate real-time semantic segmentation results of dam defects.

[0080] In order to implement the above-mentioned embodiment, the present invention also proposes a real-time semantic segmentation device for dam defects for inspection of unmanned vehicles.

[0081] Figure 6 A schematic diagram of the structure of a real-time semantic segmentation device for dam defects for inspecting unmanned vehicles provided in an embodiment of the present invention.

[0082] like Figure 6 As shown, the real-time semantic segmentation device 60 for dam defects for unmanned vehicle inspection includes: a first feature enhancement module 61, a fusion module 62, a first generation module 63, a second feature enhancement module 64, and a second generation module 65.

[0083] The first feature enhancement module 61 is used to obtain the RGB-D defect image to be processed for the unmanned vehicle monitoring the dam, and extract the feature maps of the multi-scale RGB image and the depth image in stages through the ConvNeXt-T network module, and enhance the feature maps of the two modalities in different stages through the feature enhancement module based on the pooling attention mechanism to generate the RGB enhanced feature map and the depth enhanced feature map obtained in each stage;

[0084] A fusion module 62 is used to send the RGB enhanced feature map and the depth enhanced feature map of each stage to the feature fusion module CM-FFM for fusion, optimize the RGB enhanced feature map and the depth enhanced feature map of each stage based on the cross-modal attention mechanism, and fuse them into a single fused feature map of each stage to obtain multi-level fused feature maps F1, F2, F3, and F4;

[0085] A first generating module 63 is used to input the last level fusion feature map F4 into a residual link based dilated space pyramid pooling module to generate a multi-scale semantic map;

[0086] The second feature enhancement module 64 is used to input the adjacent three-level fusion feature maps F1, F2, and F3 into the semantic-aware upsampling feature recovery module, and perform multiple convolution and pooling operations on the multi-scale semantic map based on the different representations and context information of F1, F2, and F3 to obtain a semantic segmentation map with added complementary feature information;

[0087] The second generating module 65 is used to input the semantic segmentation map into the upsampling layer to generate a real-time semantic segmentation result of the dam defect.

[0088] Furthermore, in a possible implementation of the embodiment of the present invention, the first feature enhancement module 61 is specifically configured to:

[0089] The inspection pictures of the unmanned vehicle monitoring the dam are cut into several 4×4 image blocks as the RGB-D defect images to be processed and used as the input of the ConvNeXt-T network module, where the image blocks have a certain amount of image overlap;

[0090] Input the image block into a ConvNeXt-T network module, and after each depth-separable convolution operation in the ConvNeXt-T network module, generate a first RGB image and a first depth image respectively;

[0091] Based on the adaptive average pooling operation in the feature enhancement module of the pooling attention mechanism, the first RGB image and the first depth image are reduced in dimension, and the maximum pooling operation is performed on the reduced-dimensional second RGB image and the second depth image to obtain a third RGB image and a third depth image;

[0092] Perform a 1×1 convolution operation on the third RGB image and the third depth image, activate them using the sigmoid function, scale the attention weight value to between 0 and 1, and obtain the pooled attention vector of the RGB image and the depth image;

[0093] The first RGB image is weightedly summed with the pooled attention vector of the RGB image to obtain the RGB enhanced feature map at each feature stage;

[0094] The first depth image and the pooled attention vector of the depth image are weightedly summed to obtain the depth enhanced feature map of each stage.

[0095] Furthermore, in a possible implementation of the embodiment of the present invention, the fusion module 62 is specifically configured to:

[0096] When the RGB enhanced feature map and the depth enhanced feature map of each stage are sent to the feature fusion module CM-FFM for fusion, the RGB enhanced feature map and the depth enhanced feature map of each stage are input into the 1×1 convolution layer to generate the fourth RGB image and the fourth depth image Where, N = H × W;

[0097] Through and Perform matrix multiplication to obtain the self-attention weight corresponding to the RGB image, and use the softmax function to limit the weight range to (0, 1). Subtract the matrix of self-attention weights from the identity matrix E to obtain the cross-modal attention vector corresponding to the RGB image. Similarly, according to the fourth depth image and Determine the cross-modal attention vector corresponding to the depth image

[0098] The cross-modal attention vector Multiply and add to the RGB enhanced feature map to obtain the fifth RGB image F' after complementary information enhancement rgb ; The cross-modal attention vector Multiply and add the depth enhancement feature map to obtain the five depth images F' after complementary information enhancement depth ;

[0099] The fifth RGB image and the fifth depth image are spliced ​​to generate a single fused feature map of each stage to obtain multi-level fused feature maps F1, F2, F3, and F4, wherein the single fused feature map is restored to the original size of the RGB-D defect image through a residual connection.

[0100] Furthermore, in a possible implementation of an embodiment of the present invention, the residual link-based dilated spatial pyramid pooling module includes five branches, wherein the first branch performs global average pooling on the input feature map; the second branch is a 1×1 convolution layer; the third, fourth, and fifth branches are dilated convolutions with dilated rates of 6, 12, and 18, respectively, that introduce residual links.

[0101] Further, in a possible implementation of the embodiment of the present invention, the second feature enhancement module 64 is specifically configured to:

[0102] After the adjacent three-level fusion feature maps F1, F2, and F3 are input into the semantic-aware upsampling feature recovery module. 2-i It is processed by 3×3 convolution, normalization and ReLu activation function in the sampling feature recovery module to keep the number of channels and size of the feature map unchanged. 3-i Through the global pooling, 1×1 convolution and sigmoid activation function in the sampling feature recovery module, a first global context vector is obtained that represents different representations and context information of the fused feature map, where i∈(0,1,2);

[0103] After processing F 2-i Multiply it with the global context vector to get the weighted low-level feature map In the high-level feature map F 3-i and Add together to get the first stage result;

[0104] The first stage result is multiplied by the second global context vector to obtain the second stage intermediate feature. 3-i After upsampling, it is added to the intermediate features of the second stage to obtain a semantic segmentation map with added complementary feature information, where the second global context vector is composed of the feature image F i 'Get. When i = 0, F i ' represents the multi-scale semantic map obtained by the dilated spatial pyramid pooling module based on residual links; when i = 1, 2, F i ' represents the semantic segmentation map output by the i-th upsampling feature recovery module.

[0105] Furthermore, in a possible implementation of the embodiment of the present invention, the real-time semantic segmentation device for dam defects used for patrol unmanned vehicles is migrated to the patrol unmanned vehicle system, and the patrol unmanned vehicle system includes:

[0106] A data acquisition module, used for acquiring and storing images of the dam within the shooting range;

[0107] An image segmentation module is used to receive the dam image taken in real time by the unmanned inspection vehicle, and identify and segment the defective area of ​​the dam based on the real-time semantic segmentation result of the dam defect analyzed by the real-time semantic segmentation method of the dam defect for the unmanned inspection vehicle;

[0108] A data transmission module, used for transmitting the image information and mask information of the defect area;

[0109] The inspection subsystem is used to receive the image information and the mask information, identify the defect location, generate an inspection task list based on the defect location, and send the inspection task list to the control subsystem;

[0110] The control subsystem is used to control the unmanned inspection vehicle to photograph, identify and transmit the defective areas to be identified according to the inspection task list.

[0111] It should be noted that the above explanation of the method embodiment is also applicable to the device of this embodiment, and will not be repeated here.

[0112] The real-time semantic segmentation device for dam defects for inspection unmanned vehicles of the embodiment of the present invention extracts features from the RGB-D defect images of the dam monitored by the inspection unmanned vehicle, generates multi-stage RGB images and depth images based on the ConvNeXt-T network module, and then inputs them into the pooling attention mechanism for feature enhancement; the feature map after feature enhancement in the same stage is sent to the feature fusion module, optimized based on the cross-modal attention mechanism, and fused to obtain a multi-level fusion feature map; the last level of fusion feature map is sent to the hollow pyramid pooling module to generate a multi-scale semantic map; the adjacent three-level fusion feature map is input into the upsampling recovery module, and multiple convolution and pooling operations are performed based on the different representations and contextual information of each fusion feature map to obtain a semantic segmentation map; and then the upsampling layer is input to generate the real-time semantic segmentation result of the dam defect. Thus, real-time semantic segmentation with dual feature enhancement is performed, which improves the reasoning speed and segmentation accuracy of real-time semantic segmentation. In order to realize the above embodiment, the present invention also proposes an electronic device, including:

[0113] at least one processor; and

[0114] a memory communicatively connected to the at least one processor; wherein,

[0115] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the aforementioned method.

[0116] In order to implement the above embodiment, the present invention further proposes a non-transitory computer-readable storage medium storing computer instructions, where the computer instructions are used to enable the computer to execute the above method.

[0117] In order to implement the above embodiments, the present invention further proposes a computer program product, comprising a computer program, wherein the computer program implements the above method when executed by a processor.

[0118] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0119] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0120] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present invention belong.

[0121] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0122] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0123] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0124] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0125] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A real-time semantic segmentation method for dam defects for unmanned vehicle inspection, characterized in that: The method processes two modalities, RGB image and depth image of dam defects, and includes the following steps: Obtain the RGB-D defect images that need to be processed when the unmanned vehicle is used to monitor the dam, and extract the feature maps of the two modalities of multi-scale RGB image and depth image in stages through the ConvNeXt-T network module. Through the feature enhancement module based on the pooling attention mechanism, the feature maps of the two modalities at different stages are enhanced to generate the RGB enhanced feature maps and depth enhanced feature maps at each stage. The RGB enhanced feature maps and depth enhanced feature maps of each stage are sent to the feature fusion module CM-FFM for fusion. The RGB enhanced feature maps and depth enhanced feature maps of each stage are optimized based on the cross-modal attention mechanism and fused into a single fused feature map of each stage to obtain multi-level fused feature maps F1, F2, F3, and F4. The last level fusion feature map F4 is input into the residual link-based dilated spatial pyramid pooling module to generate a multi-scale semantic map; The adjacent three-level fusion feature maps F1, F2, and F3 are input into the semantic-aware upsampling feature recovery module. Based on the different representations and context information of F1, F2, and F3, multiple convolution and pooling operations are performed on the multi-scale semantic map to obtain a semantic segmentation map with added complementary feature information. The semantic segmentation map is input into the upsampling layer to generate a real-time semantic segmentation result of the dam defect.

2. The method according to claim 1, characterized in that The method obtains the RGB-D defect images to be processed for the dam monitored by the unmanned inspection vehicle, and extracts the feature maps of the multi-scale RGB image and the depth image in stages respectively through the ConvNeXt-T network module, and enhances the feature maps of the two modes in different stages through the feature enhancement module based on the pooling attention mechanism, and generates the RGB enhanced feature maps and the depth enhanced feature maps in each stage, including: The inspection pictures of the unmanned vehicle monitoring the dam are cut into several 4×4 image blocks as the RGB-D defect images to be processed and used as the input of the ConvNeXt-T network module, where the image blocks have a certain amount of image overlap; Input the image block into a ConvNeXt-T network module, and after each depth-separable convolution operation in the ConvNeXt-T network module, generate a first RGB image and a first depth image respectively; Based on the adaptive average pooling operation in the feature enhancement module of the pooling attention mechanism, the first RGB image and the first depth image are reduced in dimension, and the maximum pooling operation is performed on the reduced-dimensional second RGB image and the second depth image to obtain a third RGB image and a third depth image; Perform a 1×1 convolution operation on the third RGB image and the third depth image, activate them using the sigmoid function, scale the attention weight value to between 0 and 1, and obtain the pooled attention vector of the RGB image and the depth image; The first RGB image is weightedly summed with the pooled attention vector of the RGB image to obtain the RGB enhanced feature map at each feature stage; The first depth image and the pooled attention vector of the depth image are weightedly summed to obtain the depth enhanced feature map of each stage.

3. The method according to claim 1, characterized in that The RGB enhanced feature map and the depth enhanced feature map of each stage are sent to the feature fusion module CM-FFM for fusion, and the RGB enhanced feature map and the depth enhanced feature map of each stage are optimized based on the cross-modal attention mechanism, and fused into a single fused feature map of each stage to obtain multi-level fused feature maps F1, F2, F3, and F4, including: When the RGB enhanced feature map and the depth enhanced feature map of each stage are sent to the feature fusion module CM-FFM for fusion, the RGB enhanced feature map and the depth enhanced feature map of each stage are input into the 1×1 convolution layer to generate the fourth RGB image and the fourth depth image Where, N = H × W; Through and Perform matrix multiplication to obtain the self-attention weight corresponding to the RGB image, and use the softmax function to limit the weight range to (0, 1). Subtract the matrix of self-attention weights from the identity matrix E to obtain the cross-modal attention vector corresponding to the RGB image. Similarly, according to the fourth depth image and Determine the cross-modal attention vector corresponding to the depth image The cross-modal attention vector Multiply and add to the RGB enhanced feature map to obtain the fifth RGB image F after complementary information enhancement r ' gb ; The cross-modal attention vector Multiply and add the depth enhancement feature map to obtain the five depth images F after complementary information enhancement d ' epth ; The fifth RGB image and the fifth depth image are spliced ​​to generate a single fused feature map of each stage to obtain multi-level fused feature maps F1, F2, F3, and F4, wherein the single fused feature map is restored to the original size of the RGB-D defect image through a residual connection.

4. The method according to claim 1, characterized in that The residual link-based atrous spatial pyramid pooling module includes five branches, wherein the first branch performs global average pooling on the input feature map; the second branch is a 1×1 convolution layer; the third, fourth, and fifth branches are atrous convolutions with residual link-introduced atrous rates of 6, 12, and 18, respectively.

5. The method according to claim 1, characterized in that The adjacent three-level fusion feature maps F1, F2, and F3 are input into the semantic-aware upsampling feature recovery module, and based on the different representations and context information of F1, F2, and F3, multiple convolution and pooling operations are performed on the multi-scale semantic map to obtain a semantic segmentation map with added complementary feature information, including: After the adjacent three-level fusion feature maps F1, F2, and F3 are input into the semantic-aware upsampling feature recovery module. 2-i It is processed by 3×3 convolution, normalization and ReLu activation function in the sampling feature recovery module to keep the number of channels and size of the feature map unchanged. 3-i Through the global pooling, 1×1 convolution and sigmoid activation function in the sampling feature recovery module, a first global context vector is obtained that represents different representations and context information of the fused feature map, where i∈(0,1,2); After processing F 2-i Multiply it with the global context vector to get the weighted low-level feature map In the high-level feature map F 3-i and Add together to get the first stage result; The first stage result is multiplied by the second global context vector to obtain the second stage intermediate feature. 3-i After upsampling, it is added to the intermediate features of the second stage to obtain a semantic segmentation map with added complementary feature information, where the second global context vector is composed of the feature image F i 'Get. When i = 0, F i ' represents the multi-scale semantic map obtained by the dilated spatial pyramid pooling module based on residual links; when i = 1, 2, F i ' represents the semantic segmentation map output by the i-th upsampling feature recovery module.

6. The method according to claim 1, characterized in that The real-time semantic segmentation method of dam defects used for unmanned vehicle inspection is migrated to the unmanned vehicle inspection system, and the unmanned vehicle inspection system includes: A data acquisition module, used for acquiring and storing images of the dam within the shooting range; An image segmentation module is used to receive the dam image taken in real time by the unmanned inspection vehicle, and identify and segment the defective area of ​​the dam based on the real-time semantic segmentation result of the dam defect analyzed by the real-time semantic segmentation method of the dam defect for the unmanned inspection vehicle; A data transmission module, used for transmitting the image information and mask information of the defect area; The inspection subsystem is used to receive the image information and the mask information, identify the defect location, generate an inspection task list based on the defect location, and send the inspection task list to the control subsystem; The control subsystem is used to control the unmanned inspection vehicle to photograph, identify and transmit the defective areas to be identified according to the inspection task list.

7. A real-time semantic segmentation device for dam defects for unmanned vehicle inspection, characterized in that: The device processes two modalities of RGB image and depth image of dam defects, including the following steps: The first feature enhancement module is used to obtain the RGB-D defect images that need to be processed when the unmanned vehicle monitors the dam, and extract the feature maps of the two modalities of multi-scale RGB image and depth image in stages through the ConvNeXt-T network module. The feature enhancement module based on the pooling attention mechanism is used to enhance the feature maps of the two modalities at different stages to generate RGB enhanced feature maps and depth enhanced feature maps at each stage; The fusion module is used to send the RGB enhanced feature map and the depth enhanced feature map of each stage to the feature fusion module CM-FFM for fusion, optimize the RGB enhanced feature map and the depth enhanced feature map of each stage based on the cross-modal attention mechanism, and fuse them into a single fused feature map of each stage to obtain multi-level fused feature maps F1, F2, F3, and F4; The first generation module is used to input the last level fusion feature map F4 into the residual link-based hollow space pyramid pooling module to generate a multi-scale semantic map; The second feature enhancement module is used to input the adjacent three-level fusion feature maps F1, F2, and F3 into the semantic-aware upsampling feature recovery module, and perform multiple convolution and pooling operations on the multi-scale semantic map based on the different representations and context information of F1, F2, and F3 to obtain a semantic segmentation map with added complementary feature information; The second generation module is used to input the semantic segmentation map into the upsampling layer to generate a real-time semantic segmentation result of the dam defect.

8. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-6.

Images