Unmanned aerial vehicle remote sensing RGB-D image semantic segmentation method, device and equipment

Abstract

The invention relates to an unmanned aerial vehicle remote sensing RGB-D image semantic segmentation method, device and equipment. According to the method, a multi-scale encoder for unmanned aerial vehicle remote sensing RGB-D image semantic segmentation is provided, the encoder comprises a plurality of ground object scene feature extraction modules, and visual texture information and geometric depth information provided by an RGB-D mode can be fully fused; global context information in a remote sensing large-scale space is constructed by using the long-sequence modeling capability of the state space model, and remote sensing multi-modal semantic segmentation under the high-performance UAV view angle is realized; a multi-modal feature guidance enhancement module is designed in front of an encoder, the quality of multi-modal feature representation is improved by using complementarity of information provided by RGB data and depth data, and the efficiency and performance of the model in a feature encoding learning stage are improved. The method is specially suitable for semantic segmentation research of the unmanned aerial vehicle remote sensing RGB-D image, and continuous development of complex environment ground feature analysis research in the field of unmanned aerial vehicle remote sensing is promoted.

Description

Technical Field

[0001] The present application relates to the technical field of image semantic segmentation, and in particular to a method, device and equipment for semantic segmentation of RGB-D images sensed by unmanned aerial vehicles (UAVs). Background Art

[0002] In recent years, with the popularity of unmanned aerial vehicles (UAVs), the popularity of UAVs in remote sensing image acquisition has increased significantly. Compared with satellite remote sensing platforms, UAVs have flexible and controllable flight path planning for remote sensing images, and lower flight altitudes provide higher image spatial resolution while reducing development and operating costs. The high-resolution remote sensing images collected by UAVs are key resources in many research fields, and can promote in-depth understanding and analysis of research applications in areas such as urban planning, environmental change monitoring, agricultural management, and disaster response.

[0003] In the field of remote sensing, the premise for various research and analysis of high-resolution remote sensing images collected from the perspective of UAVs is mostly to analyze the categories of objects in the spatial scene, that is, semantic segmentation. Semantic segmentation refers to dividing each pixel in the image into several non-overlapping regions with the same semantics according to the category of objects and labeling the pixels in these regions with specific category labels. Image semantic segmentation in the field of remote sensing emphasizes capturing contextual semantic information in large-scale space, thereby simplifying the distribution of complex objects in space. Remote sensing image semantic segmentation has become the basis of computer vision research in many remote sensing fields, such as road extraction, land use mapping, and urban feature perception.

[0004] Compared with satellite images, UAV remote sensing images can provide higher spatial resolution in semantic segmentation research. At the same time, UAV remote sensing images can provide diversified and fine-grained semantic feature representations from different heights and angles, thereby achieving a comprehensive understanding of complex remote sensing spatial scenes. However, UAV remote sensing images have a high spatial resolution, and there are rich contextual connections between multiple objects in a large-scale space. Unlike natural images taken horizontally on the road, UAV remote sensing images are collected from the air downward, so the distribution of ground features in UAV remote sensing images can exist in any direction. How to extract omnidirectional spatial features from remote sensing large-scale space and perform global context information modeling is crucial for remote sensing dense prediction tasks.

[0005] Another major advantage of drones is the flexibility of their sensor mounting. With the popularity of 3D sensors in recent years, UAV data collection tasks are not limited to obtaining RGB images, but to obtaining multimodal data such as point clouds and depth information. Research on RGB-D data in advanced scene semantic understanding has attracted attention in the past few years. Depth data can provide the 3D geometry of objects in the scene, thereby providing supplementary signals in addition to spatial texture visual information, thereby improving the accuracy of semantic segmentation. In the remote sensing spatial distribution from the perspective of UAV, there are many objects with similar texture visual appearance but large differences in geometric structure (such as grass and tall vegetation). RGB-D data that combines visual texture information and deep geometric information can provide richer ground object features, thereby achieving more accurate semantic understanding of complex remote sensing scenes. At present, the semantic analysis of RGB-D data is usually applied to the research of indoor or outdoor road ground level perspectives such as vehicle autonomous driving, robot navigation and SLAM. However, the semantic segmentation method of remote sensing RGB-D images based on the aerial perspective of UAV is relatively lacking, and is almost in a new research field. Summary of the invention

[0006] Based on this, it is necessary to provide a method, device and equipment for semantic segmentation of UAV remote sensing RGB-D images to address the above technical problems.

[0007] A method for semantic segmentation of UAV remote sensing RGB-D images, the method comprising: Obtain a set of RGB images taken by drones and depth maps of corresponding spatial dimensions, and perform preprocessing.

[0008] The preprocessed RGB modal input features and depth modal input features are used to calibrate the features of different categories of ground objects using a multimodal feature guidance enhancement module to obtain calibrated RGB modal features and depth modal features; the multimodal feature guidance enhancement module is used to use the high confidence activation of one modal feature to guide the correction of the features of another modality, thereby enhancing the similarity of features within the same category of objects and the difference between features of different categories of objects.

[0009] The calibrated RGB modal features and depth modal features are encoded with a multi-scale encoder to obtain multi-modal visual features of different scales; the multi-scale encoder is used to extract the visual texture features and deep geometric features of objects in the remote sensing scene from the perspective of drone aerial photography from different scales using multiple layers of ground object scene feature extraction modules; the ground object scene feature extraction module is used to realize the information interaction between RGB modality and depth modality through Transformer, and uses large kernel convolution operation to capture spatial geometric clues from the depth modality to refine the edge detail information of the ground object semantics to obtain multi-modal fusion features and geometric depth modal features. At the same time, the context information in the remote sensing scene is modeled from multiple directions through the state space model, and the modeling information in all directions is fused to obtain the global features of the remote sensing space. Then, the multi-modal fusion features, geometric depth modal features and the global features of the remote sensing space are connected in series and linearly projected to obtain visual features of one scale.

[0010] After fusing multimodal visual features of different scales, a lightweight decoder is used to predict the semantic labels of objects to obtain semantic segmentation results.

[0011] A semantic segmentation device for UAV remote sensing RGB-D images, the device comprising: The RGB-D image acquisition and preprocessing module is used to acquire a set of RGB images taken by drones and depth maps of corresponding spatial dimensions, and perform preprocessing.

[0012] The RGB modality and depth modality feature calibration module is used to calibrate the features of different categories of ground objects using the preprocessed RGB modality input features and depth modality input features using the multimodal feature guidance enhancement module to obtain calibrated RGB modality features and depth modality features; the multimodal feature guidance enhancement module is used to use the high confidence activation of one modality feature to guide the correction of the features of another modality, thereby enhancing the similarity of features within the same category of objects and the difference between features of different categories of objects.

[0013] The multimodal visual feature extraction module of different scales is used to use a multi-scale encoder to perform multi-scale feature encoding on the calibrated RGB modal features and depth modal features to obtain multimodal visual features of different scales; the multi-scale encoder is used to extract the visual texture features and deep geometric features of the objects in the remote sensing scene from the perspective of drone aerial photography from different scales using multiple layers of ground object scene feature extraction modules; the ground object scene feature extraction module is used to realize the information interaction between RGB modality and depth modality through Transformer, and uses large kernel convolution operation to capture spatial geometric clues from the depth modality to refine the edge detail information of the ground object semantics, obtain multimodal fusion features and geometric depth modal features, and at the same time, the context information in the remote sensing scene is modeled from multiple directions through the state space model, and the modeling information in all directions is fused to obtain the global features of the remote sensing space, and then the multimodal fusion features, geometric depth modal features and the global features of the remote sensing space are connected in series and linearly projected to obtain visual features of one scale.

[0014] The semantic segmentation module is used to fuse multimodal visual features of different scales and use a lightweight decoder to predict the semantic labels of objects to obtain semantic segmentation results.

[0015] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0016] The above-mentioned UAV remote sensing RGB-D image semantic segmentation method, device and equipment proposes a multi-scale encoder for UAV remote sensing RGB-D image semantic segmentation. The encoder includes multiple ground object scene feature extraction modules, which can fully integrate the visual texture information and geometric depth information provided by the RGB-D modality, and use the long-sequence modeling ability of the state space model to construct the global context information in the remote sensing large-scale space, so as to achieve high-performance remote sensing multi-modal semantic segmentation from the UAV perspective; a multi-modal feature guidance enhancement module is designed before the encoder, and the complementarity of the information provided by RGB data and depth data is used to improve the quality of multi-modal feature representation, thereby improving the efficiency and performance of the model in the feature encoding learning stage. This method is specifically suitable for the semantic segmentation research of UAV remote sensing RGB-D images, and promotes the continuous development of complex environment ground object analysis research in the field of UAV remote sensing. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic diagram of a flow chart of a method for semantic segmentation of a UAV remote sensing RGB-D image in one embodiment; Figure 2 This is a general architecture diagram of a semantic segmentation model for UAV remote sensing RGB-D images in another embodiment; Figure 3is an architecture diagram of a multimodal feature guidance enhancement module in another embodiment; Figure 4 A schematic diagram of a ground object scene feature extraction module in another embodiment; Figure 5 A schematic diagram of a process for implementing global spatial feature modeling for remote sensing images in another embodiment; Figure 6 is a schematic diagram of semantic categories of outdoor scene objects in another embodiment; Figure 7 It is a structural block diagram of a semantic segmentation device for remote sensing RGB-D images of a UAV in one embodiment; Figure 8 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0019] The present application provides a method for semantic segmentation of unmanned aerial vehicle (UAV) remote sensing RGB-D images. The method follows the mainstream encoder-decoder architecture for the semantic segmentation task of UAV remote sensing images. In the encoder part, multiple ground object scene feature extraction modules are used to extract visual texture features and deep geometric features of ground objects in remote sensing scenes from the perspective of UAV aerial photography from different scales, and the extracted multi-scale features are converted into predicted ground object semantic labels through a lightweight decoder.

[0020] In one embodiment, Figure 1 As shown, a method for semantic segmentation of UAV remote sensing RGB-D images is provided, and the method comprises the following steps: Step 100: Obtain a set of RGB images and depth maps of corresponding spatial dimensions taken by a drone, and perform preprocessing.

[0021] Specifically, compared with satellite remote sensing images, aerial remote sensing images taken by unmanned aerial vehicles (UAVs) have higher resolution and can provide more detailed information. At the same time, combined with depth information, the performance of semantic segmentation of remote sensing images can be improved.

[0022] Step 102: The preprocessed RGB modal input features and depth modal input features are calibrated using a multimodal feature guidance enhancement module to obtain calibrated RGB modal features and depth modal features; the multimodal feature guidance enhancement module is used to use the high confidence activation of one modal feature to guide the correction of another modal feature, thereby enhancing the similarity of features within the same category of objects and the difference between features of different categories of objects.

[0023] Specifically, given a set of UAV aerial RGB images and depth maps of corresponding spatial dimensions, a multimodal feature-guided enhancement module is first used to utilize the complementary information of the two modalities to calibrate the features of objects of different semantic categories in the scene space, suppress abnormal feature responses in edge areas, and enhance feature responses within the same semantic category.

[0024] Step 104: The calibrated RGB modal features and depth modal features are encoded with a multi-scale encoder to obtain multi-modal visual features of different scales; the multi-scale encoder is used to extract visual texture features and deep geometric features of objects in the remote sensing scene from the perspective of drone aerial photography from different scales using multiple layers of ground object scene feature extraction modules; the ground object scene feature extraction module is used to realize information interaction between RGB modality and depth modality through Transformer, and uses large kernel convolution operation to capture spatial geometric clues from the depth modality to refine the edge detail information of the ground object semantics, to obtain multi-modal fusion features and geometric depth modal features, and at the same time, the context information in the remote sensing scene is modeled from multiple directions through the state space model, and the modeling information in all directions is fused to obtain the global features of the remote sensing space, and then the multi-modal fusion features, geometric depth modal features and the global features of the remote sensing space are connected in series and linearly projected to obtain visual features of one scale.

[0025] Specifically, the calibrated RGB modal features and depth modal features are sent to the multi-scale encoder. The multi-scale encoder architecture consists of four layers of ground object scene feature extraction modules. Each layer of ground object scene feature extraction module performs multi-scale feature encoding on RGB-D features at {1 / 4, 1 / 8, 1 / 16, 1 / 32} of the original image size.

[0026] Step 106: After fusing the multimodal visual features of different scales, a lightweight decoder is used to predict the semantic labels of the objects to obtain a semantic segmentation result.

[0027] Specifically, the encoded multimodal visual features of different scales are fused and sent to a lightweight encoder to predict the semantic labels of objects.

[0028] The semantic segmentation model used in this application method is named the UAV remote sensing RGB-D image semantic segmentation model (abbreviated as: SkyMamba model), which consists of a multimodal feature guidance enhancement module (FGE), a multi-scale encoder, and a lightweight decoder. The overall architecture of the UAV remote sensing RGB-D image semantic segmentation model is as follows: Figure 2 As shown. Figure 2 It can be seen that: the RGB modal data and the depth modal data are input into the lightweight multimodal feature guidance enhancement module to calibrate the features of different categories of ground objects; then, the calibrated RGB-D modal data is input into the four-stage ground object scene feature extraction module for encoding, and finally the encoded features are passed to the lightweight decoder for final semantic prediction.

[0029] In the above-mentioned semantic segmentation method of UAV remote sensing RGB-D images, this method proposes a multi-scale encoder for semantic segmentation of UAV remote sensing RGB-D images. The encoder includes multiple ground object scene feature extraction modules, which can fully integrate the visual texture information and geometric depth information provided by the RGB-D modality, and use the long-sequence modeling ability of the state space model to construct the global context information in the remote sensing large-scale space, so as to achieve high-performance remote sensing multi-modal semantic segmentation from the UAV perspective; a multi-modal feature guidance enhancement module is designed before the encoder, which uses the complementarity of information provided by RGB data and depth data to improve the quality of multi-modal feature representation, and improves the efficiency and performance of the model in the feature encoding learning stage. This method is specifically suitable for the semantic segmentation research of UAV remote sensing RGB-D images, and promotes the continuous development of complex environment ground object analysis research in the field of UAV remote sensing.

[0030] In one embodiment, the multimodal feature guidance enhancement module includes: a homogeneous feature aggregation operation and a heterogeneous feature separation operation; step 102 specifically includes the following steps: Step 200: A homogeneous feature aggregation operation is performed on the RGB modality input features and the depth modality input features obtained by preprocessing to obtain features after RGB modality aggregation and features after depth modality aggregation; the homogeneous feature aggregation operation is used to calibrate the features using the different representation capabilities of the RGB image modality and the depth map modality, and to complementarily aggregate the multimodal features of objects of the same semantic category to a certain position in space.

[0031] Specifically, the architecture of the multimodal feature-guided enhancement module is as follows: Figure 3 As shown, the multimodal feature guidance enhancement module includes: homogeneous feature aggregation operation and heterogeneous feature separation operation; the multimodal feature guidance enhancement module uses the high confidence activation of different modal features to guide the correction of ground feature of different semantic categories, suppress the abnormal feature response of the edge area of ​​the category object, and strengthen the homogeneous feature response within the same category object.

[0032] Homogeneous feature aggregation operation (HFA): The object feature information presented by RGB modal features and depth modal features is complementary, and they can provide visual texture information and geometric structure information of objects in remote sensing space respectively. In order to aggregate homogeneous features of objects belonging to the same semantic category in outdoor scenes, it is necessary to calibrate the features using the representation capabilities of different modalities, and complementarily aggregate the multimodal features of objects of the same semantic category to a certain position in space.

[0033] Step 202: A heterogeneous feature separation operation is performed on the RGB modal input features and the depth modal input features obtained by preprocessing to obtain RGB modal features and depth modal features after modality separation; the heterogeneous feature separation operation is used to use high-confidence activation of one modality feature to filter abnormal features at the edge of another modality object, thereby achieving separation of heterogeneous features.

[0034] Specifically, heterogeneous feature separation operation (HFS): There are many objects with similar visual textures in outdoor scenes. For example, tall vegetation and grass can easily produce blurred abnormal feature responses at the edge of the object due to their texture similarity during the feature aggregation operation. However, tall vegetation and grass have large differences in spatial geometric structure, so we consider using the differences in features of different modal data to separate the heterogeneous feature responses at the edge of category objects, making the feature differences between different classes of objects more significant. In practice, the high confidence activation of one modal feature is used to filter the abnormal features at the edge of another modal object, so that the edge feature contrast of objects of different semantic categories is stronger, and the separation of heterogeneous features is achieved.

[0035] Step 204: The features after RGB modality aggregation and the RGB modality features after modality separation are merged to obtain calibrated RGB modality features.

[0036] Step 206: Fuse the features after deep modality aggregation and the deep modality features after modality separation to obtain calibrated deep modality features.

[0037] In one embodiment, step 200 includes: concatenating the RGB modality input features and the depth modality input features obtained by preprocessing and performing convolution processing to generate an RGB modality space gate and a depth modality space gate for controlling the feature information flow; fusing and mapping the RGB modality space gate and the depth modality space gate respectively to obtain homogeneous feature aggregation weights of the RGB modality and the depth modality; wherein the sum of the values ​​of the homogeneous feature aggregation weights of the RGB modality and the depth modality at the same position is 1; performing homogeneous aggregation on the RGB modality input features through the homogeneous feature aggregation weights of the RGB modality to obtain RGB modality features after homogeneous aggregation; performing homogeneous aggregation on the depth modality input features through the homogeneous feature aggregation weights of the depth modality to obtain deep modality features after homogeneous aggregation; interactively fusing the homogeneously aggregated RGB modality features and the deep modality features to obtain fused features; using the fused features to perform feature enhancement on the homogeneously aggregated RGB modality features and the deep modality features respectively to obtain features after RGB modality aggregation and features after depth modality aggregation.

[0038] Specifically, in the process of homogeneous feature aggregation: first, the RGB modality input features and the depth modality input features are concatenated and convolved to generate the corresponding spatial gates and To control the information flow of RGB and depth modal feature maps, and then fuse the modal space gates that control the feature information flow to obtain the homogeneous feature aggregation weights and : (1) (2) in, and Represent the homogeneous feature aggregation weights of RGB modality and depth modality respectively, and , and Represents the spatial gates of RGB modality features and depth modality features respectively. Homogeneous feature aggregation weight and In the same location The sum of the values ​​on is 1: (3) Homogeneous feature aggregation weights can effectively aggregate the features of objects of the same semantic category in the RGB modality and the depth modality. and After homogeneous aggregation, the RGB modal features after homogeneous aggregation are obtained. and deep modality features : (4) (5) Then, we interact the RGB features and depth features obtained after homogeneity aggregation to obtain the fusion features : (6) Finally, using the fusion features The two modal features are further enhanced to obtain the features after RGB modality aggregation And the features after deep modality aggregation , to strengthen the similarity of features of objects in the same category: (7) (8) In one embodiment, step 202 includes: fusing the pre-processed RGB modal input features and the deep modal input features and performing an MLP operation to obtain heterogeneous feature separation weights; multiplying the heterogeneous feature separation weights with the homogeneously aggregated RGB modal features and the deep modal features to filter out abnormal features at the edges of different semantic classes, and obtaining the modality-separated RGB modal features and the deep modal features.

[0039] Specifically, in the heterogeneous feature separation (HFS) process, the RGB modality input features and the depth modality input features are first fused and operated through the MLP to obtain the heterogeneous feature separation weights : , (9) in, and represents the input RGB modality input features and depth modality input features, [·,·] represents the cross-modality feature fusion operation, represents the global average pooling operation, represents the MLP network operation, Represents the sigmoid function that scales the heterogeneous feature separation weights to (0,1).

[0040] Then, the RGB modal features after homogeneity aggregation are and deep modality features and Multiply to filter the abnormal features of the edges of different semantic classes to obtain the RGB modal features after modal separation and deep modality features : (10) (11) Finally, the features after modal separation and aggregation are combined to obtain the calibrated and enhanced multimodal features: (12) (13) in, and They represent the RGB modal features and the depth modal features after feature-guided enhancement processing. The idea behind this operation is to use the complementarity of spatial texture information and spatial geometric information of objects in outdoor scenes, and use the feature complementary information as an offset to refine and correct the class semantic information under different modalities, so that the features of objects under the same semantic category are more similar, and the features of objects under different semantic categories are more separated, thereby forming more robust class semantic information for the subsequent object scene feature extraction module to better perform feature encoding learning.

[0041] In one embodiment, the multi-scale encoder includes four layers of ground object scene feature extraction modules of different scales and three downsampling layers; wherein the downsampling layer is arranged between two layers of ground object scene feature extraction modules of different scales; step 104 specifically includes the following steps: Step 300: inputting the calibrated RGB modal features and depth modal features into the first-layer ground feature scene feature extraction module to obtain first-scale visual features; Step 302: down-sample the first-scale visual features and input them into a second-layer ground object scene feature extraction module to obtain second-scale visual features.

[0042] Step 304: down-sample the second-scale visual features and input them into the third-layer ground object scene feature extraction module to obtain third-scale visual features.

[0043] Step 306: down-sample the third-scale visual features and input them into the fourth-layer ground object scene feature extraction module to obtain fourth-scale visual features.

[0044] Specifically, the multi-scale encoder includes four layers of ground object scene feature extraction modules of different scales, which are used to encode the multi-scale RGB-D features of the ground objects in the remote sensing scene. A convolution with a 3×3 kernel and a step size of 2 is set between each layer of ground object scene feature extraction modules to downsample the RGB-D cross-modal features.

[0045] In one embodiment, if Figure 4 As shown, the ground object scene feature extraction module includes: a multimodal Transformer module and a remote sensing space global attention module; the first scale visual feature includes the first scale RGB modality and the depth modality visual feature; step 300 specifically includes the following steps: Step 400: Input the calibrated RGB modality features and depth modality features into the multimodal Transformer module to obtain multimodal fusion features and geometric depth modality features; wherein the multimodal fusion features and geometric depth modality features are: (14) (15) (16) (17) (18) in, represents multimodal fusion features, represents a bilinear upsampling operation, represents the softmax operation, Indicates the dimension of the current feature, and Respectively represent the Value vector and Key vector calculated from the RGB features, represents the Query vector, represents the geometric depth modal features, represents a large kernel depthwise convolution operation, represents the Hadamard product operation, and They represent the RGB modality features and depth modality features of the input multimodal Transformer module respectively, [·,·] represents the cross-modality feature fusion operation, represents an adaptive average pooling operation of fixed size k×k, Represents an MLP network operation.

[0046] Specifically, the Multimodal Transformer module (MMT) uses Transformer to fuse RGB modality and depth modality features to establish cross-modal feature interaction relationships, enhance 3D perception in remote sensing space, and further capture semantic category information; at the same time, a large kernel convolution operation is used to further capture spatial geometric clues from the depth modality to refine the edge detail information of the semantics of ground objects.

[0047] Step 402: Input the calibrated RGB modal features into the remote sensing space global attention module, decompose the remote sensing space features and reconstruct the feature sequences in eight directions, use the state space model to capture the context information in each feature sequence, and finally fuse the modeling features in each direction to obtain the remote sensing space global features.

[0048] Specifically, the remote sensing spatial global attention module (RSSGA) decomposes the large-scale features of the remote sensing space into local features and reconstructs the feature sequence from eight directions. It models the contextual information in the remote sensing scene from eight directions through the state space model, and fuses the modeling information in eight directions to obtain the global features of the entire remote sensing space, thereby enhancing the correlation between the semantic information of the objects in the remote sensing scene.

[0049] The RGB modality and depth modality in remote sensing scenes provide visual texture information and spatial geometric information of objects respectively. In order to interact the information provided by the two modalities and enhance the semantic category features, we proposed the MMT module to interact with multimodal features. The Key vector and the Value vector are calculated; the Value vector is shown in formula (17), and the Key vector is shown in formula (18).

[0050] In the process of calculating the query vector, the MMT module concatenates the input RGB features and depth features and performs a pooling operation of a fixed size to obtain a query vector that integrates multimodal information; the query vector is shown in formula (16).

[0051] In getting the Value vector , Key vector and Query vector After that, self-attention operation is performed and up-sampled to the corresponding dimension to obtain the multimodal fusion feature; the multimodal fusion feature is shown in formula (14).

[0052] The depth modality in the remote sensing space can provide more object edge detail information. In order to retain more spatial structure information, we use a large kernel to convolve the depth feature in the MMT module, and use the Hadamard product and query vector to perform operations to embed more information, thereby strengthening the object geometric structure information in the remote sensing space and obtaining the geometric depth modality feature; the geometric depth modality feature is shown in formula (15). It is the deep feature after being processed by large kernel deep convolution and multiplied by the query vector.

[0053] Step 404: The multimodal fusion features, the geometric depth modal features, and the global features of the remote sensing space are connected in series and linearly projected to obtain a scaled RGB modality and depth modality visual feature.

[0054] Specifically, the cross-modal fusion features and geometric depth features obtained by the MMT module are concatenated and linearly projected with the remote sensing spatial global features obtained by the remote sensing spatial global attention module (RSSGA) to enhance the feature representation in the scene and thus achieve feature updating.

[0055] In one embodiment, the remote sensing space global attention module includes: 1 feature decomposition and reconstruction module, 8 state space models and 1 feature fusion module; step 402 includes: inputting the calibrated RGB modal features into the feature decomposition and reconstruction module, decomposing the calibrated RGB modal features into 12 local image blocks for global feature separation, reconstructing the feature sequences of the separated local blocks along the horizontal, vertical, diagonal and anti-diagonal directions, and obtaining eight groups of feature sequences reconstructed in different directions; using the state space model to capture the context information in different directions of the eight groups of feature sequences reconstructed in different directions, and obtaining modeling information in eight directions; using the feature fusion module to fuse the modeling information in eight directions to obtain the remote sensing space global features.

[0056] Specifically, the distribution of remote sensing spatial features from the UAV perspective is different from that from the ground road perspective. Objects have strong correlation and rich contextual information, and the object feature distribution in the remote sensing image can be in any direction. In order to capture multi-directional global information in the remote sensing space, this application proposes a remote sensing spatial global attention module (RSSGA) module to decompose remote sensing spatial features and reconstruct feature sequences in eight directions, use a state space model to capture the contextual information in each feature sequence, and finally merge the modeling features in each direction to obtain the global features in the remote sensing space.

[0057] We decompose the remote sensing image input to the RSSGA module into 12 local image blocks for global spatial feature separation, and then reconstruct the feature sequences of the decomposed local blocks along the horizontal, vertical, diagonal and anti-diagonal directions. The feature sequences reconstructed in different directions can help the encoder extract richer global features of the remote sensing space.

[0058] The schematic diagram of the process of realizing global spatial feature modeling of remote sensing images is as follows: Figure 5 As shown in the figure, the high-resolution remote sensing image is feature separated and sequence reconstructed in specific directions, and the context information is modeled in each direction through the state space model. Finally, the context information in all directions is integrated to realize the remote sensing global spatial feature modeling.

[0059] The feature sequences reconstructed in eight different directions are modeled along the new dimension using the state space model for global feature modeling. The state space model can encapsulate the state information in the dynamic system and can therefore effectively model the feature context information. First, in order to realize the dynamic perception reasoning of context information in the remote sensing space, the RSSGA module calculates the local feature map of each directional feature sequence according to the i-th local feature map. Adaptively obtain the corresponding state transfer matrix , input state matrix and the output state matrix , where the matrix and matrix By parameters Zero-order hold technique for discretization: (19) (20) in, and Represent the discretized state transfer matrix and input state matrix respectively, represents the discretization parameter, Represents the identity matrix.

[0060] Secondly, according to the adaptive parameters The dynamically changing state matrix can selectively focus on perceiving the contextual information in different local feature maps. With the input state matrix Multiply to get the state representation of the current feature fragment : (twenty one) The current local feature state represents the retention state of the previous local feature Add to realize feature context information modeling, and retain the state of the previous local features It is to selectively save the important spatial features of the previous fragment so as to give it to the current local feature state to obtain a new feature state : (twenty two) The retention state of the previous local features It is composed of the previous local features The corresponding state transfer matrix Multiplying together gives: (twenty three) Finally, the new local feature state With the output state matrix Multiply to get new local spatial features : (twenty four) The new local spatial feature integrates the spatial distribution information of the previous feature, realizes the modeling of context information in a specific direction, and and the corresponding state transfer matrix Multiply to get the retained state To provide the previous feature information for the input features of the next stage. After reconstructing multiple feature sequences in different directions, all local feature sequences are merged and recombined to realize the global feature modeling of remote sensing space: (25) in, Represents the global characteristics of remote sensing space, Represents a multi-directional feature sequence recombination operation, Represents a unidirectional reconstructed feature sequence.

[0061] In one embodiment, the lightweight decoder in step 106 is a lightweight Hamburger decoder in a SegNext network model.

[0062] Specifically, this application uses the lightweight Hamburger decoder in SegNext to aggregate and decode multi-scale RGB-D features. Due to the powerful feature extraction capability of the multi-scale encoder, selecting a lightweight decoder can achieve excellent performance.

[0063] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0064] In a verification example, UAV aerial remote sensing images have a higher resolution than satellite remote sensing images and can provide more detailed information. At the same time, combined with depth information, the performance of remote sensing image semantic segmentation can be improved. In this example, a new UAV aerial remote sensing RGB-D image semantic segmentation benchmark dataset CUS-Depth V1 is proposed. This dataset is obtained by expanding the latest comprehensive three-dimensional semantic segmentation dataset CUS3D

[61] . The CUS-DepthV1 dataset contains a total of 4336 UAV aerial RGB images and corresponding depth maps from Changsha, China at orthogonal and oblique 45° viewing angles. The image resolution is 6144×4096, of which 2601 images are used for training, 867 images are used for verification, and 868 images are used for testing. The CUS-Depth V1 dataset covers a variety of outdoor scenes such as cities, suburbs, and rural areas, covering an area of ​​2.85 square kilometers. This example provides 10 common outdoor scene semantic categories such as buildings, roads, tall vegetation, grasslands, water bodies, vehicles, farmlands, construction sites, stadiums, and bare surfaces. Figure 6 shown. Figure 6 Middle left: Schematic diagram of the area covered by the original RGB image of the CUS-Depth V1 dataset; right: Schematic diagram of the distribution range of semantic labels corresponding to the RGB image of the CUS-Depth V1 dataset.

[0065] In order to more effectively promote the development of community research on semantic segmentation of UAV aerial remote sensing RGB-D images, this application also provides a more challenging CUS-Depth V2 dataset. The image resolution, number and scene coverage of this dataset are the same as the basic version of CUS-Depth V1. The difference is that compared with the basic version of CUS-Depth V1, the dataset provides more semantic categories, including 18 semantic categories of land objects, including high-rise buildings, villas, vehicles, pedestrians, construction sites, non-motor vehicles, grasslands, tall vegetation, roads, sidewalks, farmlands, mud, water bodies, athletic fields, basketball courts, children's playgrounds, power supply equipment and sundries.

[0066] (1) Benchmark Method and Experimental Details In order to evaluate the performance of the SkyMamba model proposed in this application in the semantic segmentation of UAV aerial remote sensing RGB-D images, this embodiment selected six representative RGB-D image semantic segmentation methods, including ACNet, SA-Gate, TokenFusion, CMX, CMNext and DFormer, as benchmarks and conducted comparative experiments on the CUS-Depth V1 and CUS-Dpeth V2 datasets.

[0067] Data preprocessing: To avoid performance test errors caused by complex data enhancement techniques, we only use the two simplest data enhancement techniques, horizontal flipping and random scaling, in data preprocessing. At the same time, for the convenience of research, we process the input remote sensing images into non-overlapping 512×512 pixel blocks for input model training. The benchmark methods tested on the same dataset all use the same data preprocessing strategy to ensure experimental consistency.

[0068] Experimental setup: We used the PyTorch framework to deploy the SkyMamba model and other baseline comparison experiments on a single NVIDIA Tesla A800 GPU, and adjusted the batch size accordingly according to the actual situation of different models to maximize the computing power utilization. We combined the cross entropy loss and the Dice coefficient as the loss function of the SkyMamba model to optimize the model performance, and used the AdamW optimizer with an initial learning rate of 0.001 and a weight decay of 0.001 to learn and update the model parameters. After 150 epochs of training, the model converged. Table 1 provides detailed experimental parameter settings for the SkyMamba model and other benchmark comparison methods on the CUS-Depth V1 and CUS-DepthV2 datasets.

[0069] Table 1. Hyperparameter settings for the comparative experiments of each baseline method on the CUS-Depth V1 and CUS-Depth V2 datasets

[0070] Evaluation indicators: In order to evaluate the performance of the semantic segmentation model, we choose to use the intersection over union (IoU) of each semantic category, the overall mean intersection over union (mIoU), and the overall mean accuracy (mAcc) as evaluation indicators. The IoU and mIoU indicators are used to measure the overlap between the segmentation results and the true results, thereby providing a spatial accuracy evaluation of the model prediction. The mAcc indicator is used to measure the number of correctly classified pixels in the segmentation results, which can reflect the overall performance of the model in pixel classification.

[0071] (2) Comparative experimental results and discussion In order to verify the advancedness of the SkyMamba model proposed in this application in the semantic segmentation of UAV aerial remote sensing RGB-D images, the final SkyMamba model was compared with the RGB-D semantic segmentation baseline method on the CUS-Depth V1 and CUS-DepthV2 datasets. Table 2 shows the experimental test results of different semantic segmentation methods on the CUS-Depth V1 dataset. The SkyMamba model achieved the highest mIoU (78.46%) on the CUS-Depth V1 dataset test, which is 4.41% higher than the latest RGB-D semantic segmentation method DFormer (74.05%). At the same time, the SkyMamba model also achieved the second highest mAc (87.95%) on the CUS-Depth V1 dataset, second only to SA-Gate (88.68%). In terms of the IoU indicator of each semantic category, the SkyMamba model achieved the highest performance in most categories. Its IoU in the Lake category was 95.87%, second only to CMX (96.41%). However, it achieved an IoU of 81.45% in a small category with less distribution, such as cars, which is much higher than other methods.

[0072] Table 2. Comparison results of SkyMamba and comparison methods on the CUS-Depth V1 dataset

[0073] Note: The comparison results include average category accuracy (mAcc, %), average intersection over union (mIoU, %) and intersection over union of each category (%).

[0074] In actual situations, there are many fine and complexly distributed objects in remote sensing scenes. The experimental results on the CUS-Depth V1 dataset show that the SkyMamba model has a stronger ability to capture details in large-scale space compared with other methods. In order to further verify the semantic segmentation ability of the SkyMamba model in large-scale space, a comparative experiment was conducted on the more challenging CUS-Depth V2 dataset. Table 3 shows the experimental test results of different semantic segmentation methods on the CUS-Depth V2 dataset. From the results, it can be seen that even in the face of complex remote sensing scenes with 18 semantic categories, SkyMamba can still achieve the highest mIoU (63.82%), which is 3.25% higher than the second highest method CMNext

[62] (60.57%), and 4.76% higher than the latest RGB-D semantic segmentation method DFormer

[38] (59.06%). At the same time, SkyMamba achieves the second highest mAc (72%). It is worth noting that SkyMamba demonstrates excellent segmentation performance on sparsely distributed fine features such as cars, pedestrian lanes, athletic fields, children's playgrounds, and debris, and shows a more balanced semantic segmentation effect on complex features in large-scale space, effectively overcoming the problem of unbalanced distribution of semantic categories in the dataset to a certain extent.

[0075] Table 3. Comparison results of SkyMamba and comparison methods on the CUS-Depth V2 dataset

[0076] Note: The comparison results include the average category accuracy (mAcc, %), average intersection over union (mIoU, %), and intersection over union (IoU) of each category (%).

[0077] In summary, the SkyMamba model can achieve state-of-the-art performance in the field of semantic segmentation of UAV aerial RGB-D images. The features of objects in remote sensing scenes are distributed in a variety of directions and cover a large spatial scale. The design of the SkyMamba model can capture the global context information and the substantial spatial semantic category information of objects in large-scale space from all directions while combining the visual texture information and geometric depth information of objects, thereby achieving accurate semantic segmentation of remote sensing images.

[0078] (3) Ablation experiments and discussion The SkyMamba model is composed of multiple sets of ground scene feature extraction modules, each of which is composed of a multimodal transformer module (MMT) and a remote sensing spatial global attention module (RSSGA). At the same time, a lightweight cross-modal feature guidance enhancement module (FGE) is designed before feature encoding to perform feature refinement. In order to further study the effectiveness of each component in the SkyMamba model on semantic segmentation performance, we performed an ablation study on the CUS-Depth V1 dataset with the same hyperparameter settings.

[0079] Overall ablation study of components. The three components of the SkyMamba model, namely the FGE module, the MMT module and the RSSGA model, were taken out and experiments were conducted on the CUS-Depth V1 dataset. The results are shown in Table 4. From the ablation study results, it can be seen that all components are essential for the SkyMamba model. The FGE module can bring a 2.28% improvement in the mIoU performance of the SkyMamba model, while basically not bringing additional parameters and computational complexity. The MMT module can bring a 3.87% improvement in the semantic segmentation performance indicator mIoU, effectively integrating multimodal features to achieve efficient semantic segmentation. The RSSGA module can provide the most effective performance improvement for the SkyMamba model, with an improvement of 5.99% in the mIoU indicator, which is due to its global spatial modeling capability that can effectively capture and provide global context information of large-scale remote sensing spaces.

[0080] Table 4. Ablation test results of each component of SkyMamba model

[0081] In the FGE module, the abnormal feature responses in the edge areas of objects in the remote sensing scene are suppressed through homogeneous feature aggregation (HFA) and heterogeneous feature separation (HFS), and the homogeneous feature responses within the same category of objects are strengthened, thereby strengthening the essential semantic category information of the objects in space. In order to further study the effectiveness of the HFA and HFS operations in this module, we conducted ablation studies on them respectively. Table 5 shows the detailed ablation study results of the FGE module. From the experimental results, it can be seen that both HFA and HFS operations can improve the performance of semantic segmentation to a certain extent. The HFA operation can increase the mIoU index by 1.47%, and the HFS operation can increase the mIoU index by 0.81%. In general, the feature aggregation and separation operations in the FGE module can effectively control the multimodal data stream adaptively according to the different information provided by the cross-modal data, thereby enhancing the feature responses of the same semantic category while filtering out abnormal responses at the edge of the category. In addition, after the first network layer, the cross-modal features processed by the FGE module and those not processed by the FGE module were visualized and compared. According to the experimental results, both the RGB features and the depth features processed by the FGE module can capture more comprehensive detail information.

[0082] Table 5. Ablation experimental results of HFA and HFS operations in FGE module

[0083] The MMT module effectively integrates the information of RGB and depth modes through the attention mechanism to achieve efficient semantic segmentation. In order to explore the influence of the geometric depth information of the object on semantic segmentation, this embodiment attempts to use only RGB modality data as input and conducts experimental research with the same hyperparameter settings. It can be seen from the experimental results in Table 6 that compared with using a single RGB modality as input, the multimodal data input that integrates geometric depth information can bring a 1.58% improvement to the mIoU index. It can be seen that in large-scale remote sensing space, multimodal data that integrates visual texture information and geometric depth information can more effectively achieve semantic segmentation of objects. At the same time, in the process of calculating the Query vector by the MMT module, it is found that the semantic segmentation performance of the SkyMamba model will initially increase with the increase of the fixed pooling kernel size, and the best performance is achieved when the fixed pooling kernel size is 7×7. Table 7 shows the experimental results of different fixed pooling kernel sizes.

[0084] Table 6. Experimental results on the impact of deep modality data on semantic segmentation

[0085] Table 7. Experimental results of fixed pooling kernel size “k×k” parameter analysis for calculating Q vector in MMT module

[0086] The RSSGA module generates multiple sets of image block sequences by globally scanning the input remote sensing image from different directions such as horizontal, vertical and diagonal, and uses the state space model to capture the complex global information of the remote sensing space in different directions to effectively perform contextual comprehensive modeling. In order to study the impact of sequence combinations of different scanning directions on the final segmentation performance, we set up seven sets of comparative experiments with different direction combinations, and the experimental results are shown in Table 8. From the experimental results, it can be seen that the feature sequences of single-direction scanning can achieve similar semantic segmentation performance, and the feature sequences of any two-direction combination will bring about an improvement in segmentation performance. Omnidirectional scanning achieves the best semantic segmentation performance. More scanning direction combinations can provide richer contextual information, which helps to build global features in the remote sensing space to achieve excellent semantic segmentation performance.

[0087] Table 8. Experimental results of different scanning direction combinations of remote sensing image sequences in the RSSGA module

[0088] In one embodiment, Figure 7 As shown, a UAV remote sensing RGB-D image semantic segmentation device is provided, including: an RGB-D image acquisition and preprocessing module, an RGB modality and depth modality feature calibration module, a multi-modal visual feature extraction module of different scales and a semantic segmentation module, wherein: The RGB-D image acquisition and preprocessing module is used to acquire a set of RGB images taken by drones and depth maps of corresponding spatial dimensions, and perform preprocessing.

[0089] The RGB modality and depth modality feature calibration module is used to calibrate the features of different categories of ground objects using the preprocessed RGB modality input features and depth modality input features using the multimodal feature guidance enhancement module to obtain calibrated RGB modality features and depth modality features; the multimodal feature guidance enhancement module is used to use the high confidence activation of one modality feature to guide the correction of the features of another modality, thereby enhancing the similarity of features within the same category of objects and the difference between features of different categories of objects.

[0090] The multimodal visual feature extraction module of different scales is used to use a multi-scale encoder to perform multi-scale feature encoding on the calibrated RGB modal features and depth modal features to obtain multimodal visual features of different scales; the multi-scale encoder is used to extract the visual texture features and deep geometric features of the objects in the remote sensing scene from the perspective of drone aerial photography from different scales using multiple layers of ground object scene feature extraction modules; the ground object scene feature extraction module is used to realize the information interaction between RGB modality and depth modality through Transformer, and uses large kernel convolution operation to capture spatial geometric clues from the depth modality to refine the edge detail information of the ground object semantics, obtain multimodal fusion features and geometric depth modal features, and at the same time, the context information in the remote sensing scene is modeled from multiple directions through the state space model, and the modeling information in all directions is fused to obtain the global features of the remote sensing space, and then the multimodal fusion features, geometric depth modal features and the global features of the remote sensing space are connected in series and linearly projected to obtain visual features of one scale.

[0091] The semantic segmentation module is used to fuse multimodal visual features of different scales and use a lightweight decoder to predict the semantic labels of objects to obtain semantic segmentation results.

[0092] In one of the embodiments, the multimodal feature guidance enhancement module includes: a homogeneous feature aggregation operation and a heterogeneous feature separation operation; an RGB modality and a depth modality feature calibration module, which is also used to perform a homogeneous feature aggregation operation on the RGB modality input features and the depth modality input features obtained by preprocessing to obtain features after RGB modality aggregation and features after depth modality aggregation; the homogeneous feature aggregation operation is used to calibrate the features using the different representation capabilities of the RGB image modality and the depth map modality, and to complementarily aggregate the multimodal features of objects of the same semantic category to a certain position in the space; a heterogeneous feature separation operation is performed on the RGB modality input features and the depth modality input features obtained by preprocessing to obtain RGB modality features and depth modality features after modality separation; the heterogeneous feature separation operation is used to filter abnormal features at the edge of another modality object by using high confidence activation of one modality feature to achieve separation of heterogeneous features; the features after RGB modality aggregation and the RGB modality features after modality separation are fused to obtain calibrated RGB modality features; the features after depth modality aggregation and the depth modality features after modality separation are fused to obtain calibrated depth modality features.

[0093] In one of the embodiments, the RGB modality and depth modality feature calibration module is also used to concatenate the RGB modality input features and the depth modality input features obtained by preprocessing and then perform convolution processing to generate an RGB modality space gate and a depth modality space gate for controlling the feature information flow; the RGB modality space gate and the depth modality space gate are fused and mapped respectively to obtain the homogeneous feature aggregation weights of the RGB modality and the depth modality; wherein the sum of the values ​​of the homogeneous feature aggregation weights of the RGB modality and the depth modality at the same position is 1; the RGB modality input features are homogeneously aggregated through the homogeneous feature aggregation weights of the RGB modality to obtain the RGB modality features after homogeneity aggregation; the depth modality input features are homogeneously aggregated through the homogeneous feature aggregation weights of the depth modality to obtain the deep modality features after homogeneity aggregation; the RGB modality features and the deep modality features after homogeneity aggregation are interactively fused to obtain fused features; the fused features are used to enhance the RGB modality features and the deep modality features after homogeneity aggregation to obtain the features after RGB modality aggregation and the features after depth modality aggregation.

[0094] In one of the embodiments, the RGB modality and depth modality feature calibration module is also used to fuse the preprocessed RGB modality input features and the depth modality input features and obtain heterogeneous feature separation weights through MLP operations; the heterogeneous feature separation weights are multiplied with the RGB modality features and the depth modality features after homogeneity aggregation respectively to filter out abnormal features at the edges of different semantic classes, and obtain the RGB modality features and the depth modality features after modality separation.

[0095] In one embodiment, the multi-scale encoder includes 4 layers of ground object scene feature extraction modules of different scales and 3 downsampling layers; wherein the sampling layer is arranged between two layers of ground object scene feature extraction modules of different scales; the multimodal visual feature extraction modules of different scales are also used to input the calibrated RGB modal features and depth modal features into the first layer of ground object scene feature extraction module to obtain first scale visual features; downsample the first scale visual features and input them into the second layer of ground object scene feature extraction module to obtain second scale visual features; downsample the second scale visual features and input them into the third layer of ground object scene feature extraction module to obtain third scale visual features; downsample the third scale visual features and input them into the fourth layer of ground object scene feature extraction module to obtain fourth scale visual features.

[0096] In one embodiment, the ground object scene feature extraction module includes: a multimodal Transformer module and a remote sensing space global attention module; the first scale visual feature includes a first scale RGB modality and a depth modality visual feature; the different scale multimodal visual feature extraction module is also used to input the calibrated RGB modality feature and the depth modality feature into the multimodal Transformer module to obtain a multimodal fusion feature and a geometric depth modality feature; wherein the multimodal fusion feature and the geometric depth modality feature are shown in formulas (14) to (18).

[0097] The calibrated RGB modal features are input into the remote sensing space global attention module, the remote sensing space features are decomposed and the feature sequences are reconstructed in eight directions. The state space model is used to capture the contextual information in each feature sequence. Finally, the modeling features in each direction are fused to obtain the remote sensing space global features. The multimodal fusion features, geometric depth modal features and the global features of the remote sensing space are connected in series and linearly projected to obtain a scale of RGB modal and depth modal visual features.

[0098] In one of the embodiments, the remote sensing space global attention module includes: 1 feature decomposition and reconstruction module, 8 state space models and 1 feature fusion module; the multimodal visual feature extraction module of different scales is also used to input the calibrated RGB modal features into the feature decomposition and reconstruction module, decompose the calibrated RGB modal features into 12 local image blocks for global feature separation, and reconstruct the feature sequences of the separated local blocks along the horizontal, vertical, diagonal and anti-diagonal directions to obtain eight groups of feature sequences reconstructed in different directions; the state space models are used to capture the contextual information in different directions of the eight groups of feature sequences reconstructed in different directions to obtain modeling information in eight directions; the modeling information in eight directions is fused using the feature fusion module to obtain the global features of the remote sensing space.

[0099] In one embodiment, the lightweight decoder in the semantic segmentation module is a lightweight Hamburger decoder in the SegNext network model.

[0100] For the specific definition of the UAV remote sensing RGB-D image semantic segmentation device, please refer to the definition of the UAV remote sensing RGB-D image semantic segmentation method above, which will not be repeated here. Each module in the above-mentioned UAV remote sensing RGB-D image semantic segmentation device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0101] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for semantic segmentation of RGB-D images of remote sensing of unmanned aerial vehicles is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a key, trackball or touchpad set on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0102] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0103] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the present steps in the above method embodiment when executing the computer program.

[0104] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0105] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A semantic segmentation method for UAV remote sensing RGB-D images, characterized in that: The method comprises: Obtain a set of RGB images and depth maps of corresponding spatial dimensions taken by drones, and perform preprocessing; The RGB modal input features and the depth modal input features obtained by preprocessing are calibrated with the features of different categories of objects using a multimodal feature guidance enhancement module to obtain calibrated RGB modal features and depth modal features; the multimodal feature guidance enhancement module is used to use the high confidence activation of one modal feature to guide the correction of the features of another modality, thereby enhancing the similarity of features within the same category of objects and the difference between features of different categories of objects; The calibrated RGB modal features and depth modal features are encoded with a multi-scale encoder to obtain multi-modal visual features of different scales; the multi-scale encoder is used to extract visual texture features and deep geometric features of objects in remote sensing scenes from the perspective of drone aerial photography from different scales using multiple layers of ground object scene feature extraction modules; the ground object scene feature extraction module is used to realize information interaction between RGB modality and depth modality through Transformer, and uses large kernel convolution operation to capture spatial geometric clues from the depth modality to refine the edge detail information of ground object semantics, to obtain multi-modal fusion features and geometric depth modal features, and at the same time, the context information in the remote sensing scene is modeled from multiple directions through the state space model, and the modeling information in all directions is fused to obtain the global features of the remote sensing space, and then the multi-modal fusion features, the geometric depth modal features and the global features of the remote sensing space are connected in series and linearly projected to obtain visual features of one scale; After fusing multimodal visual features of different scales, a lightweight decoder is used to predict the semantic labels of objects to obtain semantic segmentation results.

2. The method according to claim 1, characterized in that The multimodal feature guided enhancement module includes: homogeneous feature aggregation operation and heterogeneous feature separation operation; The preprocessed RGB modal input features and depth modal input features are used to calibrate the features of different types of objects using a multimodal feature guidance enhancement module to obtain calibrated RGB modal features and depth modal features, including: A homogeneous feature aggregation operation is performed on the RGB modality input features and the depth modality input features obtained by preprocessing to obtain features after RGB modality aggregation and features after depth modality aggregation; the homogeneous feature aggregation operation is used to calibrate the features using the different representation capabilities of the RGB image modality and the depth map modality, and to complementarily aggregate the multimodal features of objects of the same semantic category to a certain position in space; A heterogeneous feature separation operation is performed on the RGB modal input features and the deep modal input features obtained by preprocessing to obtain RGB modal features and deep modal features after modality separation; the heterogeneous feature separation operation is used to filter abnormal features of the edge of another modality object by using high confidence activation of one modality feature to achieve separation of heterogeneous features; The features after RGB modality aggregation and the RGB modality features after modality separation are merged to obtain the calibrated RGB modality features; The features after deep modal aggregation and the deep modal features after modality separation are fused to obtain the calibrated deep modal features.

3. The method according to claim 2, characterized in that Homogeneous feature aggregation is performed on the RGB modality input features and the depth modality input features obtained by preprocessing to obtain RGB modality aggregated features and depth modality aggregated features, including: The RGB modality input features and the depth modality input features obtained by preprocessing are concatenated and then convolved to generate the RGB modality space gate and the depth modality space gate for controlling the feature information flow; The RGB modality space gate and the depth modality space gate are fused and mapped respectively to obtain homogeneous feature aggregation weights of the RGB modality and the depth modality; wherein the sum of the values ​​of the homogeneous feature aggregation weights of the RGB modality and the depth modality at the same position is 1; The RGB modality input features are homogeneously aggregated through the homogeneous feature aggregation weights of the RGB modality to obtain the homogeneously aggregated RGB modality features; The deep modality input features are homogeneously aggregated through the homogeneous feature aggregation weights of the deep modality to obtain the deep modality features after homogeneous aggregation; Interactively fuse the RGB modal features and the depth modal features after homogeneity aggregation to obtain fused features; The fusion features are used to enhance the RGB modality features and the depth modality features after homogeneity aggregation, respectively, to obtain the features after RGB modality aggregation and the features after depth modality aggregation.

4. The method according to claim 3, characterized in that A heterogeneous feature separation operation is performed on the RGB modal input features and the depth modal input features obtained by preprocessing to obtain the RGB modal features and the depth modal features after modality separation, including: The pre-processed RGB modality input features and the depth modality input features are fused and then operated through MLP to obtain the heterogeneous feature separation weights; The heterogeneous feature separation weights are multiplied with the RGB modal features and the depth modal features after homogeneity aggregation to filter out abnormal features at the edges of different semantic classes, thereby obtaining the RGB modal features and the depth modal features after modality separation.

5. The method according to claim 1, characterized in that The multi-scale encoder includes four layers of ground object scene feature extraction modules of different scales and three downsampling layers; wherein the downsampling layer is arranged between two layers of ground object scene feature extraction modules of different scales; The calibrated RGB modal features and depth modal features are encoded using a multi-scale encoder to obtain multi-modal visual features of different scales, including: Input the calibrated RGB modal features and depth modal features into the first-layer ground object scene feature extraction module to obtain the first-scale visual features; Downsampling the first-scale visual features and inputting them into a second-layer ground object scene feature extraction module to obtain second-scale visual features; Downsampling the second-scale visual features and inputting them into a third-layer ground object scene feature extraction module to obtain third-scale visual features; The third-scale visual features are downsampled and input into a fourth-layer ground object scene feature extraction module to obtain fourth-scale visual features.

6. The method according to claim 5, characterized in that The ground object scene feature extraction module includes: a multimodal Transformer module and a remote sensing space global attention module; the first-scale visual features include first-scale RGB modality and depth modality visual features; The calibrated RGB modal features and depth modal features are input into the first-layer ground object scene feature extraction module to obtain the first-scale visual features, including: The calibrated RGB modality features and depth modality features are input into the multimodal Transformer module to obtain multimodal fusion features and geometric depth modality features; the multimodal fusion features and geometric depth modality features are: in, represents multimodal fusion features, represents a bilinear upsampling operation, represents the softmax operation, Indicates the dimension of the current feature, and Respectively represent the Value vector and Key vector calculated from the RGB features, represents the Query vector, represents the geometric depth modal features, represents a large kernel depthwise convolution operation, represents the Hadamard product operation, and Respectively represent the RGB modality features and depth modality features of the input multimodal Transformer module, [· , ·] represents the cross-modality feature fusion operation, represents an adaptive average pooling operation of fixed size k×k, Represents the MLP network operation; The calibrated RGB modal features are input into the remote sensing space global attention module, the remote sensing space features are decomposed and feature sequences are reconstructed in several directions, a state space model is used to capture context information in each feature sequence, and finally the modeling features in each direction are fused to obtain the remote sensing space global features; The multimodal fusion features, the geometric depth modality features and the global features of the remote sensing space are connected in series and linearly projected to obtain a scaled RGB modality and depth modality visual feature.

7. The method according to claim 6, characterized in that The remote sensing spatial global attention module includes: 1 feature decomposition and reconstruction module, 8 state space models and 1 feature fusion module. The calibrated RGB modal features are input into the remote sensing space global attention module, the remote sensing space features are decomposed and feature sequences are reconstructed in eight directions, the state space model is used to capture the context information in each feature sequence, and finally the modeling features in each direction are fused to obtain the remote sensing space global features, including: The calibrated RGB modal features are input into the feature decomposition and reconstruction module, and the calibrated RGB modal features are decomposed into 12 local image blocks for global feature separation. The separated local blocks are reconstructed into feature sequences along the horizontal, vertical, diagonal and anti-diagonal directions to obtain eight groups of feature sequences reconstructed in different directions. The feature sequences reconstructed in eight different directions are respectively captured using the state space model in different directions to obtain modeling information in eight directions. The modeling information in eight directions is fused using the feature fusion module to obtain the global features of the remote sensing space.

8. The method according to claim 1, characterized in that The lightweight decoder is a lightweight Hamburger decoder in the SegNext network model.

9. A semantic segmentation device for UAV remote sensing RGB-D images, characterized in that: The device comprises: RGB-D image acquisition and preprocessing module, used to acquire a set of RGB images taken by drones and depth maps of corresponding spatial dimensions, and perform preprocessing; The RGB modality and depth modality feature calibration module is used to calibrate the features of different categories of objects using the multimodal feature guidance enhancement module to obtain calibrated RGB modality features and depth modality features by using the preprocessed RGB modality input features and the depth modality input features; the multimodal feature guidance enhancement module is used to use the high confidence activation of one modality feature to guide the correction of the features of another modality, thereby enhancing the similarity of features within the same category of objects and the difference between features of different categories of objects; A multi-scale visual feature extraction module is used to perform multi-scale feature encoding on the calibrated RGB modal features and depth modal features using a multi-scale encoder to obtain multi-scale visual features of different scales; the multi-scale encoder is used to extract visual texture features and deep geometric features of objects in remote sensing scenes from the perspective of drone aerial photography from different scales using multiple layers of ground object scene feature extraction modules; the ground object scene feature extraction module is used to realize information interaction between RGB modality and depth modality through Transformer, and uses large kernel convolution operation to capture spatial geometric clues from the depth modality to refine the edge detail information of the ground object semantics, obtain multi-modal fusion features and geometric depth modal features, and at the same time, model the context information in the remote sensing scene from multiple directions through the state space model, fuse the modeling information in all directions to obtain the global features of the remote sensing space, and then connect and linearly project the multi-modal fusion features, the geometric depth modal features and the global features of the remote sensing space to obtain visual features of one scale; The semantic segmentation module is used to fuse multimodal visual features of different scales and use a lightweight decoder to predict the semantic labels of objects to obtain semantic segmentation results.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.

Images