Method, device and equipment for semantic segmentation of RGB-D images from UAV remote sensing

Abstract

The present application relates to a method, apparatus and equipment for semantic segmentation of RGB-D images of remote sensing by unmanned aerial vehicles. The method proposes a multi-scale encoder for semantic segmentation of RGB-D images of remote sensing by unmanned aerial vehicles. The encoder includes multiple ground object scene feature extraction modules, which can fully integrate the visual texture information and geometric depth information provided by the RGB-D modality, and use the long-sequence modeling capability of the state space model to construct global context information in the remote sensing large-scale space, thereby achieving high-performance multi-modal semantic segmentation of remote sensing from the UAV perspective; a multi-modal feature guidance enhancement module is designed before the encoder, which uses the complementarity of information provided by RGB data and depth data to improve the quality of multi-modal feature representation, thereby improving the efficiency and performance of the model in the feature encoding learning stage. This method is specifically suitable for the study of semantic segmentation of RGB-D images of remote sensing by unmanned aerial vehicles, and promotes the continuous development of the study of ground object analysis in complex environments in the field of unmanned aerial vehicle remote sensing.

Description

Technical Field

[0001] The present application relates to the technical field of image semantic segmentation, and in particular to a method, apparatus, and device for semantic segmentation of RGB-D images remotely sensed by unmanned aerial vehicles (UAVs). Background Art

[0002] In recent years, with the increasing popularity of unmanned aerial vehicles (UAVs), their use in remote sensing imagery has become increasingly popular. Compared to satellite remote sensing platforms, UAVs offer flexible and controllable flight path planning, lower flight altitudes, higher spatial resolution, and reduced development and operating costs. The high-resolution remote sensing images collected by UAVs are a critical resource for numerous research fields, fostering in-depth understanding and analysis in areas such as urban planning, environmental change monitoring, agricultural management, and disaster response.

[0003] In the remote sensing field, a prerequisite for conducting various types of research and analysis on high-resolution remote sensing images acquired from UAVs is often the classification of objects within the spatial scene, a process known as semantic segmentation. Semantic segmentation involves dividing each pixel in an image into several semantically distinct, non-overlapping regions based on object category and labeling the pixels in these regions with specific category labels. Image semantic segmentation in remote sensing emphasizes capturing contextual semantic information across large spatial scales, thereby simplifying the distribution of complex objects within space. Semantic segmentation of remote sensing images has become the foundation of computer vision research in many remote sensing fields, such as road extraction, land use mapping, and urban feature perception.

[0004] Compared to satellite imagery, drone remote sensing images offer higher spatial resolution for semantic segmentation research. They can also provide diverse and fine-grained semantic feature representations from different heights and angles, enabling a comprehensive understanding of complex remote sensing spatial scenes. However, drone remote sensing images have a high spatial resolution, and rich contextual connections exist between multiple objects within large-scale spaces. Unlike natural images taken horizontally from the road, drone remote sensing images are collected from the air downward, so the distribution of ground features in drone remote sensing images can exist in any direction. Extracting omnidirectional spatial features from large-scale remote sensing spaces and modeling global contextual information is crucial for remote sensing intensive prediction tasks.

[0005] Another major advantage of drones is their flexible sensor mounting capabilities. With the recent proliferation of 3D sensors, UAV data collection tasks are no longer limited to RGB images, but are now expanding to include multimodal data such as point clouds and depth information. Research on the use of RGB-D data for high-level scene semantic understanding has garnered significant attention in recent years. Depth data can provide the 3D geometry of objects in a scene, providing complementary signals beyond visual spatial texture information, thereby improving the accuracy of semantic segmentation. In remote sensing spatial distribution from a UAV perspective, many objects have similar visual textures but significantly different geometric structures (e.g., grass and tall vegetation). RGB-D data, combining both visual texture information and depth geometry, can provide richer object features, enabling more accurate semantic understanding of complex remote sensing scenes. Currently, semantic parsing of RGB-D data is commonly applied in research such as autonomous vehicle driving, robot navigation, and SLAM, both indoors and outdoors, from a ground-level perspective. However, semantic segmentation methods for remote sensing RGB-D images from UAV aerial perspectives are relatively scarce, making this a relatively new research area. Summary of the Invention

[0006] Based on this, it is necessary to provide a method, device and equipment for semantic segmentation of UAV remote sensing RGB-D images to address the above technical problems.

[0007] A method for semantic segmentation of UAV remote sensing RGB-D images, the method comprising:

[0008] Obtain a set of RGB images taken by drones and depth maps of corresponding spatial dimensions and perform preprocessing.

[0009] The RGB modal input features and depth modal input features obtained by preprocessing are used to calibrate the features of different categories of ground objects using a multimodal feature guidance enhancement module to obtain calibrated RGB modal features and depth modal features; the multimodal feature guidance enhancement module is used to use the high confidence activation of one modal feature to guide the correction of the features of another modality, thereby enhancing the similarity of features within the same category of objects and the difference between features of different categories of objects.

[0010] The calibrated RGB modal features and depth modal features are encoded with a multi-scale encoder to obtain multi-modal visual features of different scales; the multi-scale encoder is used to extract the visual texture features and deep geometric features of objects in the remote sensing scene from the perspective of drone aerial photography from different scales using multiple layers of ground object scene feature extraction modules; the ground object scene feature extraction module is used to realize the information interaction between RGB modality and depth modality through Transformer, and uses large kernel convolution operation to capture spatial geometric clues from the depth modality to refine the edge detail information of the ground object semantics to obtain multi-modal fusion features and geometric depth modal features. At the same time, the context information in the remote sensing scene is modeled from multiple directions through the state space model, and the modeling information in all directions is fused to obtain the global features of the remote sensing space. Then, the multi-modal fusion features, geometric depth modal features and the global features of the remote sensing space are connected in series and linearly projected to obtain visual features of one scale.

[0011] After fusing multimodal visual features of different scales, a lightweight decoder is used to predict the semantic labels of objects to obtain semantic segmentation results.

[0012] A semantic segmentation device for UAV remote sensing RGB-D images, comprising:

[0013] The RGB-D image acquisition and preprocessing module is used to acquire a set of RGB images taken by drones and depth maps of corresponding spatial dimensions, and perform preprocessing.

[0014] The RGB modality and depth modality feature calibration module is used to calibrate the preprocessed RGB modality input features and depth modality input features using the multimodal feature guidance enhancement module to obtain calibrated RGB modality features and depth modality features; the multimodal feature guidance enhancement module is used to use the high confidence activation of one modality feature to guide the correction of the features of another modality, thereby enhancing the similarity of features within the same category of objects and the difference between features of different categories of objects.

[0015] The multimodal visual feature extraction module of different scales is used to encode the calibrated RGB modal features and depth modal features using a multi-scale encoder to obtain multimodal visual features of different scales; the multi-scale encoder is used to extract the visual texture features and deep geometric features of the objects in the remote sensing scene from the perspective of drone aerial photography from different scales using multiple layers of ground object scene feature extraction modules; the ground object scene feature extraction module is used to realize the information interaction between RGB modality and depth modality through Transformer, and use large kernel convolution operation to capture spatial geometric clues from the depth modality to refine the edge detail information of the ground object semantics to obtain multimodal fusion features and geometric depth modal features. At the same time, the context information in the remote sensing scene is modeled from multiple directions through the state space model, and the modeling information in all directions is fused to obtain the global features of the remote sensing space. Then, the multimodal fusion features, geometric depth modal features and the global features of the remote sensing space are connected in series and linearly projected to obtain visual features of one scale.

[0016] The semantic segmentation module is used to fuse multimodal visual features of different scales and use a lightweight decoder to predict the semantic labels of objects to obtain semantic segmentation results.

[0017] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0018] The above-mentioned method, apparatus, and device for semantic segmentation of UAV remote sensing RGB-D images proposes a multi-scale encoder for semantic segmentation of UAV remote sensing RGB-D images. This encoder includes multiple ground object scene feature extraction modules that fully integrate the visual texture information and geometric depth information provided by the RGB-D modality. It also utilizes the long-sequence modeling capabilities of the state-space model to construct global context information in the remote sensing large-scale space, achieving high-performance multimodal semantic segmentation of remote sensing from the UAV perspective. A multimodal feature guidance enhancement module is designed before the encoder, leveraging the complementarity of information provided by RGB and depth data to enhance the quality of multimodal feature representation, thereby improving the efficiency and performance of the model during the feature encoding learning phase. This method is specifically suitable for semantic segmentation research of UAV remote sensing RGB-D images and promotes the continuous development of complex environment ground object parsing research in the field of UAV remote sensing. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Schematic diagram of a flow chart of a method for semantic segmentation of RGB-D images from remote sensing of a UAV in one embodiment;

[0020] Figure 2 This is a diagram showing the overall architecture of a semantic segmentation model for UAV remote sensing RGB-D images in another embodiment;

[0021] Figure 3 is an architecture diagram of a multimodal feature guidance enhancement module in another embodiment;

[0022] Figure 4 A schematic diagram of a ground feature scene feature extraction module in another embodiment;

[0023] Figure 5 A schematic diagram of a process for implementing global spatial feature modeling for remote sensing images in another embodiment;

[0024] Figure 6 Schematic diagram of semantic categories of outdoor scene objects in another embodiment;

[0025] Figure 7 This is a structural block diagram of a semantic segmentation device for UAV remote sensing RGB-D images in one embodiment;

[0026] Figure 8 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0028] This application provides a semantic segmentation method for unmanned aerial vehicle (UAV) remote sensing RGB-D images. This method follows the mainstream encoder-decoder architecture for the semantic segmentation task of UAV remote sensing imagery. In the encoder part, multiple ground object scene feature extraction modules are used to extract and fuse the visual texture features and deep geometric features of the ground objects in the remote sensing scene from the UAV aerial perspective at different scales, and the extracted multi-scale features are converted into predicted ground object semantic labels through a lightweight decoder.

[0029] In one embodiment, Figure 1 As shown, a semantic segmentation method for UAV remote sensing RGB-D images is provided, which includes the following steps:

[0030] Step 100: Obtain a set of RGB images and depth maps of corresponding spatial dimensions taken by a drone, and perform preprocessing.

[0031] Specifically, unmanned aerial vehicle (UAV) aerial remote sensing images have higher resolution than satellite remote sensing images and can provide more detailed information. At the same time, combined with depth information, the performance of semantic segmentation of remote sensing images can be improved.

[0032] Step 102: The pre-processed RGB modal input features and depth modal input features are calibrated using a multimodal feature guidance enhancement module to obtain calibrated RGB modal features and depth modal features; the multimodal feature guidance enhancement module is used to use the high confidence activation of one modal feature to guide the correction of the features of another modality, thereby enhancing the similarity of features within the same category and the difference between features of different categories.

[0033] Specifically, given a set of UAV aerial RGB images and depth maps of corresponding spatial dimensions, a multimodal feature-guided enhancement module is first used to utilize the complementary information of the two modalities to calibrate the features of objects of different semantic categories in the scene space, suppress abnormal feature responses in edge areas, and enhance feature responses within the same semantic category.

[0034] Step 104: The calibrated RGB modal features and depth modal features are encoded using a multi-scale encoder to obtain multi-modal visual features of different scales; the multi-scale encoder is used to extract visual texture features and depth geometric features of objects in the remote sensing scene from the perspective of drone aerial photography from different scales using multiple layers of ground object scene feature extraction modules; the ground object scene feature extraction module is used to realize information interaction between RGB modalities and depth modalities through Transformer, and use large kernel convolution operations to capture spatial geometric clues from the depth modality to refine the edge detail information of the ground object semantics to obtain multi-modal fusion features and geometric depth modal features. At the same time, the context information in the remote sensing scene is modeled from multiple directions through the state space model, and the modeling information in all directions is fused to obtain the global features of the remote sensing space. Then, the multi-modal fusion features, geometric depth modal features, and the global features of the remote sensing space are connected in series and linearly projected to obtain visual features of a scale.

[0035] Specifically, the calibrated RGB modal features and depth modal features are fed into a multi-scale encoder. The multi-scale encoder architecture consists of four layers of ground object scene feature extraction modules. Each layer of ground object scene feature extraction modules performs multi-scale feature encoding on RGB-D features at {1 / 4, 1 / 8, 1 / 16, 1 / 32} of the original image size.

[0036] Step 106: After fusing the multimodal visual features of different scales, a lightweight decoder is used to predict the semantic labels of the objects to obtain the semantic segmentation results.

[0037] Specifically, the encoded multimodal visual features of different scales are fused and sent into a lightweight encoder to predict the semantic labels of the objects.

[0038] The semantic segmentation model used in this application is named the UAV remote sensing RGB-D image semantic segmentation model (abbreviated as: SkyMamba model). The model consists of a multimodal feature-guided enhancement module (FGE), a multi-scale encoder, and a lightweight decoder. The overall architecture of the UAV remote sensing RGB-D image semantic segmentation model is as follows: Figure 2 As shown. Figure 2 It can be seen that: the RGB modal data and depth modal data are input into the lightweight multimodal feature guidance enhancement module to calibrate the features of different categories of ground objects; then, the calibrated RGB-D modal data is input into the four-stage ground object scene feature extraction module for encoding, and finally the encoded features are passed to the lightweight decoder for final semantic prediction.

[0039] The aforementioned semantic segmentation method for UAV remote sensing RGB-D images proposes a multi-scale encoder for semantic segmentation of UAV remote sensing RGB-D images. This encoder includes multiple ground object scene feature extraction modules that fully integrate the visual texture information and geometric depth information provided by the RGB-D modality. It also leverages the long-sequence modeling capabilities of the state-space model to construct global contextual information within the large-scale remote sensing space, achieving high-performance multimodal semantic segmentation of remote sensing from a UAV perspective. A multimodal feature guidance enhancement module is designed before the encoder, leveraging the complementary information provided by RGB and depth data to enhance the quality of multimodal feature representation, improving the efficiency and performance of the model during the feature encoding learning phase. This method is specifically suitable for semantic segmentation research on UAV remote sensing RGB-D images and promotes the continuous development of complex environment feature parsing research in the field of UAV remote sensing.

[0040] In one embodiment, the multimodal feature guidance enhancement module includes: a homogeneous feature aggregation operation and a heterogeneous feature separation operation; step 102 specifically includes the following steps:

[0041] Step 200: A homogeneous feature aggregation operation is performed on the RGB modality input features and the depth modality input features obtained by preprocessing to obtain RGB modality aggregated features and depth modality aggregated features; the homogeneous feature aggregation operation is used to calibrate the features using the different representation capabilities of the RGB image modality and the depth map modality, and to complementarily aggregate the multimodal features of objects of the same semantic category to a certain position in space.

[0042] Specifically, the architecture of the multimodal feature guidance enhancement module is as follows Figure 3 As shown in the figure, the multimodal feature guidance enhancement module includes: homogeneous feature aggregation operation and heterogeneous feature separation operation; the multimodal feature guidance enhancement module uses the high-confidence activation of different modal features to guide the correction of ground feature of different semantic categories, suppress the abnormal feature response in the edge area of ​​the category object, and strengthen the homogeneous feature response within the same category object.

[0043] Homogeneous Feature Aggregation (HFA): The object feature information presented by RGB and depth modal features is complementary, providing visual texture information and geometric structure information of objects in the remote sensing space, respectively. To aggregate homogeneous features belonging to objects of the same semantic category in outdoor scenes, it is necessary to calibrate the features using the representation capabilities of different modalities, and complementary aggregate the multimodal features of objects of the same semantic category to a single location in space.

[0044] Step 202: A heterogeneous feature separation operation is performed on the RGB modal input features and the depth modal input features obtained by preprocessing to obtain RGB modal features and depth modal features after modal separation. The heterogeneous feature separation operation is used to use high-confidence activation of one modal feature to filter abnormal features on the edge of another modal object, thereby achieving separation of heterogeneous features.

[0045] Specifically, heterogeneous feature separation (HFS): In outdoor scenes, there are many objects with similar visual textures. For example, tall vegetation and grass can easily produce blurred, abnormal feature responses at the edges of objects due to their texture similarity during feature aggregation. However, tall vegetation and grass have significant differences in spatial geometric structure. Therefore, we consider leveraging the differences in features from different modal data to separate heterogeneous feature responses at the edges of categorical objects, making the feature distinctions between different classes more significant. In practice, high-confidence activation of one modal feature is used to filter abnormal features at the edges of objects from another modality, making the edge features of objects of different semantic categories more contrasting and achieving heterogeneous feature separation.

[0046] Step 204: Fuse the RGB modality features after modality aggregation and the RGB modality features after modality separation to obtain calibrated RGB modality features.

[0047] Step 206: Fuse the features after deep modality aggregation and the deep modality features after modality separation to obtain calibrated deep modality features.

[0048] In one embodiment, step 200 includes: concatenating the RGB modal input features and the depth modal input features obtained by preprocessing and performing convolution processing to generate an RGB modal space gate and a depth modal space gate for controlling the feature information flow; fusing and mapping the RGB modal space gate and the depth modal space gate respectively to obtain homogeneous feature aggregation weights of the RGB modality and the depth modality; wherein the sum of the values ​​of the homogeneous feature aggregation weights of the RGB modality and the depth modality at the same position is 1; homogeneously aggregating the RGB modal input features through the homogeneous feature aggregation weights of the RGB modality to obtain homogeneously aggregated RGB modal features; homogeneously aggregating the depth modal input features through the homogeneous feature aggregation weights of the depth modality to obtain homogeneously aggregated depth modal features; interactively fusing the homogeneously aggregated RGB modal features and the depth modal features to obtain fused features; and using the fused features to enhance the homogeneously aggregated RGB modal features and the depth modal features respectively to obtain RGB modal aggregated features and depth modal aggregated features.

[0049] Specifically, during the homogeneous feature aggregation operation: first, the RGB modality input features and the depth modality input features are concatenated and convolved to generate the corresponding spatial gates. and To control the information flow of RGB and depth modal feature maps, and then fuse the modal space gates that control the feature information flow to obtain the homogeneous feature aggregation weights and :

[0050] (1)

[0051] (2)

[0052] in, and Represent the homogeneous feature aggregation weights of RGB modality and depth modality respectively, and , and Represents the spatial gates of RGB modality features and depth modality features respectively. Homogeneous feature aggregation weight and In the same location The sum of the values ​​on is 1:

[0053] (3)

[0054] Homogeneous feature aggregation weights can effectively aggregate the features of the same semantic category in the RGB modality and the depth modality, and the input feature and After homogeneous aggregation, the RGB modal features after homogeneous aggregation are obtained. and deep modal features :

[0055] (4)

[0056] (5)

[0057] Then, we interact the RGB features and depth features obtained after homogeneity aggregation to obtain the fusion features :

[0058] (6)

[0059] Finally, using the fusion features The two modal features are further enhanced to obtain the features after RGB modality aggregation and features after deep modality aggregation , to strengthen the similarity of features of objects in the same category:

[0060] (7)

[0061] (8)

[0062] In one embodiment, step 202 includes: fusing the pre-processed RGB modal input features and the depth modal input features and performing an MLP operation to obtain heterogeneous feature separation weights; multiplying the heterogeneous feature separation weights with the homogeneity-aggregated RGB modal features and the depth modal features to filter abnormal features at the edges of different semantic classes, thereby obtaining RGB modal features and depth modal features after modality separation.

[0063] Specifically, in the heterogeneous feature separation (HFS) process, the RGB modality input features and the depth modality input features are first fused and operated through the MLP to obtain the heterogeneous feature separation weights. :

[0064] , (9)

[0065] in, and represents the input RGB modality input features and depth modality input features, [·,·] represents the cross-modal feature fusion operation, represents the global average pooling operation, Represents the MLP network operation, Represents a sigmoid function that scales the heterogeneous feature separation weights to (0,1).

[0066] Then, the RGB modal features after homogeneity aggregation are and deep modal features and Multiply to filter the abnormal features of the edges of different semantic classes to obtain the RGB modal features after modal separation and deep modal features :

[0067] (10)

[0068] (11)

[0069] Finally, the features after modal separation and aggregation are combined to obtain the calibrated and enhanced multimodal features:

[0070] (12)

[0071] (13)

[0072] in, and The following represent the RGB modal features and depth modal features after feature-guided enhancement, respectively. The idea behind this operation is to leverage the complementarity between the spatial texture information and spatial geometric information of objects in outdoor scenes, using this complementary feature information as an offset to refine and correct the class semantic information in different modalities. This makes the features of objects in the same semantic category more similar, while the features of objects in different semantic categories are more separated, thereby forming more robust class semantic information for better feature encoding learning by the subsequent object scene feature extraction module.

[0073] In one embodiment, the multi-scale encoder includes four layers of ground object scene feature extraction modules of different scales and three downsampling layers; wherein the downsampling layer is provided between two layers of ground object scene feature extraction modules of different scales; step 104 specifically includes the following steps:

[0074] Step 300: Input the calibrated RGB modal features and depth modal features into the first-layer ground feature scene feature extraction module to obtain first-scale visual features;

[0075] Step 302: downsample the first-scale visual features and input them into the second-layer ground object scene feature extraction module to obtain second-scale visual features.

[0076] Step 304: downsample the second-scale visual features and input them into the third-layer ground object scene feature extraction module to obtain third-scale visual features.

[0077] Step 306: downsample the third-scale visual features and input them into the fourth-layer ground object scene feature extraction module to obtain fourth-scale visual features.

[0078] Specifically, the multi-scale encoder includes four layers of ground object scene feature extraction modules of different scales, which are used to encode the multi-scale RGB-D features of ground objects in remote sensing scenes. A convolution with a 3×3 kernel and a stride of 2 is set between each layer of ground object scene feature extraction modules of different scales to downsample the RGB-D cross-modal features.

[0079] In one embodiment, Figure 4 As shown, the ground object scene feature extraction module includes: a multimodal Transformer module and a remote sensing space global attention module; the first-scale visual features include first-scale RGB modality and depth modality visual features; step 300 specifically includes the following steps:

[0080] Step 400: Input the calibrated RGB modal features and depth modal features into the multimodal Transformer module to obtain multimodal fusion features and geometric depth modal features; wherein the multimodal fusion features and geometric depth modal features are:

[0081] (14)

[0082] (15)

[0083] (16)

[0084] (17)

[0085] (18)

[0086] in, represents multimodal fusion features, represents a bilinear upsampling operation, represents the softmax operation, Indicates the dimension of the current feature, and Represents the Value vector and Key vector calculated from the RGB features, represents the Query vector, represents the geometric depth modal features, represents a large kernel depthwise convolution operation, represents the Hadamard product operation, and Represent the RGB modality features and depth modality features of the input multimodal Transformer module respectively, [·,·] represents the cross-modal feature fusion operation, represents an adaptive average pooling operation of fixed k×k size, Represents an MLP network operation.

[0087] Specifically, the Multimodal Transformer module (MMT) uses Transformer to fuse RGB modality and depth modality features to establish cross-modal feature interaction relationships, enhance 3D perception in remote sensing space, and further capture semantic category information; at the same time, a large kernel convolution operation is used to further capture spatial geometric clues from the depth modality to refine the edge detail information of the semantics of ground objects.

[0088] Step 402: Input the calibrated RGB modal features into the remote sensing spatial global attention module, decompose the remote sensing spatial features and reconstruct the feature sequences in eight directions, use the state space model to capture the contextual information in each feature sequence, and finally fuse the modeling features in each direction to obtain the remote sensing spatial global features.

[0089] Specifically, the Remote Sensing Spatial Global Attention Module (RSSGA) decomposes the large-scale features of the remote sensing space into local features and reconstructs feature sequences from eight directions. It models the contextual information in the remote sensing scene from eight directions through the state space model, and fuses the modeling information in the eight directions to obtain the global features of the entire remote sensing space, thereby enhancing the correlation between the semantic information of the ground objects in the remote sensing scene.

[0090] The RGB modality and depth modality in remote sensing scenes provide visual texture information and spatial geometry information of objects respectively. In order to interact the information provided by the two modalities and enhance the semantic category features, we propose an MMT module to interact with multimodal features. The MMT module extracts the input RGB modal features from the image. The Key vector and Value vector are calculated; the Value vector is shown in formula (17), and the Key vector is shown in formula (18).

[0091] During the calculation of the query vector, the MMT module concatenates the input RGB features and depth features and performs a pooling operation of a fixed size to obtain a query vector that integrates multimodal information; the query vector is shown in formula (16).

[0092] In getting the Value vector , Key vector and Query vector After that, self-attention operation is performed and up-sampled to the corresponding dimension to obtain multimodal fusion features; the multimodal fusion features are shown in formula (14).

[0093] The depth modality in the remote sensing space can provide more detailed information about the edge of objects. To preserve more spatial structural information, we use a large kernel to convolve the depth features in the MMT module and perform Hadamard product operations with the query vector to embed more information, thereby enhancing the geometric structure information of objects in the remote sensing space and obtaining geometric depth modality features. The geometric depth modality features are shown in Formula (15). It is the deep feature after the large kernel depth convolution processing and the product of the query vector.

[0094] Step 404: The multimodal fusion features, the geometric depth modal features, and the global features of the remote sensing space are concatenated and linearly projected to obtain a scale of RGB modal and depth modal visual features.

[0095] Specifically, the cross-modal fusion features and geometric depth features obtained by the MMT module are concatenated and linearly projected with the remote sensing spatial global features obtained by the remote sensing spatial global attention module (RSSGA) to enhance the feature representation in the scene and thus achieve feature updating.

[0096] In one embodiment, the remote sensing space global attention module includes: 1 feature decomposition and reconstruction module, 8 state space models and 1 feature fusion module; step 402 includes: inputting the calibrated RGB modal features into the feature decomposition and reconstruction module, decomposing the calibrated RGB modal features into 12 local image blocks for global feature separation, reconstructing the feature sequences of the separated local blocks along the horizontal, vertical, diagonal and anti-diagonal directions, and obtaining eight groups of feature sequences reconstructed in different directions; using the state space model to capture the context information in different directions of the eight groups of feature sequences reconstructed in different directions, and obtaining modeling information in eight directions; using the feature fusion module to fuse the modeling information in the eight directions to obtain the remote sensing space global features.

[0097] Specifically, the distribution of remote sensing spatial features from a UAV perspective is different from that from a ground road perspective. Objects have strong correlations and rich contextual information, and the distribution of object features in remote sensing images can be in any direction. To capture multi-directional global information in the remote sensing space, this application proposes a Remote Sensing Spatial Global Attention Module (RSSGA) that decomposes remote sensing spatial features and reconstructs feature sequences in eight directions. It uses a state-space model to capture the contextual information in each feature sequence, and finally merges the modeled features in each direction to obtain the global features in the remote sensing space.

[0098] We decompose the remote sensing image input to the RSSGA module into 12 local image blocks for global spatial feature separation. We then reconstruct feature sequences from these decomposed local blocks along the horizontal, vertical, diagonal, and anti-diagonal directions. Reconstructing feature sequences in different directions helps the encoder extract richer global features from the remote sensing space.

[0099] The schematic diagram of the process of realizing global spatial feature modeling of remote sensing images is as follows: Figure 5 As shown in the figure, high-resolution remote sensing images are feature separated and sequence reconstructed according to specific directions. Context information is modeled in each direction using a state-space model. Finally, the context information in all directions is fused to achieve remote sensing global spatial feature modeling.

[0100] The eight sets of feature sequences reconstructed in different directions are modeled along the new dimension using the state space model for global feature modeling. The state space model can encapsulate the state information in the dynamic system and can therefore effectively model the feature context information. First, in order to realize the dynamic perception and reasoning of context information in the remote sensing space, the RSSGA module is based on the i-th local feature map of each input directional feature sequence. Adaptively obtain the corresponding state transfer matrix , input state matrix and the output state matrix , where the matrix and matrix By parameters Zero-order hold technology is used for discretization processing:

[0101] (19)

[0102] (20)

[0103] in, and Represent the discretized state transfer matrix and input state matrix respectively, represents the discretization parameter, Represents the identity matrix.

[0104] Secondly, according to the adaptive parameters The dynamically changing state matrix can selectively focus on perceiving the contextual information in different local feature maps. With the input state matrix Multiply to get the state representation of the current feature fragment :

[0105] (twenty one)

[0106] The current local feature state represents the retained state of the previous local feature Add to realize feature context information modeling, and retain the state of the previous local features It is to selectively save the important spatial features of the previous fragment in order to give it to the current local feature state to obtain a new feature state :

[0107] (twenty two)

[0108] The retention state of the previous local features It is composed of the previous local features And the corresponding state transfer matrix Multiplying them gives:

[0109] (twenty three)

[0110] Finally, the new local feature state and the output state matrix Multiply to get new local spatial features :

[0111] (twenty four)

[0112] The new local spatial feature integrates the spatial distribution information of the previous feature, realizes the context information modeling in a specific direction, and at the same time, the current local feature and the corresponding state transfer matrix Multiply to get the retained state To provide the previous feature information for the input features of the next stage. After reconstructing multiple feature sequences in different directions, all local feature sequences are merged and recombined to realize the global feature modeling of the remote sensing space:

[0113] (25)

[0114] in, Represents the global characteristics of remote sensing space, Represents a multi-directional feature sequence recombination operation, Represents a unidirectional reconstructed feature sequence.

[0115] In one embodiment, the lightweight decoder in step 106 is a lightweight Hamburger decoder in the SegNext network model.

[0116] Specifically, this application uses the lightweight Hamburger decoder in SegNext to aggregate and decode multi-scale RGB-D features. Due to the powerful feature extraction capability of the multi-scale encoder, choosing a lightweight decoder can achieve excellent performance.

[0117] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0118] In a verification embodiment, UAV aerial remote sensing images have higher resolution than satellite remote sensing images and can provide more detailed information. At the same time, the combination of depth information can improve the performance of remote sensing image semantic segmentation. In this embodiment, a new UAV aerial remote sensing RGB-D image semantic segmentation benchmark dataset CUS-Depth V1 is proposed. This dataset is obtained by expanding the latest comprehensive three-dimensional semantic segmentation dataset CUS3D

[61] . The CUS-DepthV1 dataset contains a total of 4336 UAV aerial RGB images and corresponding depth maps from Changsha, China at orthogonal and 45° oblique viewing angles. The image resolution is 6144×4096, of which 2601 images are used for training, 867 images are used for verification, and 868 images are used for testing. The CUS-Depth V1 dataset covers a variety of outdoor scenes such as cities, suburbs, and rural areas, covering an area of ​​2.85 square kilometers. This embodiment provides 10 common outdoor scene semantic categories such as buildings, roads, tall vegetation, grasslands, water bodies, vehicles, farmlands, construction sites, stadiums, and bare surfaces. Figure 6 shown. Figure 6 Middle left: Schematic diagram of the area covered by the original RGB image of the CUS-Depth V1 dataset; right: Schematic diagram of the distribution range of semantic labels corresponding to the RGB image of the CUS-Depth V1 dataset.

[0119] In order to more effectively promote the development of community research on semantic segmentation of UAV aerial remote sensing RGB-D images, this application also provides the more challenging CUS-Depth V2 dataset. The image resolution, number and scene coverage of this dataset are the same as the basic version CUS-Depth V1. The difference is that compared with the basic version CUS-Depth V1, the dataset provides more semantic categories, including 18 semantic categories of land objects, including high-rise buildings, villas, vehicles, pedestrians, construction sites, non-motor vehicles, grass, tall vegetation, roads, sidewalks, farmland, mud, water, track and field, basketball courts, children's playgrounds, power supply equipment and miscellaneous items.

[0120] (1) Benchmark method and experimental details

[0121] In order to evaluate the performance of the SkyMamba model proposed in this application in the semantic segmentation of UAV aerial remote sensing RGB-D images, this example selected six representative RGB-D image semantic segmentation methods, including ACNet, SA-Gate, TokenFusion, CMX, CMNext, and DFormer, as benchmarks and conducted comparative experiments on the CUS-Depth V1 and CUS-Dpeth V2 datasets.

[0122] Data Preprocessing: To avoid performance errors caused by complex data augmentation techniques, we used only two simple data augmentation techniques: horizontal flipping and random scaling. Furthermore, for ease of research, we processed the input remote sensing images into non-overlapping 512×512 pixel blocks for model training. All baseline methods tested on the same dataset used the same data preprocessing strategy to ensure experimental consistency.

[0123] Experimental Setup: The SkyMamba model and other baseline comparison experiments were deployed using the PyTorch framework on a single NVIDIA Tesla A800 GPU. Batch sizes were adjusted based on the specific performance of each model to maximize computational efficiency. We optimized the SkyMamba model's performance by combining cross-entropy loss and the Dice coefficient. The AdamW optimizer was used to update model parameters with an initial learning rate of 0.001 and a weight decay of 0.001. The model converged after 150 epochs of training. Table 1 provides detailed experimental parameter settings for the SkyMamba model and other baseline comparison methods on the CUS-Depth V1 and CUS-Depth V2 datasets.

[0124] Table 1. Hyperparameter settings for the comparative experiments of each baseline method on the CUS-Depth V1 and CUS-Depth V2 datasets

[0125]

[0126] Evaluation Metrics: To evaluate the performance of semantic segmentation models, we use the intersection-over-union (IoU) for each semantic category, the overall mean IoU (mIoU), and the overall mean accuracy (mAcc) as evaluation metrics. The IoU and mIoU metrics measure the degree of overlap between segmentation results and ground truth, providing an assessment of the spatial accuracy of model predictions. The mAcc metric measures the number of correctly classified pixels in the segmentation results and reflects the model's overall performance in pixel classification.

[0127] (2) Comparative experimental results and discussion

[0128] To validate the advanced performance of the SkyMamba model proposed in this application for semantic segmentation of UAV aerial remote sensing RGB-D images, the final SkyMamba model was compared with baseline RGB-D semantic segmentation methods on the CUS-Depth V1 and CUS-Depth V2 datasets. Table 2 shows the experimental test results of different semantic segmentation methods on the CUS-Depth V1 dataset. The SkyMamba model achieved the highest mean Intersection Over Union (mIoU) on the CUS-Depth V1 dataset (78.46%), a 4.41% improvement over the state-of-the-art RGB-D semantic segmentation method, DFormer (74.05%). The SkyMamba model also achieved the second-highest mAc (87.95%) on the CUS-Depth V1 dataset, second only to SA-Gate (88.68%). In terms of the IoU metric for each semantic category, the SkyMamba model achieved the highest performance in most categories. Its IoU in the Lake category was 95.87%, second only to CMX (96.41%). However, it achieved an IoU of 81.45% in a small category with a smaller distribution, such as cars, far exceeding other methods.

[0129] Table 2. Comparison results of SkyMamba and comparison methods on the CUS-Depth V1 dataset

[0130]

[0131] Note: The comparison results include average category accuracy (mAcc, %), average intersection over union (mIoU, %), and intersection over union (IoU) of each category (%).

[0132] In real-world scenarios, there are many fine and complexly distributed objects in remote sensing scenes. The experimental results on the CUS-Depth V1 dataset show that the SkyMamba model has a stronger ability to capture details in large-scale spaces than other methods. To further verify the semantic segmentation capability of the SkyMamba model in large-scale spaces, a comparative experiment was conducted on the more challenging CUS-Depth V2 dataset. Table 3 shows the experimental test results of different semantic segmentation methods on the CUS-Depth V2 dataset. The results show that even in the face of complex remote sensing scenes with 18 semantic categories, SkyMamba can still achieve the highest mIoU (63.82%), which is 3.25% higher than the second-highest method CMNext

[62] (60.57%) and 4.76% higher than the latest RGB-D semantic segmentation method DFormer

[38] (59.06%). At the same time, SkyMamba achieves the second-highest mAc (72%). Notably, SkyMamba demonstrates excellent segmentation performance on sparsely distributed fine features such as cars, pedestrian lanes, athletic fields, children's playgrounds, and debris. It also achieves a more balanced semantic segmentation effect on complex features in large-scale spaces, effectively overcoming the imbalanced distribution of semantic categories in the dataset to a certain extent.

[0133] Table 3. Comparison results of SkyMamba and comparison methods on the CUS-Depth V2 dataset

[0134]

[0135] Note: The comparison results include the average category accuracy (mAcc,%), average intersection-over-union (mIoU,%), and the intersection-over-union (IoU) of each category (%).

[0136] In summary, the SkyMamba model achieves state-of-the-art performance in semantic segmentation of UAV aerial RGB-D imagery. Object features in remote sensing scenes are distributed in diverse directions and cover a wide range of spatial scales. The SkyMamba model is designed to combine the visual texture and geometric depth information of objects, capturing global contextual information and substantive spatial semantic category information from all directions across large spatial scales, thereby achieving accurate semantic segmentation of remote sensing images.

[0137] (3) Ablation experiments and discussion

[0138] The SkyMamba model consists of multiple ground feature extraction modules, each composed of a Multimodal Transformer (MMT) module and a Remote Sensing Spatial Global Attention (RSSGA) module. A lightweight Cross-Modal Feature Guided Enhancement (FGE) module is designed for feature refinement before feature encoding. To further investigate the effectiveness of each component of the SkyMamba model on semantic segmentation performance, we conducted an ablation study on the CUS-Depth V1 dataset using the same hyperparameter settings.

[0139] Overall component ablation study. The FGE module, MMT module, and RSSGA model were extracted from the SkyMamba model and experiments were conducted on the CUS-Depth V1 dataset. The results are shown in Table 4. The ablation study results show that all components are essential for the SkyMamba model. The FGE module can bring a 2.28% improvement in the semantic segmentation performance of the SkyMamba model, while basically not incurring any additional parameters or computational complexity. The MMT module can bring a 3.87% improvement in the semantic segmentation performance indicator mIoU, effectively fusing multimodal features to achieve efficient semantic segmentation. The RSSGA module can provide the most effective performance improvement for the SkyMamba model, with a 5.99% improvement in the mIoU indicator. This is due to its global spatial modeling capability that can effectively capture and provide global context information of large-scale remote sensing spaces.

[0140] Table 4. Ablation test results of each component of the SkyMamba model

[0141]

[0142] In the FGE module, homogeneous feature aggregation (HFA) and heterogeneous feature separation (HFS) are used to suppress anomalous feature responses at the edges of objects in remote sensing scenes and strengthen homogeneous feature responses within objects of the same category, thereby enhancing the essential semantic category information of the objects in space. To further investigate the effectiveness of the HFA and HFS operations in this module, we conducted ablation studies on them. Table 5 shows the detailed ablation results of the FGE module. The experimental results show that both HFA and HFS improve semantic segmentation performance to a certain extent. HFA can increase the mean Intersection Over Union (MIOU) metric by 1.47%, and HFS can increase the mIoU metric by 0.81%. Overall, the feature aggregation and separation operations in the FGE module effectively and adaptively control the multimodal data flow based on the diverse information provided by cross-modal data, thereby enhancing feature responses within the same semantic category while filtering out anomalous responses at the category edges. In addition, after the first network layer, the cross-modal features processed by the FGE module and those not processed by the FGE module were visualized and compared. According to the experimental results, it can be seen that the RGB features and depth features processed by the FGE module can capture more comprehensive detail information.

[0143] Table 5. Ablation experiment results of HFA and HFS operations in the FGE module

[0144]

[0145] The MMT module effectively integrates the information of RGB and depth modalities through the attention mechanism to achieve efficient semantic segmentation. In order to explore the impact of the geometric depth information of the object on semantic segmentation, this embodiment attempts to use only RGB modality data as input and conduct experimental research with the same hyperparameter settings. From the experimental results in Table 6, it can be seen that compared with using a single RGB modality as input, the multimodal data input that integrates geometric depth information can bring a 1.58% improvement to the mIoU indicator. This shows that in large-scale remote sensing space, multimodal data that integrates visual texture information and geometric depth information can more effectively achieve semantic segmentation of ground objects. At the same time, in the process of calculating the query vector in the MMT module, it is found that the semantic segmentation performance of the SkyMamba model will initially improve with the increase of the fixed pooling kernel size, and the best performance is achieved when the fixed pooling kernel size is 7×7. Table 7 shows the experimental results of different fixed pooling kernel sizes.

[0146] Table 6. Experimental results on the impact of deep modal data on semantic segmentation

[0147]

[0148] Table 7. Experimental results of fixed pooling kernel size “k×k” parameter analysis for calculating Q vector in MMT module

[0149]

[0150] The RSSGA module generates multiple image block sequences by globally scanning the input remote sensing image in different directions, such as horizontally, vertically, and diagonally. It then uses a state-space model to capture the complex global information of the remote sensing space in different directions, effectively modeling the context comprehensively. To study the impact of sequence combinations of different scanning directions on the final segmentation performance, we conducted seven comparative experiments with different direction combinations. The experimental results are shown in Table 8. The experimental results show that feature sequences scanned in a single direction can achieve similar semantic segmentation performance. Feature sequences in any two-way combination of directions will improve segmentation performance, and omnidirectional scanning achieves the best semantic segmentation performance. More scanning direction combinations can provide richer contextual information, which helps to construct global features in the remote sensing space and achieve excellent semantic segmentation performance.

[0151] Table 8. Experimental results of different scanning direction combinations of remote sensing image sequences in the RSSGA module

[0152]

[0153] In one embodiment, Figure 7 As shown, a semantic segmentation device for UAV remote sensing RGB-D images is provided, comprising: an RGB-D image acquisition and preprocessing module, an RGB modality and depth modality feature calibration module, a multi-modal visual feature extraction module at different scales, and a semantic segmentation module, wherein:

[0154] The RGB-D image acquisition and preprocessing module is used to acquire a set of RGB images taken by drones and depth maps of corresponding spatial dimensions, and perform preprocessing.

[0155] The RGB modality and depth modality feature calibration module is used to calibrate the preprocessed RGB modality input features and depth modality input features using the multimodal feature guidance enhancement module to obtain calibrated RGB modality features and depth modality features; the multimodal feature guidance enhancement module is used to use the high confidence activation of one modality feature to guide the correction of the features of another modality, thereby enhancing the similarity of features within the same category of objects and the difference between features of different categories of objects.

[0156] The multimodal visual feature extraction module of different scales is used to encode the calibrated RGB modal features and depth modal features using a multi-scale encoder to obtain multimodal visual features of different scales; the multi-scale encoder is used to extract the visual texture features and deep geometric features of the objects in the remote sensing scene from the perspective of drone aerial photography from different scales using multiple layers of ground object scene feature extraction modules; the ground object scene feature extraction module is used to realize the information interaction between RGB modality and depth modality through Transformer, and use large kernel convolution operation to capture spatial geometric clues from the depth modality to refine the edge detail information of the ground object semantics to obtain multimodal fusion features and geometric depth modal features. At the same time, the context information in the remote sensing scene is modeled from multiple directions through the state space model, and the modeling information in all directions is fused to obtain the global features of the remote sensing space. Then, the multimodal fusion features, geometric depth modal features and the global features of the remote sensing space are connected in series and linearly projected to obtain visual features of one scale.

[0157] The semantic segmentation module is used to fuse multimodal visual features of different scales and use a lightweight decoder to predict the semantic labels of objects to obtain semantic segmentation results.

[0158] In one embodiment, the multimodal feature guidance enhancement module includes: a homogeneous feature aggregation operation and a heterogeneous feature separation operation; an RGB modality and depth modality feature calibration module, which is also used to perform a homogeneous feature aggregation operation on the pre-processed RGB modality input features and depth modality input features to obtain RGB modality aggregated features and depth modality aggregated features; the homogeneous feature aggregation operation is used to calibrate features using the different representation capabilities of the RGB image modality and the depth map modality, and to complementarily aggregate the multimodal features of objects of the same semantic category to a certain position in space; a heterogeneous feature separation operation is performed on the pre-processed RGB modality input features and depth modality input features to obtain modality-separated RGB modality features and depth modality features; the heterogeneous feature separation operation is used to use the high-confidence activation of one modality feature to filter abnormal features on the edge of another modality object to achieve heterogeneous feature separation; the features after RGB modality aggregation and the RGB modality features after modality separation are fused to obtain calibrated RGB modality features; the features after depth modality aggregation and the depth modality features after modality separation are fused to obtain calibrated depth modality features.

[0159] In one embodiment, the RGB modality and depth modality feature calibration module is further used to concatenate the pre-processed RGB modality input features and the depth modality input features and then perform convolution processing to generate an RGB modality space gate and a depth modality space gate for controlling the feature information flow; the RGB modality space gate and the depth modality space gate are fused and mapped respectively to obtain the homogeneous feature aggregation weights of the RGB modality and the depth modality; wherein the sum of the values ​​of the homogeneous feature aggregation weights of the RGB modality and the depth modality at the same position is 1; the RGB modality input features are homogeneously aggregated through the homogeneous feature aggregation weights of the RGB modality to obtain the homogeneously aggregated RGB modality features; the depth modality input features are homogeneously aggregated through the homogeneous feature aggregation weights of the depth modality to obtain the homogeneously aggregated depth modality features; the homogeneously aggregated RGB modality features and the depth modality features are interactively fused to obtain fused features; the fused features are used to enhance the homogeneously aggregated RGB modality features and the depth modality features respectively to obtain the RGB modality aggregated features and the depth modality aggregated features.

[0160] In one embodiment, the RGB modality and depth modality feature calibration module is also used to fuse the pre-processed RGB modality input features and the depth modality input features and then perform an MLP operation to obtain heterogeneous feature separation weights; the heterogeneous feature separation weights are multiplied by the homogeneously aggregated RGB modality features and the depth modality features respectively to filter out abnormal features at the edges of different semantic classes, thereby obtaining the modality-separated RGB modality features and the depth modality features.

[0161] In one embodiment, the multi-scale encoder includes four layers of ground object scene feature extraction modules of different scales and three downsampling layers; the sampling layer is arranged between two layers of ground object scene feature extraction modules of different scales; the multimodal visual feature extraction module of different scales is further used to input the calibrated RGB modal features and depth modal features into the first layer of ground object scene feature extraction module to obtain first-scale visual features; downsample the first-scale visual features and input them into the second layer of ground object scene feature extraction module to obtain second-scale visual features; downsample the second-scale visual features and input them into the third layer of ground object scene feature extraction module to obtain third-scale visual features; downsample the third-scale visual features and input them into the fourth layer of ground object scene feature extraction module to obtain fourth-scale visual features.

[0162] In one embodiment, the ground object scene feature extraction module includes: a multimodal Transformer module and a remote sensing spatial global attention module; the first-scale visual features include first-scale RGB modality and depth modality visual features; the different-scale multimodal visual feature extraction module is further used to input the calibrated RGB modality features and depth modality features into the multimodal Transformer module to obtain multimodal fusion features and geometric depth modality features; wherein the multimodal fusion features and geometric depth modality features are shown in formulas (14) to (18).

[0163] The calibrated RGB modal features are input into the remote sensing space global attention module, the remote sensing space features are decomposed and the feature sequences are reconstructed in eight directions. The state space model is used to capture the contextual information in each feature sequence. Finally, the modeling features in each direction are fused to obtain the remote sensing space global features; the multimodal fusion features, geometric depth modal features and the global features of the remote sensing space are connected in series and linearly projected to obtain a scale of RGB modal and depth modal visual features.

[0164] In one embodiment, the remote sensing space global attention module includes: 1 feature decomposition and reconstruction module, 8 state space models and 1 feature fusion module; the multimodal visual feature extraction module of different scales is also used to input the calibrated RGB modal features into the feature decomposition and reconstruction module, decompose the calibrated RGB modal features into 12 local image blocks for global feature separation, and reconstruct the feature sequences of the separated local blocks along the horizontal, vertical, diagonal and anti-diagonal directions to obtain eight groups of feature sequences reconstructed in different directions; the state space models are used to capture the contextual information in different directions of the eight groups of feature sequences reconstructed in different directions to obtain modeling information in eight directions; the modeling information in eight directions is fused using the feature fusion module to obtain the global features of the remote sensing space.

[0165] In one embodiment, the lightweight decoder in the semantic segmentation module is a lightweight Hamburger decoder in the SegNext network model.

[0166] For the specific definition of the UAV remote sensing RGB-D image semantic segmentation device, please refer to the definition of the UAV remote sensing RGB-D image semantic segmentation method above, which will not be repeated here. The various modules in the above-mentioned UAV remote sensing RGB-D image semantic segmentation device can be implemented in whole or in part by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0167] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor, memory, network interface, display screen and input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a semantic segmentation method for RGB-D images of remote sensing of unmanned aerial vehicles is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a key, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse.

[0168] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0169] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the present steps in the above method embodiment when executing the computer program.

[0170] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0171] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A semantic segmentation method for UAV remote sensing RGB-D images, characterized in that: The method comprises: Obtain a set of RGB images and depth maps of corresponding spatial dimensions taken by drones, and perform preprocessing; The preprocessed RGB modal input features and depth modal input features are calibrated using a multimodal feature guidance enhancement module to obtain calibrated RGB modal features and depth modal features. The multimodal feature guidance enhancement module is used to use the high confidence activation of one modal feature to guide the correction of the features of another modality, thereby enhancing the similarity of features within the same category and the difference between features of different categories. The calibrated RGB modal features and depth modal features are encoded with a multi-scale encoder to obtain multi-modal visual features of different scales; the multi-scale encoder is used to extract the visual texture features and depth geometric features of the objects in the remote sensing scene from the perspective of drone aerial photography from different scales using multiple layers of ground object scene feature extraction modules; the ground object scene feature extraction module is used to realize the information interaction between the RGB modality and the depth modality through the Transformer, and uses a large kernel convolution operation to capture spatial geometric clues from the depth modality to refine the edge detail information of the ground object semantics to obtain multi-modal fusion features and geometric depth modal features. At the same time, the context information in the remote sensing scene is modeled from multiple directions through the state space model, and the modeling information in all directions is fused to obtain the global features of the remote sensing space. Then, the multi-modal fusion features, the geometric depth modal features and the global features of the remote sensing space are connected in series and linearly projected to obtain visual features of one scale. After fusing multimodal visual features of different scales, a lightweight decoder is used to predict the semantic labels of objects to obtain semantic segmentation results. Among them, the multimodal feature guidance enhancement module includes: homogeneous feature aggregation operation and heterogeneous feature separation operation; The pre-processed RGB modal input features and depth modal input features are calibrated using the multi-modal feature guidance enhancement module to calibrate the features of different types of objects. The calibrated RGB modal features and depth modal features are obtained, including: A homogeneous feature aggregation operation is performed on the preprocessed RGB modality input features and the depth modality input features to obtain RGB modality aggregated features and depth modality aggregated features; the homogeneous feature aggregation operation is used to calibrate the features using the different representation capabilities of the RGB image modality and the depth map modality, and to complementary aggregate the multimodal features of objects of the same semantic category to a certain position in space; A heterogeneous feature separation operation is performed on the RGB modal input features and the depth modal input features obtained by preprocessing to obtain modality-separated RGB modal features and depth modal features; the heterogeneous feature separation operation is used to use high-confidence activation of one modality feature to filter abnormal features on the edge of another modality object, thereby achieving heterogeneous feature separation.

2. The method according to claim 1, characterized in that In the multimodal feature-guided enhancement module: The features after RGB modality aggregation and the RGB modality features after modality separation are fused to obtain the calibrated RGB modality features; The features after deep modal aggregation and the deep modal features after modality separation are fused to obtain the calibrated deep modal features.

3. The method according to claim 2, characterized in that Homogeneous feature aggregation is performed on the preprocessed RGB modality input features and depth modality input features to obtain RGB modality aggregated features and depth modality aggregated features, including: The pre-processed RGB modality input features and depth modality input features are concatenated and then convolved to generate RGB modality space gates and depth modality space gates for controlling the feature information flow; The RGB modality space gate and the depth modality space gate are fused and mapped respectively to obtain homogeneous feature aggregation weights of the RGB modality and the depth modality; wherein the sum of the homogeneous feature aggregation weights of the RGB modality and the depth modality at the same position is 1; The RGB modality input features are homogeneously aggregated using the homogeneous feature aggregation weights of the RGB modality to obtain homogeneously aggregated RGB modality features; Homogeneously aggregate the depth modality input features using the depth modality homogeneous feature aggregation weights to obtain homogeneously aggregated depth modality features; Interactively fuse the RGB modal features and depth modal features after homogeneity aggregation to obtain fused features; The fusion features are used to enhance the RGB modality features and the depth modality features after homogeneity aggregation, respectively, to obtain the features after RGB modality aggregation and the features after depth modality aggregation.

4. The method according to claim 3, characterized in that The heterogeneous feature separation operation is performed on the RGB modal input features and the depth modal input features obtained by preprocessing to obtain the RGB modal features and depth modal features after modality separation, including: The pre-processed RGB modality input features and the depth modality input features are fused and then processed through MLP to obtain the heterogeneous feature separation weights; The heterogeneous feature separation weights are multiplied by the RGB modal features and the depth modal features after homogeneity aggregation to filter out abnormal features at the edges of different semantic classes, thereby obtaining the RGB modal features and the depth modal features after modality separation.

5. The method according to claim 1, wherein The multi-scale encoder includes four layers of ground object scene feature extraction modules of different scales and three downsampling layers; wherein the downsampling layer is arranged between two layers of ground object scene feature extraction modules of different scales; The calibrated RGB modal features and depth modal features are encoded using a multi-scale encoder to obtain multi-modal visual features of different scales, including: The calibrated RGB modal features and depth modal features are input into the first-layer ground object scene feature extraction module to obtain the first-scale visual features; Downsampling the first-scale visual features and inputting them into the second-layer ground object scene feature extraction module to obtain second-scale visual features; Downsampling the second-scale visual features and inputting them into the third-layer ground object scene feature extraction module to obtain third-scale visual features; The third-scale visual features are down-sampled and then input into a fourth-layer ground object scene feature extraction module to obtain fourth-scale visual features.

6. The method according to claim 5, characterized in that The ground object scene feature extraction module includes: a multimodal Transformer module and a remote sensing spatial global attention module; the first-scale visual features include first-scale RGB modality and depth modality visual features; The calibrated RGB modal features and depth modal features are input into the first-layer ground object scene feature extraction module to obtain the first-scale visual features, including: The calibrated RGB modal features and depth modal features are input into the multimodal Transformer module to obtain multimodal fusion features and geometric depth modal features; the multimodal fusion features and geometric depth modal features are: in, represents multimodal fusion features, represents a bilinear upsampling operation, represents the softmax operation, Indicates the dimension of the current feature, and Represents the Value vector and Key vector calculated from the RGB features, represents the Query vector, represents the geometric depth modal features, represents a large kernel depthwise convolution operation, represents the Hadamard product operation, and Represent the RGB modality features and depth modality features of the input multimodal Transformer module respectively, [· , ·] represents the cross-modal feature fusion operation, represents an adaptive average pooling operation of fixed k×k size, Represents the MLP network operation; The calibrated RGB modal features are input into the remote sensing spatial global attention module, the remote sensing spatial features are decomposed and feature sequences are reconstructed in several directions, a state space model is used to capture the contextual information in each feature sequence, and finally the modeling features in each direction are fused to obtain the remote sensing spatial global features; The multimodal fusion features, the geometric depth modality features and the global features of the remote sensing space are connected in series and linearly projected to obtain a scale of RGB modality and depth modality visual features.

7. The method according to claim 6, characterized in that The remote sensing spatial global attention module includes: 1 feature decomposition and reconstruction module, 8 state space models and 1 feature fusion module. The calibrated RGB modal features are input into the remote sensing spatial global attention module. The remote sensing spatial features are decomposed and reconstructed into feature sequences in eight directions. The state space model is used to capture the contextual information in each feature sequence. Finally, the modeling features in each direction are fused to obtain the remote sensing spatial global features, including: The calibrated RGB modal features are input into the feature decomposition and reconstruction module, which decomposes the calibrated RGB modal features into 12 local image blocks for global feature separation. The separated local blocks are then reconstructed into feature sequences along the horizontal, vertical, diagonal, and anti-diagonal directions to obtain eight sets of feature sequences reconstructed in different directions. The feature sequences reconstructed in eight different directions are respectively used with the state space model to capture the context information in different directions, thus obtaining the modeling information in eight directions. The modeling information in eight directions is fused using a feature fusion module to obtain the global features of the remote sensing space.

8. The method according to claim 1, characterized in that The lightweight decoder is a lightweight Hamburger decoder in the SegNext network model.

9. A semantic segmentation device for UAV remote sensing RGB-D images, characterized in that: The device comprises: RGB-D image acquisition and preprocessing module, used to acquire a set of RGB images and depth maps of corresponding spatial dimensions taken by drones, and perform preprocessing; The RGB modality and depth modality feature calibration module is used to calibrate the preprocessed RGB modality input features and depth modality input features using the multimodal feature guidance enhancement module to obtain calibrated RGB modality features and depth modality features. The multimodal feature guidance enhancement module is used to use the high confidence activation of one modality feature to guide the correction of the features of another modality, thereby enhancing the similarity of features within the same category and the difference between features of different categories. A multi-scale visual feature extraction module is used to perform multi-scale feature encoding on the calibrated RGB modal features and depth modal features using a multi-scale encoder to obtain multi-scale visual features. The multi-scale encoder is used to extract visual texture features and depth geometric features of objects in remote sensing scenes from the perspective of drone aerial photography from different scales using multiple layers of ground object scene feature extraction modules. The ground object scene feature extraction module is used to realize information interaction between RGB modalities and depth modalities through Transformer, and uses large kernel convolution operations to capture spatial geometric clues from the depth modality to refine the edge detail information of the ground object semantics to obtain multi-modal fusion features and geometric depth modal features. At the same time, the context information in the remote sensing scene is modeled from multiple directions through a state space model, and the modeling information in all directions is fused to obtain global features of the remote sensing space. The multi-modal fusion features, the geometric depth modal features and the global features of the remote sensing space are then connected in series and linearly projected to obtain visual features of one scale. The semantic segmentation module is used to fuse multimodal visual features of different scales and use a lightweight decoder to predict the semantic labels of objects to obtain semantic segmentation results; Among them, the multimodal feature guidance enhancement module includes: homogeneous feature aggregation operation and heterogeneous feature separation operation; the RGB modality and depth modality feature calibration module is also used to apply homogeneous feature aggregation operation to the RGB modality input features and depth modality input features obtained by preprocessing to obtain RGB modality aggregated features and depth modality aggregated features; the homogeneous feature aggregation operation is used to calibrate the features using the different representation capabilities of the RGB image modality and the depth map modality, and to complementarily aggregate the multimodal features of objects of the same semantic category to a certain position in space; the RGB modality input features and depth modality input features obtained by preprocessing are applied with heterogeneous feature separation operation to obtain modality-separated RGB modality features and depth modality features; the heterogeneous feature separation operation is used to use the high confidence activation of one modality feature to filter the abnormal features of the edge of another modality object, thereby realizing the separation of heterogeneous features.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.

Images