Unmanned aerial vehicle image semantic segmentation method based on encoder and context prior learning

Abstract

The embodiment of the invention provides an unmanned aerial vehicle image semantic segmentation method based on an encoder and context prior learning, and the method comprises the steps: carrying out the feature extraction of a first modal image and a multi-scale image through a feature extraction module of a semantic segmentation model, so as to obtain a first multi-scale feature map and a second multi-scale feature map; a feature interaction module in the semantic segmentation model can fuse the first feature map and the second feature map of the same scale to obtain a multi-scale fused feature map. A deep feature fusion module in the semantic segmentation model can perform feature enhancement and fusion on the first feature map with the minimum scale and the fusion feature map to obtain a target fusion feature map; a cascade decoder in the semantic segmentation model can perform hierarchical up-sampling and feature fusion on a target fusion feature map according to partial fusion feature maps in the multi-scale fusion feature maps to obtain a semantic segmentation map. According to the invention, semantic segmentation precision and noise robustness can be improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to a method for semantic segmentation of UAV images based on encoder and contextual prior learning. Background Technology

[0002] With the rapid iteration of sensor technology, remote sensing data is evolving from single-modal to multi-modal collaborative observation. Multimodal images can include high-resolution optical images, multispectral / hyperspectral data, SAR radar images, etc. Multimodal images can provide complementary information on ground feature representation from multiple dimensions, improving the accuracy and dimensionality of ground feature representation.

[0003] Semantic segmentation involves pixel-level labeling of all image pixels, categorizing them into different object classes. To improve the accuracy of semantic segmentation, multimodal images have been widely used in this field.

[0004] Currently, one semantic segmentation approach based on two modalities of images first extracts features from each modality separately, then performs a simple weighted fusion of these features to perform semantic segmentation based on the fused features. However, this approach, by extracting features from each modality separately, may lose usable information, reducing the accuracy of semantic segmentation. Furthermore, performing feature extraction separately for each modality before simple weighted fusion fails to effectively suppress interference from noisy modalities, resulting in poor robustness to noise. Another semantic segmentation approach based on two modalities of images utilizes a Transformer-based model to capture global contextual dependencies (such as spatial relationships between ground features and scene topology) in different modalities through a self-attention mechanism, enabling semantic segmentation based on these global contextual dependencies. However, while this self-attention-based global modeling can capture contextual priors, it easily overlooks local fine structures (such as ground feature edges and small targets), reducing the accuracy of semantic segmentation. Summary of the Invention

[0005] This application provides a method for semantic segmentation of UAV images based on encoder and contextual prior learning, which can improve semantic segmentation accuracy and robustness to noise.

[0006] In a first aspect, embodiments of this application provide a UAV image semantic segmentation method based on encoder and context prior learning, comprising: obtaining a first modal image and a second modal image of the same region collected by a UAV; inputting the first modal image and the second modal image into a semantic segmentation model to obtain a semantic segmentation map; wherein, the semantic segmentation model includes: a feature extraction module, a feature interaction module, a deep feature fusion module, and a cascaded decoder; the feature extraction module is used to perform multi-scale feature extraction on the first modal image to obtain a multi-scale first feature map, and to perform feature extraction on the multi-scale image at corresponding scales to obtain a multi-scale second feature map; the multi-scale image includes the second modal image and a multi-scale third feature map; the... The scale of the second modality image is larger than the scale of the multi-scale third feature map; the feature interaction module is used to fuse the first feature map and the second feature map of the same scale to obtain a multi-scale fused feature map; the multi-scale third feature map is the fused feature map of the multi-scale fused feature map except for the minimum scale fused feature map; the deep feature fusion module is used to perform feature enhancement and fusion on the minimum scale first feature map and the minimum scale fused feature map based on a self-attention mechanism and an adaptive cross-attention mechanism to obtain a target fused feature map; the cascaded decoder is used to perform hierarchical upsampling and feature fusion on the target fused feature map based on the multi-scale third feature map and combined with context prior learning to obtain the semantic segmentation map.

[0007] In one possible implementation, the deep feature fusion module includes a first self-attention layer, an adaptive cross-attention layer, and a second self-attention layer; wherein, the first self-attention layer is used to perform feature enhancement on the first feature map at the smallest scale and the fused feature map at the smallest scale respectively through a self-attention mechanism to obtain a first enhanced feature map and a fused enhanced feature map; the adaptive cross-attention layer is used to fuse the features in the fused enhanced feature map into the first enhanced feature map through an adaptive cross-attention mechanism to obtain a second enhanced feature map, and to fuse the features in the first enhanced feature map into the fused enhanced feature map through an adaptive cross-attention mechanism to obtain a third enhanced feature map; the second self-attention layer is used to perform feature enhancement and fusion on the second enhanced feature map and the third enhanced feature map through a self-attention mechanism to obtain the target fused feature map.

[0008] In one possible implementation, the cascaded decoder includes multiple cascaded decoders and a context prior layer. When the context prior layer is located in the first decoder among the multiple cascaded decoders, it is used to perform constrained enhancement on the target fused feature map by extracting the similarity of intra-class features and the discriminative power of inter-class features to obtain a first constrained enhanced map. The multiple cascaded decoders are used to perform hierarchical upsampling and feature fusion on the first constrained enhanced map based on the multi-scale third feature map to obtain the semantic segmentation map. When the context prior layer is located in the target decoder, the decoder preceding the target decoder in the multiple cascaded decoders... The context prior layer is used to perform hierarchical upsampling and feature fusion on the target fused feature map based on the third feature map at the corresponding scale to obtain an upsampled fused feature map; the context prior layer is used to perform constraint enhancement on the upsampled fused feature map by extracting the similarity of intra-class features and the discriminativeness of inter-class features to obtain a second constraint enhancement map; the target decoder and the decoder located after the target decoder in the plurality of cascaded decoders are used to perform hierarchical upsampling and feature fusion on the second constraint enhancement map based on the third feature map at the corresponding scale to obtain the semantic segmentation map; the target decoder is any one of all decoders in the plurality of cascaded decoders except for the first decoder.

[0009] In one possible implementation, the number of the plurality of cascaded decoders is greater than or equal to 3, and the target decoder is the decoder at the middle position among the plurality of cascaded decoders.

[0010] In one possible implementation, the method further includes: training an original model of the semantic segmentation model to obtain the semantic segmentation model; wherein training the original model of the semantic segmentation model includes: obtaining multiple sample image groups; wherein each sample image group includes a first modality sample image and a second modality sample image, a semantic segmentation sample image, and an affinity image; the semantic segmentation sample image in each sample image group is obtained by semantically annotating the first modality sample image and the second modality sample image in the same sample image group; the affinity image in each sample image group is obtained based on the semantic segmentation sample image in the same sample image group; dividing the multiple sample image groups into multiple training sample sets, and using any one of the multiple training sample sets as a first training sample set; performing a training step, the training step including The original model is trained using first modality sample images and second modality sample images from each sample image group in the first training sample set to obtain multiple predictive semantic segmentation maps and multiple predictive context prior maps that correspond one-to-one. The multiple predictive semantic segmentation maps correspond one-to-one with the multiple sample image groups in the first training sample set. A main loss is determined based on the multiple predictive semantic segmentation maps and the semantic segmentation sample maps in the multiple sample image groups in the first training sample set. A first affinity loss is determined based on the multiple predictive context prior maps and the affinity maps in the multiple sample image groups in the first training sample set. The parameters of the original model are adjusted based on the main loss and the first affinity loss. Any remaining training sample set is selected as the first training sample set, and the training steps are executed until the training stopping condition is met.

[0011] In one possible implementation, determining the main loss based on the plurality of predicted semantic segmentation maps and semantic segmentation sample maps in the plurality of sample image groups in the first training sample set includes: determining the main loss using the following formula:

[0012] in, For the main loss, For the semantic segmentation sample image in the b-th sample image group, perform a one-hot encoding on the c-th category. Let b be the probability that the predicted semantic segmentation map belongs to the c-th category. and Correspondingly, B is the number of sample image groups in the first training sample set, and C is the number of categories.

[0013] In one possible implementation, determining the first affinity loss based on the plurality of predicted context prior maps and affinity maps in the plurality of sample image groups in the first training sample set includes: determining a second affinity loss for each of the predicted context prior maps according to the following formula;

[0014] in, For a second affinity loss of the prediction context prior graph, For binary cross-entropy loss, The in-class predicted value of the prediction context prior graph. The true within-class ratio of the predicted context prior graph. The true inter-class ratio of the predicted context prior graph;

[0015] in, The number of pixels in the prediction context prior map, which is an N-row prediction context prior map. An image with N columns, where n is the pixel number. The pixel value of the nth pixel in the affinity map of the sample image group corresponding to the prediction context prior map in the first training sample set. Let be the pixel value of the nth pixel in the prediction context prior map;

[0016] in, For line numbers, For column number, The first affinity graph in the group of sample images corresponding to the prediction context prior map in the first training sample set. Line 1 The pixel value of the column pixel. For the first in the prediction context prior graph Line 1 The pixel value of the column pixel;

[0017]

[0018] The first affinity loss is determined based on the second affinity loss of each of the predicted context prior graphs.

[0019] Secondly, embodiments of this application provide a UAV image semantic segmentation device based on encoder and context prior learning, comprising: an acquisition module for acquiring a first modal image and a second modal image of the same region collected by a UAV; an input module for inputting the first modal image and the second modal image into a semantic segmentation model to obtain a semantic segmentation map; wherein the semantic segmentation model includes: a feature extraction module, a feature interaction module, a deep feature fusion module, and a cascaded decoder; the feature extraction module is used to perform multi-scale feature extraction on the first modal image to obtain a multi-scale first feature map, and to perform feature extraction on the multi-scale image at corresponding scales to obtain a multi-scale second feature map; the multi-scale image includes the second modal image and the multi-scale first modal image. The three feature maps; the scale of the second modality image is larger than the scale of the multi-scale third feature map; the feature interaction module is used to fuse the first feature map and the second feature map of the same scale to obtain a multi-scale fused feature map; the multi-scale third feature map is the fused feature map in the multi-scale fused feature map except for the minimum scale fused feature map; the deep feature fusion module is used to perform feature enhancement and fusion on the minimum scale first feature map and the minimum scale fused feature map based on the self-attention mechanism and the adaptive cross-attention mechanism to obtain a target fused feature map; the cascaded decoder is used to perform hierarchical upsampling and feature fusion on the target fused feature map according to the multi-scale third feature map and combined with context prior learning to obtain the semantic segmentation map.

[0020] Thirdly, embodiments of this application provide an electronic device, including a memory and a processor; the memory is used to store a computer program; the processor is used to implement, when executing the computer program, the UAV image semantic segmentation method based on encoder and context prior learning as described in the first aspect.

[0021] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the UAV image semantic segmentation method based on encoder and context prior learning as described in the first aspect.

[0022] The beneficial effects of the UAV image semantic segmentation method based on encoder and context prior learning in this application embodiment are: The semantic segmentation model comprises a feature extraction module, a feature interaction module, a deep feature fusion module, and a cascaded decoder. The feature extraction module performs multi-scale feature extraction on the first modality image to obtain a multi-scale first feature map, and performs feature extraction on the multi-scale image at corresponding scales to obtain a multi-scale second feature map. The multi-scale image includes a second modality image and a multi-scale third feature map, with the scale of the second modality image being larger than that of the multi-scale third feature map. The feature interaction module fuses the first and second feature maps at the same scale to obtain a multi-scale fused feature map. The multi-scale third feature map is the fused feature map excluding the smallest-scale fused feature map. The deep feature fusion module enhances and fuses the smallest-scale first feature map and the smallest-scale fused feature map using a self-attention mechanism and an adaptive cross-attention mechanism to obtain a target fused feature map. The cascaded decoder performs hierarchical upsampling and feature fusion on the target fused feature map based on the multi-scale third feature map and contextual prior learning to obtain a semantic segmentation map.

[0023] In other words, the feature extraction module performs multi-scale feature extraction separately for the first modality image, which can fully capture the features of the first modality image at different resolutions (i.e., different scales) (such as small-scale features like texture and edges, and large-scale features like target contours and global layout), avoiding the limitation of single-scale extraction which can only cover some scale features. At the same time, corresponding scale features are extracted for the multi-scale image containing the second modality image and multi-scale third feature maps. The second modality image, as the largest scale input, can provide initial large-scale contextual information, while the third feature maps at each scale carry cross-modal features fused from different scales in the previous stage. This makes the resulting multi-scale second feature map contain the features of the original second modality image and also fuses complementary cross-modal information at different scales. Compared with extracting features from each modality image separately, this reduces the loss of usable information, reduces the difficulty of cross-modal feature alignment, and improves the accuracy of cross-modal feature extraction.

[0024] Furthermore, since feature maps at the same scale are compatible in terms of spatial resolution and feature granularity, the feature interaction module can avoid feature misalignment and semantic deviation caused by scale differences during cross-scale fusion by fusing the first and second feature maps at the same scale. This reduces the difficulty of feature alignment during the fusion process and improves the accuracy of feature alignment, achieving complementary fusion of cross-modal features at the same scale. This preserves the features of each modality and eliminates feature conflicts between modalities, improving not only the accuracy of feature fusion but also the feature consistency and effectiveness of multi-scale fused feature maps. Simultaneously, fusing the first and second feature maps at the same scale can suppress redundant noise interference, improve robustness to noise, and achieve cross-modal feature calibration.

[0025] Furthermore, the multi-scale third feature map originates from the fusion of the first and second feature maps at the same scale by the feature interaction module, and participates in the feature extraction of the corresponding scale as a component of the multi-scale image, forming a progressive process of feature fusion-feature extraction-feature fusion. After the large-scale fusion feature map in the early stage is reused, the verified cross-modal fusion features can be passed to the feature extraction process of the next scale. This allows the second feature map of the subsequent scale to not need to learn cross-modal associations from scratch, but to further extract based on the effective information in the early stage, reducing redundant calculations in feature learning. At the same time, it also strengthens the correlation between features at different scales, so that the multi-scale fusion feature map forms a feature representation from local to global, providing more reliable and accurate feature support for subsequent processing, thereby improving the accuracy and robustness of subsequent processing.

[0026] Furthermore, by employing self-attention and adaptive cross-attention mechanisms to enhance and fuse the first feature map and the smallest-scale fused feature map at the smallest scale, complementary feature fusion and enhancement can be achieved. This also avoids ignoring local fine structures, further improving the accuracy of feature extraction. The cascaded decoder combines the multi-scale third feature map with contextual priors to progressively restore the scale of the target fused feature map through hierarchical upsampling. During the scale restoration process, corresponding scale features are fused to correct local misjudgments, thereby improving the accuracy of the semantic segmentation map and the robustness of semantic segmentation.

[0027] Thus, the semantic segmentation accuracy, robustness to noise, and overall robustness of the semantic segmentation model are all improved. Therefore, using the semantic segmentation model to perform semantic segmentation on both the first and second modal images can improve both semantic segmentation accuracy and robustness to noise. Attached Figure Description

[0028] Figure 1 A flowchart illustrating a UAV image semantic segmentation method based on encoder and context prior learning, provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of the semantic segmentation model provided in the embodiments of this application; Figure 3 This is a schematic diagram illustrating the operation flow of the context prior layer provided in the embodiments of this application; Figure 4 A schematic diagram illustrating the training process of the original model of the semantic segmentation model provided in the embodiments of this application; Figure 5 A schematic diagram of the structure of a UAV image semantic segmentation device based on encoder and context prior learning provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0029] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Although some embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the accompanying drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0030] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0031] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to"; the term "based on" means "at least partially based on"; the term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments"; and the term "optionally" means "optional embodiments". Definitions of other terms will be given in the following description. It should be noted that the concepts of "first," "second," etc., mentioned in this invention are used only to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0032] It should be noted that the terms "a" and "a plurality of" used in this invention are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0033] The names of the messages or information exchanged between the multiple devices in the embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0034] This application provides a method for semantic segmentation of UAV images based on a two-stage encoder and contextual prior learning. This method can be applied to electronic devices, including but not limited to laptops and desktop computers.

[0035] like Figure 1 As shown in the embodiments of this application, a UAV image semantic segmentation method based on encoder and contextual prior learning may include the following steps: 110. Obtain first and second modal images of the same area collected by a drone.

[0036] Modality refers to different forms of image representation or acquisition methods. For example, RGB images, depth maps, visible light images, and infrared images of the same area are images of different modalities of the same area.

[0037] The first modal image and the second modal image have different modes. The first modal image and the second modal image can be any two of the following images of the same region: RGB image, depth map, visible light image and infrared image.

[0038] By acquiring images of different modalities of the same area, information about that area can be complemented and supplemented through these different modalities. For example, in flood monitoring, optical images are easily obscured by clouds, making it difficult to identify flooded areas, while SAR images, although able to penetrate clouds, suffer from speckle noise that affects detail extraction. By fusing the two, SAR can be used to reconstruct the topography of cloud-covered areas, while optical images can be used to refine the classification in cloudless areas, thus improving monitoring accuracy.

[0039] 120. Input the first modality image and the second modality image into the semantic segmentation model to obtain a semantic segmentation map.

[0040] For example, such as Figure 2 As shown, the semantic segmentation model may include: a feature extraction module 210, a feature interaction module 220, a deep feature fusion module 230, and a cascaded decoder 240.

[0041] The feature extraction module 210 can perform multi-scale feature extraction on the first modality image to obtain a multi-scale first feature map, and perform feature extraction on the multi-scale image at corresponding scales to obtain a multi-scale second feature map.

[0042] Multi-scale second feature map and multi-scale first feature map Figure 1 In a one-to-one correspondence, the scales of the corresponding first feature map and second feature map are the same.

[0043] A multi-scale image may include a second modality image and a multi-scale third feature map. The scale of the second modality image is larger than the scale of the multi-scale third feature map.

[0044] The feature interaction module 220 can fuse the first feature map and the second feature map of the same scale to obtain a multi-scale fused feature map.

[0045] The multi-scale third feature map is the fusion feature map other than the minimum scale fusion feature map in the multi-scale fusion feature map.

[0046] The deep feature fusion module 230 can enhance and fuse the first feature map and the smallest scale fusion feature map based on the self-attention mechanism and the adaptive cross-attention mechanism to obtain the target fusion feature map.

[0047] The cascaded decoder 240 can perform hierarchical upsampling and feature fusion on the target fusion feature map based on the multi-scale third feature map and combined with contextual prior learning to obtain a semantic segmentation map.

[0048] Clearly, the feature extraction module 210 performs multi-scale feature extraction separately for the first modality image, which can fully capture the features of the first modality image at different resolutions (i.e., different scales) (such as small-scale features like texture and edges, and large-scale features like target contours and global layout), avoiding the limitation of single-scale extraction which can only cover some scale features. At the same time, corresponding scale features are extracted for the multi-scale image containing the second modality image and the multi-scale third feature map. The second modality image, as the largest scale input, can provide initial large-scale contextual information, while the third feature maps at each scale carry cross-modal features fused from different scales in the early stage. This makes the resulting multi-scale second feature map contain both the features of the original second modality image and the complementary cross-modal information fused from different scales. Compared with extracting features from each modality image separately, this reduces the loss of usable information, reduces the difficulty of cross-modal feature alignment, and improves the accuracy of cross-modal feature extraction.

[0049] Furthermore, since feature maps of the same scale are compatible in terms of spatial resolution and feature granularity, the feature interaction module 220 can avoid feature misalignment and semantic deviation caused by scale differences during cross-scale fusion by fusing the first and second feature maps of the same scale. This reduces the difficulty of feature alignment during the fusion process and improves the accuracy of feature alignment, achieving complementary fusion of cross-modal features at the same scale. This preserves the features of each modality and eliminates feature conflicts between modalities, improving not only the accuracy of feature fusion but also the feature consistency and effectiveness of multi-scale fused feature maps. Simultaneously, by fusing the first and second feature maps of the same scale, redundant noise interference can be suppressed, robustness to noise can be improved, and cross-modal feature calibration can be achieved.

[0050] Furthermore, the multi-scale third feature map comes from the fusion of the first and second feature maps of the same scale by the feature interaction module 220, and participates in the feature extraction of the corresponding scale as a component of the multi-scale image, forming a progressive process of feature fusion-feature extraction-feature fusion. After the large-scale fusion feature map in the early stage is reused, the verified cross-modal fusion features can be passed to the feature extraction process of the next scale, so that the second feature map of the subsequent scale does not need to learn cross-modal association from scratch, but further extracts based on the effective information in the early stage, reducing redundant calculations in feature learning, and strengthening the correlation between features of different scales. This makes the multi-scale fusion feature map form a feature representation from local to global, providing more reliable and accurate feature support for subsequent processing, thereby improving the accuracy and robustness of subsequent processing.

[0051] Furthermore, by employing self-attention and adaptive cross-attention mechanisms to enhance and fuse the first feature map and the smallest-scale fused feature map at the smallest scale, complementary feature fusion and enhancement can be achieved. This also avoids ignoring local fine structures, further improving the accuracy of feature extraction. The cascaded decoder 240 combines the multi-scale third feature map with contextual priors to progressively recover the scale of the target fused feature map through hierarchical upsampling. During the scale recovery process, corresponding scale features are fused to correct local misjudgments, thereby improving the accuracy of the semantic segmentation map and the robustness of semantic segmentation.

[0052] Thus, the semantic segmentation accuracy, robustness to noise, and overall robustness of the semantic segmentation model are all improved. Therefore, using the semantic segmentation model to perform semantic segmentation on both the first and second modal images can improve both semantic segmentation accuracy and robustness to noise.

[0053] It should be noted that the feature extraction module 210, the feature interaction module 220, and the deep feature fusion module 230 can be understood as encoders.

[0054] The following section provides a detailed explanation of each component in the semantic segmentation model.

[0055] like Figure 2 As shown, the feature extraction module 210 may include two feature extractors. One feature extractor is used to perform multi-scale feature extraction on the first modality image to obtain a multi-scale first feature map, and the other feature extractor is used to perform feature extraction on the multi-scale image at the corresponding scale to obtain a multi-scale second feature map.

[0056] For example, each feature extractor consists of a ResNet50 and is divided into 4 convolutional layers. Each convolutional layer downsamples the input image by a factor of 2, and the feature map output from each layer is input to the feature interaction module 220.

[0057] The feature interaction module 220 may include multiple feature interactors (SFFs), each corresponding to a different scale. Each SFF is used to fuse a first feature map and a second feature map at the same scale (i.e., the corresponding scale) to obtain a fused feature map at the corresponding scale. In this way, a multi-scale fused feature map can be obtained through multiple feature interactors (SFFs).

[0058] Specifically, the fusion process of a Feature Filter (SFF) can be as follows: The SFF performs global average pooling on the first and second feature maps of the same scale, and then performs convolution on them to obtain the channel weights of the first and second feature maps of the same scale. The first feature map of the same scale is multiplied by its channel weights to obtain the first channel weighted feature map. The second feature map of the same scale is multiplied by its channel weights to obtain the second channel weighted feature map. The first channel weighted feature map and the second channel weighted feature map are concatenated based on the channel dimension to obtain the fused feature map of the same scale.

[0059] The deep feature fusion module 230 may include a first self-attention layer SA1 (SA Layer 1), an adaptive cross-attention layer AMBA (i.e., Ada-MBA Layer), and a second self-attention layer SA2 (SA Layer 2).

[0060] The first self-attention layer SA1 can perform feature enhancement on the first feature map at the smallest scale and the fused feature map at the smallest scale through a self-attention mechanism to obtain the first enhanced feature map and the fused enhanced feature map.

[0061] The Adaptive Cross-Attention Layer (AMBA) can fuse features from the fused feature enhancement map into the first enhanced feature map through an adaptive cross-attention mechanism to obtain the second enhanced feature map, and fuse features from the first enhanced feature map into the fused feature enhancement map through an adaptive cross-attention mechanism to obtain the third enhanced feature map.

[0062] The second self-attention layer SA2 can enhance and fuse the second and third enhanced feature maps through a self-attention mechanism to obtain the target fused feature map.

[0063] For example, the deep feature fusion module 230 may include 12 ViT blocks. The first three ViT blocks perform self-attention within a modality, the middle six ViT blocks perform self-attention between modalities, and the last three ViT blocks perform self-attention within a modality. That is, the first self-attention layer SA1 is composed of the first three ViT blocks, the adaptive cross-attention layer AMBA can be composed of the middle six ViT blocks, and the second self-attention layer SA2 can be composed of the last three ViT blocks.

[0064] The cascaded decoder 240 may include multiple cascaded decoders and context prior layers.

[0065] The following section describes the specific implementation process of multiple cascaded decoders and context prior layers, based on their different settings.

[0066] The first approach involves setting the context prior layer in the first decoder of a multi-cascaded decoder system. The contextual prior layer can enhance the target fusion feature map by extracting the similarity of intra-class features and the discriminativeness of inter-class features, so as to obtain the first constrained enhancement map.

[0067] Multiple cascaded decoders can perform hierarchical upsampling and feature fusion on the first constraint enhancement map based on the multi-scale third feature map to obtain a semantic segmentation map.

[0068] The second approach involves setting the context prior layer within the target decoder, which is any one of the multiple cascaded decoders except for the first decoder. In a multi-cascaded decoder, the decoder preceding the target decoder can perform hierarchical upsampling and feature fusion on the target fused feature map based on the third feature map of the corresponding scale to obtain an upsampled fused feature map.

[0069] The contextual prior layer can perform constrained enhancement on the upsampled fused feature map by extracting the similarity of intra-class features and the discriminativeness of inter-class features to obtain a second constrained enhanced map.

[0070] The target decoder and the decoders following the target decoder in the multiple cascaded decoders can perform hierarchical upsampling and feature fusion on the second constraint enhancement map based on the third feature map of the corresponding scale to obtain the semantic segmentation map.

[0071] In some embodiments, the number of multiple cascaded decoders is greater than or equal to 3. Based on this, in order to balance computational efficiency, the target decoder is the decoder in the middle position among the multiple cascaded decoders.

[0072] Specifically, when the number of cascaded decoders is odd, the target decoder is the decoder in the middle position among the cascaded decoders. When the number of cascaded decoders is even, the target decoder is either one of the two decoders in the middle position among the cascaded decoders.

[0073] Below, in conjunction with Figure 2 The semantic segmentation model is described in detail below, with each feature extractor in feature extraction module 210 comprising 4 convolutional layers, feature interaction module 220 comprising 4 feature interactors (SFF), first self-attention layer SA1 comprising 3 ViT blocks, adaptive cross-attention layer AMBA comprising 6 ViT blocks, second self-attention layer SA2 comprising 3 ViT blocks, and cascaded decoder 240 comprising 3 decoders and a context prior layer, with the context prior layer set in the second decoder.

[0074] The first modality image X and the second modality image Y are respectively input into the corresponding feature extractors, wherein, , s represents the first mode, t represents the second mode, and H and W represent the height and width of the two modal images, respectively. The number of channels for the first modality image X. This represents the number of channels in the second modality image Y. For example, RGB images typically have three channels, while SAR data has only one.

[0075] Each convolutional layer in each feature extractor obtains a feature map at the corresponding scale by downsampling the input image by a factor of 2 and extracting features. Specifically, the four convolutional layers in a feature extractor perform four consecutive steps of downsampling and feature extraction on the first modality image X to obtain four first feature maps at four different scales. These four first feature maps are then input into their respective feature interaction units (SFFs). In another feature extractor, the first convolutional layer first downsamples and extracts features from the second modality image Y by a factor of 2, obtaining a second feature map at the corresponding scale. This second feature map is then input to the corresponding feature interaction unit (SFF). The SFF fuses the received first and second feature maps at the same scale to obtain a fused feature map at the corresponding scale. This fused feature map is then input to the second convolutional layer, which downsamples and extracts features from the fused feature map at the same scale by a factor of 2, obtaining a second feature map at the corresponding scale. This second feature map is then input to the corresponding feature interaction unit (SFF), allowing the SFF to fuse the first and second feature maps at the same scale to obtain a fused feature map at the corresponding scale. Figure 2 Continue repeating the above process until the minimum scale fused feature map is obtained.

[0076] It should be noted that, after the above feature extraction and fusion, the output size of the a-th convolutional layer of an encoder (i.e., a feature extractor) is... Feature map (i.e., the first feature map), the output size of the a-th convolutional layer of another encoder (i.e., another feature extractor) is... Feature map (i.e., the second feature map), where 'a' takes the value of an integer from 1 to 4, and the fused feature map output by the Feature Interactor (SFF) can be represented as: .

[0077] For easier post-processing, optionally, the first feature map at the smallest scale can also be used. and minimum scale fusion feature map The input channel is transformed into Then the first feature map at the smallest scale Flattened to size sequence fused feature maps at the smallest scale Flattened to size sequence ,in The sequence length is then used. Next, specific position embeddings are added. and The first feature map at the smallest scale after transformation is used to preserve positional information. and the transformed minimum scale fused feature map The input is fed into the first self-attention layer SA1. It should be noted that the smallest scale first feature map can also be directly used. and minimum scale fusion feature map The input is fed into the first self-attention layer SA1. Below, the transformed minimum-scale first feature map is used... and the transformed minimum scale fused feature map The following explanation uses the input to the first self-attention layer SA1 as an example.

[0078] The first self-attention layer SA1 uses a self-attention mechanism to process the transformed minimum-scale first feature map. and the transformed minimum scale fused feature map The specific implementation process of performing feature enhancement to obtain the first enhanced feature map and the fused enhanced feature map can be shown below: Each ViT block in the first self-attention layer SA1 includes two self-attention modules, two multilayer perceptron modules, and a layer normalization operator. Given... and The input, the first self-attention layer SA1, uses a multi-head self-attention mechanism to derive the global relation for each modality. The value of 'a' is 1, 2, or 3. The output of the 'a'-th ViT block of the first self-attention layer SA1 can be written as... and :in, and The calculation process can be summarized as follows:

[0079]

[0080]

[0081]

[0082] Where LN is the layer normalization, For self-attention, It is a multilayer perceptron.

[0083] and The result of the multi-head self-attention output in the a-th VIT block of SA1 after residual connection is given, where, Used to preserve key semantic information of the first modality (dominant modality) image. Used to optimize cross-modal fusion features and enhance the complementary information that has already been fused.

[0084] Clearly, after feature enhancement by the first self-attention layer SA1, the first enhanced feature map is output. and fusion feature enhancement map .

[0085] Furthermore, an adaptive cross-attention layer (AMBA) is used to fuse multimodal features in the abstract semantic space with rich contextual information. In this feature fusion stage, each ViT block in the AMBA layer may include a self-attention layer (CA) computed across modalities and a self-attention layer (SA) computed within a single modality to understand the correlation between the two modalities. The output of the a-th ViT block of the AMBA layer can be represented as... and , .in, and The calculation process can be summarized as follows:

[0086] set up ,but:

[0087]

[0088] In the a-th ViT block of the adaptive cross-attention layer AMBA and These are the outputs calculated using an adaptive cross-attention mechanism, representing the main modality (first modality) and cross-modal fusion features, respectively. and yes and The result after concatenating the input features with residuals. LN is a layer normalization. For adaptive cross-attention layer, It is a multilayer perceptron.

[0089] in, The specific implementation process can be described as follows: In the Adaptive Cross-Attention Layer (AMBA), multimodal features are input. and Divided into Equal segments, respectively using and express, ,in This refers to the number of attention heads. Two sets of matrices representing multimodal information. and Linear projection can be used separately. and Calculations are performed. Then, the self-attention layer (SA) information computed within a single modality is simultaneously derived for both modalities. Self-attention layer CA information computed across modalities SA uses and CA is used to calculate intramodal information. and Calculate intermodal information. This process enables feature extraction and fusion within a ViT. The specific calculation formula is as follows:

[0090]

[0091]

[0092]

[0093]

[0094]

[0095] in, Here are the normalization parameters, where and These are the softmax function and the matrix transpose operator, respectively. The above equation achieves guided fusion of the two modes by exchanging some matrices. These are the characteristic sequences of the main mode (first mode). Through linear projection matrix The resulting matrix consists of query, key, and value elements. Similarly. and These are feature maps obtained by calculating the main modality feature sequence and the fused feature sequence through intramodal self-attention (SA), respectively. and These are feature maps obtained by cross-modal cross-attention (CA) from the main modality feature sequence and the fused feature sequence, respectively.

[0096] Next, the following adaptive mechanism is used to fuse SA and CA.

[0097]

[0098]

[0099] in, , , and These are learnable weighting coefficients used to balance the contributions of SA and CA.

[0100] Clearly, after the adaptive cross-attention layer (AMBA), the second enhanced feature map is output. and the third enhanced feature map .

[0101] The feature enhancement and fusion process of the second self-attention layer SA2 can be described as follows: The second self-attention layer SA2 uses the same principle as the first self-attention layer SA1 to enhance the second feature map. and the third enhanced feature map Feature enhancement is performed separately, and the two enhanced feature maps are fused based on the channel dimension to obtain the target fused feature map. The target fused feature map is output from the second self-attention layer SA2. .

[0102] Then, the target feature map is fused. The output is sent to the cascade decoder 240. The cascade decoder 240 calculates the multi-scale third feature map (i.e., , , ) and combine contextual prior learning to fuse feature maps of the target. Hierarchical upsampling and feature fusion are performed to obtain a semantic segmentation map.

[0103] Specifically, the first decoder 241 fuses the target feature map Reconstructing a two-dimensional sequence into a size of The three-dimensional tensor, in which This is the number of input channels for the first decoder. Each decoder block is skipped with its corresponding Feature Filter (SFF), progressively restoring the feature map size to its original value. Each decoder block includes an upsampling operator, a convolutional (Conv) layer, and a ReLU layer. A context prior layer 244 is embedded in the second decoder 242 for calculating the affinity loss; the entire context prior layer 244 does not change the dimension of the input features.

[0104] The first decoder 241 will reshape the target fused feature map of the size. After upsampling (e.g., scale 512, H / 16, W / 16), it is compared with the third feature map of the same scale. The fusion process is performed, and then upsampling is performed to obtain the first upsampled feature map (e.g., scale 256, H / 8, W / 8).

[0105] like Figure 3 As shown, the context prior layer 244 first upsamples the first upsampled fused feature map to obtain feature map X. After information aggregation, feature map X is obtained as feature map X. Feature map After the prior mapping learning module, a context prior map is obtained. The prior mapping learning module may include... Convolutional layers (conv), batch normalization (BN) layers, and the Sigmaod function are used to learn prior mappings. Prior mapping It is a 0-1 matrix, where pixels within a class are 1 and the rest are 0. Inner classes can be accessed through... Let's calculate it. Y can be understood as a contextual prior map. In this formula, the prior mapping... It can adaptively select in-class pixels as the in-class context for each pixel in the feature map. (Compared to prior mapping) The corresponding one is inter-class prior mapping. Similarly, it can adaptively select inter-class pixels. Under the supervision of affinity loss, relationships between intra-class and inter-class pixels can be constructed, thereby enhancing the model's understanding of details. Finally, the original features are concatenated with the two contextual features to output the final prediction result. (i.e., the second constraint enhancement graph), where, , Instructions for splicing. Second constraint reinforcement diagram. The scale is 128, H / 4, W / 4.

[0106] The second decoder 242 will enhance the second constraint map. Compared with the third feature map of the same scale After fusion, upsampling is performed to obtain a second upsampled fused feature map with dimensions of 64, H / 2, W / 2.

[0107] The third decoder 243 combines the second upsampled fused feature map with the third feature map of the same scale. After fusion, upsampling is performed to obtain a semantic segmentation map with dimensions 32, H, W.

[0108] In some embodiments, the method provided in this application may further include: training the original model of the semantic segmentation model to obtain the semantic segmentation model. The structure of the original model is the same as that of the semantic segmentation model described above, and the operating principles of each structure in the original model are the same as those of each structure in the semantic segmentation model described above, which will not be repeated here.

[0109] like Figure 4 As shown, the process of training the original model of the semantic segmentation model can be described as follows: 410. Obtain multiple sample image groups.

[0110] Each sample image group includes a first modality sample image, a second modality sample image, a semantic segmentation sample image, and an affinity map. The semantic segmentation sample image in each sample image group is obtained by semantically annotating the first modality sample image and the second modality sample image in the same sample image group.

[0111] The affinity map in each sample image group is obtained based on the semantic segmentation sample maps in the same sample image group. Specifically, for example... Figure 2 As shown, an affinity map can be obtained by downsampling the semantic segmentation sample map. The size of the affinity map is related to the prediction context prior map (which can be understood as...). Figure 3 The size is the same as that of the Context PriorMap in the context.

[0112] In this embodiment, multiple image groups can be acquired by an aircraft. Each image group includes a first image and a second image, which are images of different modalities acquired from the same area. These multiple image groups can be used as multiple sample image groups, with the first and second images in each image group designated as the first modal sample image and the second modal sample image, respectively.

[0113] In some embodiments, to improve data diversity, data augmentation can be performed on multiple image groups, defining the augmented image groups as multiple sample image groups. Data augmentation can be achieved through random cropping, vertical flipping, and horizontal mirroring. Random cropping involves extracting a local region the size of a window from the original image to enhance the model's adaptability to spatial distribution. Vertical flipping and horizontal mirroring are both triggered with a 50% probability and are performed through index reversal operations. Different modalities within the same group are synchronously subjected to the same geometric transformation to ensure spatial alignment consistency among multimodal data.

[0114] 420. Divide multiple sample image groups into multiple training sample sets, and use any one of the multiple training sample sets as the first training sample set.

[0115] 430. Perform a training step, which may include: training the original model with first modality sample images and second modality sample images in each sample image group of the first training sample set to obtain multiple one-to-one corresponding predictive semantic segmentation maps and multiple predictive context prior maps.

[0116] Among them, multiple predicted semantic segmentation maps correspond one-to-one with multiple sample image groups in the first training sample set.

[0117] 440. Determine the main loss based on multiple predicted semantic segmentation maps and semantic segmentation sample maps in multiple sample image groups in the first training sample set.

[0118] For example, the main loss can be determined using the following formula:

[0119] in, Main loss, For the semantic segmentation sample image in the b-th sample image group, perform a one-hot encoding on the c-th category. Let b be the probability that the predicted semantic segmentation map belongs to the c-th category. and Correspondingly, B represents the number of sample image groups in the first training sample set, and C represents the number of categories.

[0120] 450. Determine the first affinity loss based on multiple prediction context prior maps and affinity maps in multiple sample image groups in the first training sample set.

[0121] For example, the specific implementation process of 450 can be as follows: First, the second affinity loss for each prediction context prior graph is determined according to the following formula;

[0122] in, For a prediction context prior graph, the second affinity loss is used. For binary cross-entropy loss, To predict the in-class predictions of the context prior graph, To predict the true in-class rate of the context prior graph, To predict the true inter-class ratio of the context prior graph.

[0123]

[0124] in, To predict the number of pixels in the context prior map, the prediction context prior map is an N-row map. An image with N columns, where n is the pixel number. Let be the pixel value of the nth pixel in the affinity map of the sample image group corresponding to the prediction context prior map in the first training sample set. To predict the pixel value of the nth pixel in the context prior map.

[0125]

[0126] in, For line numbers, For column number, The first affinity graph in the group of sample images corresponding to the prediction context prior map in the first training sample set. Line 1 The pixel value of the column pixel. To predict the first in the context prior graph Line 1 The pixel value of the column pixel.

[0127]

[0128]

[0129] Then, the first affinity loss is determined based on the second affinity loss of each prediction context prior graph.

[0130] For example, the first affinity loss can be determined by taking the mean of the second affinity loss.

[0131] 460. Adjust the parameters of the original model based on the main loss and the first affinity loss.

[0132] 470. Select any of the remaining training sample sets as the first training sample set and execute the training steps until the training stopping condition is met.

[0133] Training termination conditions include, but are not limited to: the number of training iterations reaching the preset number of iterations, and the validation set performance converging.

[0134] After training, the optimal model is saved and designated as the semantic segmentation model. The optimal model is then tested; specifically, it is loaded and its performance is evaluated using a test set or other datasets. The stored optimal model parameters are loaded, and the semantic segmentation map and context prior matrix of the test set samples are calculated through forward propagation. Multi-dimensional performance evaluation is performed using the F1 coefficient and average intersection-union ratio (AUC).

[0135] To verify the effectiveness of the semantic segmentation model provided in this application, six types of image data common to the Vaihingen and WHU-OPT-SAR datasets were selected as different domains, including residential, farmland, forest, industrial, parking lot and river, and domain adaptation was performed between each pair of domains.

[0136] Experiments on the WHU-OPT-SAR dataset in Table 1 demonstrate that the proposed algorithm (i.e., the "ours" in Table 1) significantly outperforms existing algorithms, exceeding Unet, DeepLabv3+, CFNet, MCANet, and FtransUNet by 5.16%, 3.59%, 3.23%, 1.55%, and 0.59%, respectively. The average intersection-over-union ratio (IoU) is similar to that of BCLnet. In terms of classification accuracy for individual categories, the model achieves the following accuracies: farmland 82.6%, city 54.4%, village 61.9%, water body 65.3%, forest 91.4%, road 58.6%, and others 10.0%. This confirms that the network successfully integrates shallow and deep features by extracting complementary information from auxiliary modalities and effectively obtaining robust representations, demonstrating the effectiveness of the invention.

[0137] Table 1: Experimental results on the WHU-OPT-SAR dataset (unit: %)

[0138] The ablation experiments on the Vaihingen dataset in Table 2 demonstrate that feature fusion is crucial in both shallow and deep feature extraction stages. The Feature Interactor (SFF) can learn robust representations of key features, such as shape, boundary, color, and texture. The learned shallow features are then utilized in the Adaptive Cross-Attention Layer (AMBA) to learn more complex relationships.

[0139] Table 2: Ablation Experiment

[0140] In summary, this application provides a semantic segmentation method for UAV images based on encoder and contextual prior learning. This method can accurately capture the features of ground object edges and small targets while enhancing long-range dependency modeling capabilities.

[0141] like Figure 5 As shown, this application embodiment also provides a UAV image semantic segmentation device based on encoder and context prior learning. The device 500 may include: The acquisition module 510 is used to acquire a first modal image and a second modal image of the same area collected by a drone; The input module 520 is used to input the first modal image and the second modal image into the semantic segmentation model to obtain a semantic segmentation map; The semantic segmentation model includes: a feature extraction module, a feature interaction module, a deep feature fusion module, and a cascaded decoder; The feature extraction module is used to perform multi-scale feature extraction on the first modality image to obtain a multi-scale first feature map, and to perform feature extraction on the multi-scale image at corresponding scales to obtain a multi-scale second feature map. The multi-scale image includes a second modality image and a multi-scale third feature map; the scale of the second modality image is larger than the scale of the multi-scale third feature map; The feature interaction module is used to fuse the first feature map and the second feature map of the same scale to obtain a multi-scale fused feature map. The multi-scale third feature map is the fusion feature map in the multi-scale fusion feature map excluding the minimum scale fusion feature map; The deep feature fusion module is used to enhance and fuse the first feature map at the smallest scale and the fused feature map at the smallest scale based on the self-attention mechanism and the adaptive cross-attention mechanism to obtain the target fused feature map. The cascaded decoder is used to perform hierarchical upsampling and feature fusion on the target fusion feature map based on the multi-scale third feature map and combined with contextual prior learning, so as to obtain the semantic segmentation map.

[0142] The specific implementation process and effects of this device can be found in the relevant description above, and will not be repeated here.

[0143] like Figure 6 As shown, an electronic device 600 provided in this embodiment of the invention may include a processor 610 and a memory 620; the memory 620 is used to store a computer program; the processor 610 is used to implement the UAV image semantic segmentation method based on encoder and context prior learning as described above when the computer program is executed.

[0144] This invention provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the UAV image semantic segmentation method based on encoder and context prior learning as described above.

[0145] The present invention will now describe an electronic device 600 that can serve as a server or client of the present invention, which is an example of a hardware device that can be applied to various aspects of the present invention. Electronic device 600 is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0146] Electronic device 600 includes a computing unit that can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) or a computer program loaded from a storage unit into random access memory (RAM). The RAM may also store various programs and data required for device operation. The computing unit, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0147] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc. In this application, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention according to actual needs. Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units can be implemented in hardware or as software functional units.

[0148] While the present invention has been disclosed above, its scope of protection is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention, and all such changes and modifications will fall within the scope of protection of the present invention.

Claims

1. A method for unmanned aerial vehicle image semantic segmentation based on encoder and context prior learning, characterized in that, The method comprises: obtaining first and second modality images of the same region collected by a UAV; inputting the first and second modality images into a semantic segmentation model to obtain a semantic segmentation map; wherein the semantic segmentation model comprises a feature extraction module, a feature interaction module, a deep feature fusion module and a cascaded decoder; the feature extraction module is configured to perform multi-scale feature extraction on the first modality image to obtain a multi-scale first feature map, and perform feature extraction on the multi-scale image at a corresponding scale to obtain a multi-scale second feature map; the multi-scale image comprises the second modality image and a multi-scale third feature map; the scale of the second modality image is larger than the scale of the multi-scale third feature map; the feature interaction module is configured to fuse the first and second feature maps of the same scale to obtain a multi-scale fused feature map; the multi-scale third feature map is a fused feature map in the multi-scale fused feature map other than a smallest scale fused feature map; the deep feature fusion module is configured to perform feature enhancement and fusion on the smallest scale first feature map and the smallest scale fused feature map based on a self-attention mechanism and an adaptive cross-attention mechanism to obtain a target fused feature map; the cascaded decoder is configured to perform hierarchical upsampling and feature fusion on the target fused feature map based on the multi-scale third feature map and in combination with context prior learning to obtain the semantic segmentation map.

2. The method of claim 1, wherein, the deep feature fusion module comprises a first self-attention layer, an adaptive cross-attention layer and a second self-attention layer; wherein the first self-attention layer is configured to perform feature enhancement on the smallest scale first feature map and the smallest scale fused feature map respectively by a self-attention mechanism to obtain a first enhanced feature map and a fused feature enhanced map; the adaptive cross-attention layer is configured to fuse features in the fused feature enhanced map into the first enhanced feature map by an adaptive cross-attention mechanism to obtain a second enhanced feature map, and fuse features in the first enhanced feature map into the fused feature enhanced map by an adaptive cross-attention mechanism to obtain a third enhanced feature map; the second self-attention layer is configured to perform feature enhancement and fusion on the second and third enhanced feature maps by a self-attention mechanism to obtain the target fused feature map.

3. The method of claim 1, wherein, the cascaded decoder comprises a plurality of cascaded decoders and a context prior layer; in the case where the context prior layer is arranged in the first decoder in the plurality of cascaded decoders, the context prior layer is configured to perform constraint enhancement on the target fused feature map by extracting the similarity of intra-class features and the discriminability of inter-class features to obtain a first constraint enhanced map; the plurality of cascaded decoders are configured to perform hierarchical upsampling and feature fusion on the first constraint enhanced map based on the multi-scale third feature map to obtain the semantic segmentation map. In the case that the context prior layer is arranged in the target decoder, a decoder before the target decoder in the plurality of cascaded decoders is configured to perform hierarchical upsampling and feature fusion on the target fusion feature map according to the third feature map of the corresponding scale to obtain an upsampled fusion feature map; the context prior layer is configured to perform constraint enhancement on the upsampled fusion feature map by extracting similarity of intra-class features and discriminability of inter-class features to obtain a second constraint enhanced map; the target decoder and a decoder after the target decoder in the plurality of cascaded decoders are configured to perform hierarchical upsampling and feature fusion on the second constraint enhanced map according to the third feature map of the corresponding scale to obtain the semantic segmentation map; the target decoder is any one of all decoders in the plurality of cascaded decoders except the first decoder.

4. The method of claim 1, wherein, The number of the plurality of cascaded decoders is greater than or equal to 3, and the target decoder is a decoder at a middle position in the plurality of cascaded decoders.

5. The method of claim 1, wherein, The method further comprises: training an original model of the semantic segmentation model to obtain the semantic segmentation model; wherein the training of the original model of the semantic segmentation model comprises: obtaining a plurality of sample image groups; wherein each of the sample image groups comprises a first modality sample image and a second modality sample image, a semantic segmentation sample image and an affinity graph; the semantic segmentation sample image in each of the sample image groups is obtained by performing semantic annotation on the first modality sample image and the second modality sample image in the same sample image group; the affinity graph in each of the sample image groups is obtained based on the semantic segmentation sample image in the same sample image group; dividing the plurality of sample image groups into a plurality of training sample sets, and taking any one of the plurality of training sample sets as a first training sample set; performing a training step, the training step comprising: training the original model by the first modality sample image and the second modality sample image in each sample image group in the first training sample set to obtain a one-to-one corresponding plurality of predicted semantic segmentation images and a plurality of predicted context prior images; wherein the plurality of predicted semantic segmentation images correspond one-to-one to the plurality of sample image groups in the first training sample set; determining a main loss according to the plurality of predicted semantic segmentation images and the semantic segmentation sample images in the plurality of sample image groups in the first training sample set; determining a first affinity loss according to the plurality of predicted context prior images and the affinity graphs in the plurality of sample image groups in the first training sample set; adjusting parameters of the original model according to the main loss and the first affinity loss; determining any one of the remaining training sample sets as the first training sample set, and performing the training step until a training stop condition is met.

6. The method of claim 5, wherein, The determination of the main loss according to the plurality of predicted semantic segmentation images and the semantic segmentation sample images in the plurality of sample image groups in the first training sample set comprises: the main loss is determined by the following formula: wherein, is the main loss, is a one-hot encoding of a semantic segmentation sample image in the b-th sample image group on the c-th class, is a probability that the b-th predicted semantic segmentation map belongs to the c-th class, and correspondingly, B is the number of sample image groups in the first training sample set, and C is the number of classes.

7. The method of claim 5, wherein, The determining a first affinity loss according to the plurality of predicted context prior maps and affinity maps in a plurality of sample image groups in the first training sample set comprises: determining a second affinity loss of each of the predicted context prior maps according to the following formula; wherein, is a second affinity loss for one of the predicted context prior maps, is a binary cross-entropy loss, is an intra-class prediction value for the predicted context prior map, is a true intra-class rate for the predicted context prior map, is a true inter-class rate for the predicted context prior map. wherein, N is the number of pixels of the predicted contextual prior map, the predicted contextual prior map being an N row N column image, n being an index of a pixel, is a pixel value of an n-th pixel in an affinity map in a group of sample images corresponding to the predicted contextual prior map in the first training sample set, is a pixel value of an n-th pixel in the predicted contextual prior map; in, For line numbers, For column number, The first affinity graph in the group of sample images corresponding to the prediction context prior map in the first training sample set. Line 1 The pixel value of the column pixel. For the first in the prediction context prior graph Line 1 The pixel value of the column pixel; determining the first affinity loss according to the second affinity loss of each of the predicted context prior maps.

8. An apparatus for unmanned aerial vehicle image semantic segmentation based on encoder and context prior learning, characterized in that, Comprise: An obtaining module is configured to obtain first modality images and second modality images of a same region collected by a UAV; An input module is configured to input the first modality images and the second modality images into a semantic segmentation model to obtain a semantic segmentation map; The semantic segmentation model comprises a feature extraction module, a feature interaction module, a deep feature fusion module, and a cascaded decoder; The feature extraction module is configured to perform multi-scale feature extraction on the first modality images to obtain multi-scale first feature maps, and perform feature extraction of corresponding scales on multi-scale images respectively to obtain multi-scale second feature maps; The multi-scale images comprise the second modality images and multi-scale third feature maps; the scale of the second modality images is larger than the scale of the multi-scale third feature maps; The feature interaction module is configured to fuse the first feature maps and the second feature maps of the same scale to obtain multi-scale fused feature maps; The multi-scale third feature maps are fused feature maps in the multi-scale fused feature maps except for a minimum scale fused feature map; The deep feature fusion module is configured to perform feature enhancement and fusion on the first feature maps of the minimum scale and the minimum scale fused feature maps based on a self-attention mechanism and an adaptive cross-attention mechanism to obtain a target fused feature map; The cascaded decoder is configured to perform hierarchical upsampling and feature fusion on the target fused feature map according to the multi-scale third feature maps and in combination with context prior learning to obtain the semantic segmentation map.

9. An electronic device, comprising: Comprise a memory and a processor; The memory is configured to store a computer program; The processor is configured to implement the method of claim 1-7 when executing the computer program.

10. A computer-readable storage medium, characterized in that, The storage medium has a computer program stored thereon, and when the computer program is executed by a processor, the method of claim 1-7 based on the encoder and the context prior learning of the UAV image semantic segmentation is realized.

Images