Remote sensing image instance segmentation method and model based on dynamic convolutional pyramid module

Abstract

The present invention discloses a remote sensing image instance segmentation method and model based on a dynamic convolutional pyramid, belonging to the technical field of remote sensing image processing, and the method comprises the following steps: constructing a remote sensing image instance segmentation model based on a dynamic convolutional pyramid module, wherein the remote sensing image instance segmentation model comprises a feature extraction network, a feature fusion network, and a detection head based on a dynamic convolutional pyramid module connected in sequence; training the remote sensing image instance segmentation model; and performing instance segmentation on the remote sensing image to be tested using the trained remote sensing image instance segmentation model. Through the above method and model, the present invention can more effectively extract features at different scales, and solve the problem of insufficient positioning accuracy of the target mask area in the remote sensing image instance segmentation model.

Description

Technical Field

[0001] The invention belongs to the technical field of remote sensing image processing, and in particular relates to a remote sensing image instance segmentation method and model based on a dynamic convolutional pyramid. Background Art

[0002] Remote sensing images have broad application prospects in the fields of national defense security, environmental monitoring, urban planning, etc. Remote sensing image instance segmentation technology has always been one of the research focuses in the field of remote sensing. Remote sensing images have the characteristics of top-down views, arbitrary rotation angles of targets, and arbitrary directions, which makes instance segmentation challenging. The current mainstream instance segmentation model based on convolutional neural networks is relatively insufficient in the refinement of target features in remote sensing images and cannot accurately locate the target mask area. This is because the conventional convolution operator is limited by the size of the convolution kernel, and local features can only interact with each other in a relatively narrow receptive field, and the perception of the spatial structure information of the features is poor.

[0003] Therefore, it is necessary to design a targeted feature processing module to help the instance segmentation model dynamically perceive the target features of remote sensing images and explore higher-dimensional potential regional feature space representations. Summary of the invention

[0004] The present invention proposes an innovative solution to the problem of insufficient positioning accuracy of target mask regions in remote sensing image instance segmentation models.

[0005] In order to achieve the above object, the present invention adopts the following technical solution:

[0006] On the one hand, the present invention discloses a remote sensing image instance segmentation method based on a dynamic convolutional pyramid, comprising the following steps:

[0007] Constructing a remote sensing image instance segmentation model based on a dynamic convolutional pyramid module, wherein the remote sensing image instance segmentation model includes a feature extraction network, a feature fusion network, and a detection head based on the dynamic convolutional pyramid module that are connected in sequence;

[0008] Training the remote sensing image instance segmentation model;

[0009] The trained remote sensing image instance segmentation model is used to perform instance segmentation on the remote sensing image to be tested.

[0010] Furthermore, the trained remote sensing image instance segmentation model is used to perform instance segmentation on the remote sensing image to be tested, which specifically includes the following steps:

[0011] Using the feature extraction network to extract features from the input remote sensing image to be tested, and obtaining a feature map corresponding to the remote sensing image to be tested;

[0012] The feature fusion network is used to fuse the feature map to obtain the fused feature map;

[0013] The fused feature maps are decoupled using a detection head based on a dynamic convolutional pyramid module.

[0014] Furthermore, the fused feature map is decoupled using a detection head based on a dynamic convolutional pyramid module, which specifically includes the following steps:

[0015] The convolution module is used to process the shape and channel dimension of the remote sensing image features in the fused feature map to obtain a first feature map;

[0016] Use the dynamic convolutional pyramid module to extract features from the fused feature map to obtain a second feature map;

[0017] Concatenate the first feature map and the second feature map to obtain a concatenated feature map;

[0018] The prediction function is used to predict the spliced ​​feature map to obtain the classification prediction information, position prediction information and mask prediction information of the remote sensing image to be tested.

[0019] Furthermore, the dynamic convolutional pyramid module is used to extract features from the fused feature map to obtain a second feature map, which specifically includes the following steps:

[0020] Using a channel division module, the fused feature map is evenly divided into two sub-feature maps along the channel dimension, including a first sub-feature map and a second sub-feature map;

[0021] Using the first dynamic convolution module, the normalization module and the activation function in sequence to perform feature extraction calculation on the first sub-feature map, and obtain the output feature map of the first dynamic convolution module;

[0022] Using the first basic convolution module, the normalization module and the activation function in sequence to perform feature extraction calculation on the second sub-feature map, and obtain the output feature map of the first basic convolution module;

[0023] Perform feature splicing on the feature map output by the first dynamic convolution module and the feature map output by the first basic convolution module to obtain a channel-divided splicing feature map;

[0024] The channel partitioning and splicing feature map is processed in sequence using the self-attention module and the first multi-layer perceptron module to obtain the output feature map of the self-attention module;

[0025] The channel partition and splicing feature map is processed in sequence using a second dynamic convolution module, a normalization module, and an activation function to obtain an output feature map of the second dynamic convolution module;

[0026] Using the third dynamic convolution module, the normalization module and the activation function in sequence to perform feature extraction calculation on the fused feature map, and obtain the output feature map of the third dynamic convolution module;

[0027] The second basic convolution module, the normalization module and the activation function are used in sequence to perform feature extraction calculation on the fused feature map to obtain the output feature map of the second basic convolution module;

[0028] Perform feature splicing on the feature map output by the third dynamic convolution module and the feature map output by the second basic convolution module to obtain a fused feature splicing feature map;

[0029] The CA attention module is used to process the fused feature concatenation feature map to obtain the CA attention module output feature map;

[0030] The feature map output by the self-attention module, the feature map output by the second dynamic convolution module, and the feature map output by the CA attention module are feature spliced, and the spliced ​​feature map is input into the second multi-layer perceptron module to output the second feature map.

[0031] Furthermore, in the step of training the remote sensing image instance segmentation model, the model loss function includes a focus loss function, a rotation frame intersection-over-union loss function, and a cross entropy loss function;

[0032] Among them, the focal loss function is used for the classification prediction branch of the remote sensing image instance segmentation task; the rotated box intersection-over-union loss function is used to calculate the distance between the predicted box and the target true value label corresponding to the input remote sensing image; the cross entropy loss function is used to calculate the distance between the predicted mask and the target true value label corresponding to the input remote sensing image.

[0033] On the other hand, the present invention discloses a remote sensing image instance segmentation model based on a dynamic convolutional pyramid, comprising a feature extraction network, a feature fusion network, and a detection head based on a dynamic convolutional pyramid module connected in sequence; wherein the feature extraction network is used to extract features from an input remote sensing image to be tested, and obtain a feature map corresponding to the remote sensing image to be tested;

[0034] The feature fusion network is used to fuse the feature maps to obtain the fused feature maps;

[0035] The detection head based on the dynamic convolutional pyramid module is used to decouple the fused feature map.

[0036] Preferably, the feature extraction network includes a plurality of stacked basic convolution modules, and the operation layer calculation steps of each basic convolution module for the remote sensing image to be measured include basic convolution operation, normalization operation and activation function operation in sequence.

[0037] Preferably, the detection head based on the dynamic convolution pyramid module includes a convolution module, a dynamic convolution pyramid module, a feature addition module and a prediction module;

[0038] The convolution module is used to perform shape and channel dimension processing on the remote sensing image features in the fused feature map to obtain a first feature map;

[0039] The dynamic convolution pyramid module is used to extract features from the fused feature map to obtain a second feature map;

[0040] The feature addition module is used to splice the first feature map and the second feature map to obtain a spliced ​​feature map;

[0041] The prediction module is used to predict the spliced ​​feature map to obtain classification prediction information, position prediction information and mask prediction information of the remote sensing image to be tested.

[0042] Preferably, the dynamic convolution pyramid module includes a channel division module, a first dynamic convolution module, a first basic convolution module, a second dynamic convolution module, a self-attention module, a first multi-layer perceptron module, a third dynamic convolution module, a second basic convolution module, a CA attention module and a second multi-layer perceptron module; the data execution process of the dynamic convolution pyramid module is as follows:

[0043] Using a channel division module, the fused feature map is evenly divided into two sub-feature maps along the channel dimension, including a first sub-feature map and a second sub-feature map;

[0044] Using the first dynamic convolution module, the normalization module and the activation function in sequence to perform feature extraction calculation on the first sub-feature map, and obtain the output feature map of the first dynamic convolution module;

[0045] Using the first basic convolution module, the normalization module and the activation function in sequence to perform feature extraction calculation on the second sub-feature map, and obtain the output feature map of the first basic convolution module;

[0046] Perform feature splicing on the feature map output by the first dynamic convolution module and the feature map output by the first basic convolution module to obtain a channel-divided splicing feature map;

[0047] The channel partitioning and splicing feature map is processed in sequence using the self-attention module and the first multi-layer perceptron module to obtain the output feature map of the self-attention module;

[0048] The channel partition and splicing feature map is processed in sequence using a second dynamic convolution module, a normalization module, and an activation function to obtain an output feature map of the second dynamic convolution module;

[0049] Using the third dynamic convolution module, the normalization module and the activation function in sequence to perform feature extraction calculation on the fused feature map, and obtain the output feature map of the third dynamic convolution module;

[0050] The second basic convolution module, the normalization module and the activation function are used in sequence to perform feature extraction calculation on the fused feature map to obtain the output feature map of the second basic convolution module;

[0051] Perform feature splicing on the feature map output by the third dynamic convolution module and the feature map output by the second basic convolution module to obtain a fused feature splicing feature map;

[0052] The CA attention module is used to process the fused feature concatenation feature map to obtain the CA attention module output feature map;

[0053] The feature map output by the self-attention module, the feature map output by the second dynamic convolution module, and the feature map output by the CA attention module are feature spliced, and the spliced ​​feature map is input into the second multi-layer perceptron module to output the second feature map.

[0054] Preferably, in the dynamic convolution pyramid module, the first basic convolution module includes a 3*3 convolution module, and the second basic convolution module includes a 5*5 convolution module.

[0055] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a remote sensing image instance segmentation method and model based on a dynamic convolutional pyramid, which has the following beneficial effects:

[0056] 1. The instance segmentation method combined with the dynamic convolutional pyramid module proposed in the present invention can more effectively extract features at different scales and capture the multi-scale information of the target more comprehensively than the traditional method, which is crucial for correctly identifying and locating potential target objects in remote sensing images.

[0057] 2. The detection head based on the dynamic convolutional pyramid module proposed in the present invention can fully explore the feature representation of remote sensing images in different dimensions in the feature decoupling stage, and refine the features to achieve accurate positioning of potential target areas, effectively improving the problem of insufficient mask positioning accuracy of current instance segmentation methods, thereby achieving accurate remote sensing image instance segmentation task recognition.

[0058] 3. The dynamic convolutional pyramid module proposed in the present invention can perform refined feature extraction from shallow to deep, which can ensure that the model learns richer and more abstract feature representations, thereby helping the instance segmentation model to understand the image content more accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0060] Figure 1 The present invention provides a schematic diagram of the overall process of the remote sensing image instance segmentation method.

[0061] Figure 2 Schematic diagram of the network structure of the dynamic convolutional pyramid module provided by the present invention.

[0062] Figure 3 This is a schematic diagram of the network structure of the remote sensing image instance segmentation model provided by the present invention. DETAILED DESCRIPTION

[0063] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] The present invention introduces a remote sensing image instance segmentation method and model based on dynamic convolutional pyramid, aiming to solve the problem of insufficient positioning accuracy of target mask area in remote sensing image instance segmentation model.

[0065] Embodiment 1

[0066] Embodiment 1 discloses a remote sensing image instance segmentation method based on dynamic convolutional pyramid, such as Figure 1 As shown, the following steps are included:

[0067] Construct a remote sensing image instance segmentation model based on a dynamic convolutional pyramid module, wherein the remote sensing image instance segmentation model includes a feature extraction network, a feature fusion network, and a detection head based on a dynamic convolutional pyramid module that are connected in sequence;

[0068] Train the instance segmentation model for remote sensing images;

[0069] The trained remote sensing image instance segmentation model is used to perform instance segmentation on the remote sensing image to be tested.

[0070] In the above embodiment, the remote sensing image instance segmentation model based on the dynamic convolutional pyramid module can be built through the following steps.

[0071] Step 1: Build a dynamic convolutional pyramid module;

[0072] Step 2: Build a detection head combined with a dynamic convolutional pyramid module and its corresponding instance segmentation model;

[0073] Then the constructed instance segmentation model is trained and tested for remote sensing image instance segmentation tasks.

[0074] The dynamic convolutional pyramid module structure is as follows Figure 2 As shown in the figure, its core modules include channel division module, dynamic convolution, 3*3 convolution, 5*5 convolution, normalization module, activation function, multi-layer perceptron, self-attention module, CA attention module and feature splicing module. The dynamic convolution pyramid module includes sub-feature map processing branch and complete feature map processing branch, aiming to comprehensively utilize the feature expression information under different receptive fields.

[0075] The dynamic convolutional pyramid module processes the input feature map as follows:

[0076] First, the input feature map is divided into 1 / 2 sub-feature maps along the channel dimension by the channel division module, so as to form a different operation branch from the original feature map in the subsequent calculation to extract the difference feature information under different dimensions. The calculation process can be formally expressed as: ;in, is the input feature map, is the sub-feature map obtained after channel division, and its number of channels is Slice is a channel partitioning module, which is used to divide the feature map along the channel dimension. Furthermore, dynamic convolution and basic convolution are used to extract features from feature maps of different branches. The calculation process can be formally expressed as:

[0077] ;

[0078] ;

[0079] in, represents the dynamic convolution module, Represents a 3*3 convolution module, which is used as a basic feature extraction unit to calculate the feature map. BN represents a feature normalization module, which is used to process the feature map data to normalize it to the standard feature space, thereby eliminating the adverse effects caused by singular sample data. Reul represents an activation function, which is used to introduce nonlinear information into the feature map to help the model distinguish the features of the potential target area. Feature map describing the output , .

[0080] In addition to the sub-feature map branch, the complete feature map will also be processed by dynamic convolution and basic convolution to extract supplementary feature information. The calculation process can be formally expressed as:

[0081] ;

[0082] ;

[0083] in, represents a 5*5 convolution module, The feature map representing the output is , Furthermore, the feature concatenation module is used to combine the feature maps of different branches to obtain the feature information under different receptive fields. The calculation process can be formally expressed as:

[0084] ;

[0085] ;

[0086] Represents the feature map obtained after feature concatenation. Further, the self-attention module and dynamic convolution module are used to perform sub-feature map The calculation process can be formally expressed as follows:

[0087] ;

[0088] ;

[0089] Among them, SA represents the self-attention module, Mlp represents the multi-layer perceptron module, and F sa , F dc They are the feature maps output after being extracted by the self-attention module and the dynamic convolution module. The dynamic convolution module is the same as described above. The SA self-attention module is described in detail below.

[0090] The calculation process of the SA self-attention module includes word vector embedding and Softmax calculation, etc. Taking the feature map as input as an example, the calculation process can be formally expressed as: , where PE represents the word vector embedding process, which transforms the feature map F into i Expanded into N word vectors of size 1×1 and number of channels C . Furthermore, the self-attention module will The calculation process can be formally expressed as:

[0091] ;

[0092] Among them, Q represents the query matrix, K represents the key-value matrix, and V represents the value matrix. The three are composed of the input word vector Obtained through linear mapping. d represents the dimension of its feature map. The self-attention mechanism maps the input vector to the word vector probability of a given output to obtain a prediction of the content of each sequence. Furthermore, the feature normalization module is used to process the outputs of different branches to obtain a feature map of an adaptive shape. Its calculation process can be formally expressed as:

[0093] ;

[0094] in, Represents the feature map after feature normalization.

[0095] , Mlp represents a multi-layer perceptron module, and the number of neurons in its input layer is The number of channels is C, the number of neurons in the hidden layer is 20×C, and the number of neurons in the output layer is 2×C.

[0096] Furthermore, the CA attention module is used to The calculation process can be formally expressed as:

[0097] ;

[0098] Among them, CA stands for channel-spatial attention module, which aims to mine the correlation characteristics of feature maps in spatial dimension and channel dimension. CA uses sigmoid activation function and point convolution to convert the input feature map into The spatial feature information is encoded in the horizontal and vertical directions of the intermediate feature map to obtain the transformation weight matrix , and then expand the output and to obtain the feature information of the output feature map at the (i, j) spatial position , the calculation formula is as follows:

[0099] ;

[0100] ;

[0101] ;

[0102] in, Represents the feature information of the input feature map at the (i, j) spatial position, and Respectively represent the transformation weight information of the transformation weight matrix in the horizontal and vertical directions, Represents the sigmoid activation function operation, PointConv represents the point convolution processing operation, and Z C Represents the spatial feature representation matrix, H×W represents the feature latitude in the horizontal and vertical directions, It represents the aggregation of features along the two dimensions H and W.

[0103] The spatial attention module CA realizes the integration of spatial information by encoding spatial information through channel attention. After the calculation and transformation of the spatial feature matrix and the original feature map, it can effectively extract high-value multi-scale feature information and filter out the noise interference caused by the size change of the fused feature map. Finally, it uses the form of matrix multiplication to calculate the attention mechanism of the input data, thereby improving the value of the feature information and reducing the interference of noise.

[0104] Finally, the feature concatenation module is used to integrate the feature maps from different computation branches. The calculation formula is as follows: .

[0105] Multilayer Perceptron module is used to adjust the feature map The shape of , in order to facilitate subsequent calculations, the calculation process can be formally expressed as: , where Mlp represents a multi-layer perceptron module, and the number of neurons in its input layer is F d The number of channels is 3C, the number of neurons in the hidden layer is 20×C, and the number of neurons in the output layer is 2×C. is the final output feature map.

[0106] Traditional instance segmentation models based on convolutional neural networks mostly use backbone feature extraction network-feature fusion network-detection head as the basic model architecture. The remote sensing image instance segmentation model combined with the dynamic convolutional pyramid module constructed in the present invention follows this paradigm and introduces the dynamic convolutional pyramid module to optimize the detection head. The overall structure of the remote sensing image instance segmentation model based on the dynamic convolutional pyramid module is as follows: Figure 3 shown.

[0107] The following will describe in detail how to construct the remote sensing image instance segmentation model based on the dynamic convolutional pyramid module in this paper.

[0108] First, the remote sensing image and its label information are input into the model through the data loader. The calculation process of the data loader is expressed as: ;in, represents an input remote sensing image with a size of H×W and a channel number of 3; labels is the annotation information of the target in the image, including location and category information; f dataloder It is a data loader function used to load data and enhance data;

[0109] Then the backbone network is responsible for feature extraction of the input image. The process of feature extraction network extracting features from the input image imgs is expressed as:

[0110] ;

[0111] in, Represents the feature map obtained after feature extraction, with a size of H i ×W i , and there is C i channels; f backbone It is a feature extraction network, which obtains the feature information of the input remote sensing image by stacking convolution modules in layers. Typical backbone feature extraction networks include: ResNet, VGG, EfficientNet, etc. The method shown in this embodiment takes the basic convolution module as an example, but it should be noted that through simple model replacement, the method in this article can be adapted to a variety of general backbone feature extraction networks.

[0112] In this embodiment, the feature extraction network f backbone It is composed of multiple basic convolution modules stacked together. First, the basic convolution module is introduced. Each basic convolution module contains a basic convolution operator (Conv), a normalization module (BatchNormalization) and an activation function (Relu). The basic convolution operator calculates the channel information of the input remote sensing image at a pixel level to achieve the purpose of feature extraction. The normalization module normalizes the calculated channel results to the standard range to facilitate subsequent calculations while avoiding abnormal problems in gradient optimization. The activation function activates and inhibits the feature matrix of the potential target area to help the model distinguish between foreground and background information. The calculation process of the operation layer of each module can be formally expressed as:

[0113] ;

[0114] Conv represents the basic convolution operation, BN is the normalization module, which keeps the number of channels for receiving and outputting features unchanged, imgs is the input original remote sensing image, and F 1 The output feature map is obtained after feature extraction, and Reul is the activation function responsible for activating the potential target area of ​​the output feature map. Conv is defined as a convolution module containing a basic convolution operator, a normalization module, and an activation function. Therefore, the deep feature extraction process of a deep neural network can be formally expressed as: ; Conv n represents the convolutional module of the nth layer, F n It is a multi-level feature map extracted by ordinary convolution operator. Based on this, we can obtain n feature maps of remote sensing images after deep feature extraction.

[0115] The feature fusion network is responsible for receiving the feature maps of the backbone network for fusion to obtain the multi-scale characteristic information of the potential target.

[0116] The basic model of this embodiment is selected from the backbone feature extraction network f backbone Extracted feature map The feature fusion network is passed to the feature fusion network for feature fusion. The calculation process of the feature fusion network is expressed as: , where f Neck The feature fusion network is used for feature fusion, which first receives the feature map of the backbone feature extraction network .

[0117] Furthermore, the shape of the feature map is changed by the convolution operator to achieve fusion to obtain the fused feature map , the calculation process of the feature fusion network can be expressed as:

[0118] ;

[0119] ;

[0120] ;

[0121] Among them, Concat represents the feature concatenation operator, Conv represents the convolution operator, and Resize represents the feature reshaping operator.

[0122] Finally, the detection head is used to decouple the fused feature map. The detection head improved based on the dynamic convolution pyramid module of the present invention includes a dynamic convolution pyramid module, a feature addition module, and classification, position and mask prediction branches. The calculation process of the detection head improved based on the dynamic convolution pyramid module is described in detail below.

[0123] First, the convolution module is used to receive the input remote sensing image features and process them into appropriate shapes and channel dimensions for subsequent calculations. The dynamic convolution pyramid module is used to obtain high-quality feature information of remote sensing images to assist prediction. Feature splicing is used to integrate feature maps from different branches, and its calculation process is expressed as: ; Wherein, DP represents the dynamic convolutional pyramid module described in step 1, Represents the output feature map. Add represents the feature addition module. Then, the prediction function is used to predict the location and category information of the potential target area of ​​the feature map and output the prediction result. The calculation process of the detection head can be expressed as:

[0124] ;

[0125] ;

[0126] ;

[0127] in, Classification information obtained for prediction; Location information obtained for prediction; is the mask information obtained by prediction; P i The feature map output by the feature fusion network; f cls is the classification function, f bbox is the rectangular box prediction function, f mask is the mask prediction function.

[0128] At this point, the remote sensing image instance segmentation model combined with the dynamic convolutional pyramid module has been completed.

[0129] After building the instance segmentation model based on the dynamic convolutional pyramid module, the network is trained for the remote sensing image instance segmentation task using focal loss (FocalLoss), rotated box intersection over union loss (Rotated IoU) and cross entropy loss (CrossEntropyLoss). The network is trained for the remote sensing image instance segmentation task, which is formally expressed as:

[0130] ;

[0131] Among them, FocallLoss represents the focal loss, which is used for the classification prediction branch of the remote sensing image instance segmentation task. RotatedIoULoss represents the rotated box IoU loss function, which is used to calculate the distance between the predicted box and the target true value label corresponding to the input remote sensing image. CrossEntropyLoss represents the cross entropy loss, which is used to calculate the distance between the predicted mask and the target true value label corresponding to the input remote sensing image. Loss represents the total loss used for network training. When the network is trained until the loss no longer decreases, it means that the network training has reached stability, the training process is completed, and the trained instance segmentation model combined with the dynamic convolutional pyramid module is obtained.

[0132] Furthermore, the trained instance segmentation model combined with the dynamic convolutional pyramid module is used to test the remote sensing image to be tested, which is formally expressed as: =DinsDetector( ); where DinsDetector represents the trained instance segmentation model combined with the dynamic convolutional pyramid module. and They respectively represent the remote sensing image to be tested and the corresponding instance segmentation result.

[0133] Embodiment 2

[0134] Based on the first embodiment, the second embodiment discloses a remote sensing image instance segmentation model based on a dynamic convolutional pyramid. The overall structure of the model is as follows: Figure 3 As shown, it includes a feature extraction network, a feature fusion network, and a detection head based on a dynamic convolutional pyramid module, which are connected in sequence; wherein the feature extraction network is used to extract features of the input remote sensing image to be measured, and obtain a feature map corresponding to the remote sensing image to be measured; the feature fusion network is used to fuse the feature map to obtain the fused feature map; the detection head based on the dynamic convolutional pyramid module is used to decouple the fused feature map.

[0135] Specifically, the feature extraction network includes a plurality of stacked basic convolution modules, and the operation layer calculation steps of each basic convolution module for the remote sensing image to be measured include basic convolution operation, normalization operation and activation function operation in sequence. For a more specific operation process, refer to Example 1.

[0136] The detection head based on the dynamic convolution pyramid module includes a convolution module, a dynamic convolution pyramid module, a feature addition module and a prediction module; wherein the convolution module is used to perform shape and channel dimension processing on the remote sensing image features in the fused feature map to obtain a first feature map; the dynamic convolution pyramid module is used to perform feature extraction on the fused feature map to obtain a second feature map; the feature addition module is used to splice the first feature map and the second feature map to obtain a spliced ​​feature map; the prediction module is used to predict the spliced ​​feature map to obtain classification prediction information, position prediction information and mask prediction information of the remote sensing image to be tested.

[0137] Dynamic convolutional pyramid module structure reference Figure 2 , including a channel division module, a first dynamic convolution module, a first basic convolution module, a second dynamic convolution module, a self-attention module, a first multi-layer perceptron module, a third dynamic convolution module, a second basic convolution module, a CA attention module and a second multi-layer perceptron module.

[0138] The data execution process of the dynamic convolutional pyramid module is as follows.

[0139] Using a channel division module, the fused feature map is evenly divided into two sub-feature maps along the channel dimension, including a first sub-feature map and a second sub-feature map;

[0140] The first dynamic convolution module, the normalization module and the activation function are used in sequence to perform feature extraction calculation on the first sub-feature map to obtain the output feature map of the first dynamic convolution module; the first basic convolution module, the normalization module and the activation function are used in sequence to perform feature extraction calculation on the second sub-feature map to obtain the output feature map of the first basic convolution module; the output feature map of the first dynamic convolution module and the output feature map of the first basic convolution module are feature spliced ​​to obtain the channel division splicing feature map.

[0141] The channel partition and splicing feature map is processed in sequence using the self-attention module and the first multi-layer perceptron module to obtain the output feature map of the self-attention module; the channel partition and splicing feature map is processed in sequence using the second dynamic convolution module, the normalization module and the activation function to obtain the output feature map of the second dynamic convolution module.

[0142] The third dynamic convolution module, the normalization module and the activation function are used in sequence to perform feature extraction calculations on the fused feature map to obtain the output feature map of the third dynamic convolution module; the second basic convolution module, the normalization module and the activation function are used in sequence to perform feature extraction calculations on the fused feature map to obtain the output feature map of the second basic convolution module; the output feature map of the third dynamic convolution module and the output feature map of the second basic convolution module are feature spliced ​​to obtain a fused feature splicing feature map; the CA attention module is used to process the fused feature splicing feature map to obtain the output feature map of the CA attention module.

[0143] The feature map output by the self-attention module, the feature map output by the second dynamic convolution module, and the feature map output by the CA attention module are feature spliced, and the spliced ​​feature map is input into the second multi-layer perceptron module to output the second feature map.

[0144] It should be pointed out that in the embodiments of the present invention, the first, second, third, fourth, etc. are only used to distinguish structural modules at different positions, and the numbers themselves do not have limiting meanings.

[0145] In summary, the present invention proposes an innovative solution to the problem of insufficient positioning accuracy of target mask regions in remote sensing image instance segmentation models. By designing an instance segmentation model based on a dynamic convolutional pyramid module to promote the feature quality in the feature decoupling stage, the multi-dimensional features of typical remote sensing targets are refined, and the model is further guided to accurately locate the mask region of potential targets, effectively improving the performance of remote sensing image instance segmentation. This innovation provides a more reliable and efficient solution for the field of remote sensing image instance segmentation, and provides strong support for future remote sensing application fields.

[0146] The present invention constructs a feature refinement processing pyramid module based on dynamic convolution to achieve progressive feature extraction from shallow to deep, which is of great significance for optimizing the structure of instance segmentation model. This method can extract remote sensing image features at multiple scales, enabling the model to learn richer and more abstract feature representations, helping the model to understand the image content more accurately, and effectively detect and segment targets at different scales. At the same time, through the flexible parameter adjustment of dynamic convolution, it can save computing resources and improve model efficiency.

[0147] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0148] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A remote sensing image instance segmentation method based on dynamic convolutional pyramid, characterized in that: The following steps are involved: Constructing a remote sensing image instance segmentation model based on a dynamic convolutional pyramid module, wherein the remote sensing image instance segmentation model includes a feature extraction network, a feature fusion network, and a detection head based on the dynamic convolutional pyramid module that are connected in sequence; Training the remote sensing image instance segmentation model; Using the trained remote sensing image instance segmentation model to perform instance segmentation on the remote sensing image to be tested, specifically including: using a feature extraction network to extract features from the input remote sensing image to be tested, and obtaining a feature map corresponding to the remote sensing image to be tested; The feature fusion network is used to fuse the feature map to obtain the fused feature map; The fused feature maps are decoupled using a detection head based on a dynamic convolutional pyramid module; Among them, decoupling the fused feature map using the detection head based on the dynamic convolutional pyramid module specifically includes the following steps: The convolution module is used to process the shape and channel dimension of the remote sensing image features in the fused feature map to obtain a first feature map; Use the dynamic convolutional pyramid module to extract features from the fused feature map to obtain a second feature map; Concatenate the first feature map and the second feature map to obtain a concatenated feature map; The spliced ​​feature map is predicted using the prediction function to obtain the classification prediction information, position prediction information and mask prediction information of the remote sensing image to be tested; The dynamic convolutional pyramid module is used to extract features from the fused feature map to obtain the second feature map, which specifically includes: Using a channel division module, the fused feature map is evenly divided into two sub-feature maps along the channel dimension, including a first sub-feature map and a second sub-feature map; Using the first dynamic convolution module, the normalization module and the activation function in sequence to perform feature extraction calculation on the first sub-feature map, and obtain the output feature map of the first dynamic convolution module; Using the first basic convolution module, the normalization module and the activation function in sequence to perform feature extraction calculation on the second sub-feature map, and obtain the output feature map of the first basic convolution module; Perform feature splicing on the feature map output by the first dynamic convolution module and the feature map output by the first basic convolution module to obtain a channel-divided splicing feature map; The channel partitioning and splicing feature map is processed in sequence using the self-attention module and the first multi-layer perceptron module to obtain the output feature map of the self-attention module; The channel partition and splicing feature map is processed in sequence using a second dynamic convolution module, a normalization module, and an activation function to obtain an output feature map of the second dynamic convolution module; Using the third dynamic convolution module, the normalization module and the activation function in sequence to perform feature extraction calculation on the fused feature map, and obtain the output feature map of the third dynamic convolution module; The second basic convolution module, the normalization module and the activation function are used in sequence to perform feature extraction calculation on the fused feature map to obtain the output feature map of the second basic convolution module; Perform feature splicing on the feature map output by the third dynamic convolution module and the feature map output by the second basic convolution module to obtain a fused feature splicing feature map; The CA attention module is used to process the fused feature concatenation feature map to obtain the CA attention module output feature map; The feature map output by the self-attention module, the feature map output by the second dynamic convolution module, and the feature map output by the CA attention module are feature spliced, and the spliced ​​feature map is input into the second multi-layer perceptron module to output the second feature map.

2. The remote sensing image instance segmentation method based on dynamic convolutional pyramid according to claim 1, characterized in that: In the step of training the remote sensing image instance segmentation model, the model loss function includes a focus loss function, a rotation frame intersection-over-union loss function, and a cross entropy loss function; Among them, the focal loss function is used for the classification prediction branch of the remote sensing image instance segmentation task; the rotated box intersection-over-union loss function is used to calculate the distance between the predicted box and the target true value label corresponding to the input remote sensing image; the cross entropy loss function is used to calculate the distance between the predicted mask and the target true value label corresponding to the input remote sensing image.

3. A remote sensing image instance segmentation model based on dynamic convolutional pyramid, characterized in that: It includes a feature extraction network, a feature fusion network, and a detection head based on a dynamic convolutional pyramid module connected in sequence; wherein the feature extraction network is used to extract features from an input remote sensing image to be tested, and obtain a feature map corresponding to the remote sensing image to be tested; The feature fusion network is used to fuse the feature maps to obtain the fused feature maps; The detection head based on the dynamic convolutional pyramid module is used to decouple the fused feature map; The detection head based on the dynamic convolutional pyramid module includes a convolution module, a dynamic convolutional pyramid module, a feature addition module and a prediction module; The convolution module is used to perform shape and channel dimension processing on the remote sensing image features in the fused feature map to obtain a first feature map; The dynamic convolution pyramid module is used to extract features from the fused feature map to obtain a second feature map; The feature addition module is used to splice the first feature map and the second feature map to obtain a spliced ​​feature map; The prediction module is used to predict the spliced ​​feature map to obtain classification prediction information, position prediction information and mask prediction information of the remote sensing image to be tested; The dynamic convolution pyramid module includes a channel division module, a first dynamic convolution module, a first basic convolution module, a second dynamic convolution module, a self-attention module, a first multi-layer perceptron module, a third dynamic convolution module, a second basic convolution module, a CA attention module and a second multi-layer perceptron module; the data execution process of the dynamic convolution pyramid module is as follows: Using a channel division module, the fused feature map is evenly divided into two sub-feature maps along the channel dimension, including a first sub-feature map and a second sub-feature map; Using the first dynamic convolution module, the normalization module and the activation function in sequence to perform feature extraction calculation on the first sub-feature map, and obtain the output feature map of the first dynamic convolution module; Using the first basic convolution module, the normalization module and the activation function in sequence to perform feature extraction calculation on the second sub-feature map, and obtain the output feature map of the first basic convolution module; Perform feature splicing on the feature map output by the first dynamic convolution module and the feature map output by the first basic convolution module to obtain a channel-divided splicing feature map; The channel partitioning and splicing feature map is processed in sequence using the self-attention module and the first multi-layer perceptron module to obtain the output feature map of the self-attention module; The channel partition and splicing feature map is processed in sequence using a second dynamic convolution module, a normalization module, and an activation function to obtain an output feature map of the second dynamic convolution module; Using the third dynamic convolution module, the normalization module and the activation function in sequence to perform feature extraction calculation on the fused feature map, and obtain the output feature map of the third dynamic convolution module; The second basic convolution module, the normalization module and the activation function are used in sequence to perform feature extraction calculation on the fused feature map to obtain the output feature map of the second basic convolution module; Perform feature splicing on the feature map output by the third dynamic convolution module and the feature map output by the second basic convolution module to obtain a fused feature splicing feature map; The CA attention module is used to process the fused feature concatenation feature map to obtain the CA attention module output feature map; The feature map output by the self-attention module, the feature map output by the second dynamic convolution module, and the feature map output by the CA attention module are feature spliced, and the spliced ​​feature map is input into the second multi-layer perceptron module to output the second feature map.

4. The remote sensing image instance segmentation model based on dynamic convolutional pyramid according to claim 3, characterized in that: The feature extraction network includes a plurality of stacked basic convolution modules, and the operation layer calculation steps of each basic convolution module for the remote sensing image to be measured include basic convolution operation, normalization operation and activation function operation in sequence.

5. The remote sensing image instance segmentation model based on dynamic convolutional pyramid according to claim 3, characterized in that: In the dynamic convolution pyramid module, the first basic convolution module includes a 3*3 convolution module, and the second basic convolution module includes a 5*5 convolution module.

Images