Remote sensing image panoramic segmentation network and panoramic segmentation method

Abstract

The invention provides a remote sensing image panoramic segmentation network and a panoramic segmentation method. Relates to the field of image detection, and the remote sensing image panoramic segmentation network comprises an image feature extraction backbone network used for obtaining remote sensing image features based on a remote sensing image to be subjected to panoramic segmentation; the pre-trained remote sensing characteristic knowledge extraction module is used for obtaining remote sensing characteristic knowledge characteristics based on the remote sensing image to be subjected to panoramic segmentation; the registration and fusion module is used for performing registration and fusion on the remote sensing image features and the remote sensing feature knowledge features to obtain remote sensing fusion features; the instance segmentation branch is used for performing instance segmentation on the remote sensing fusion features to obtain remote sensing instance segmentation results; the semantic segmentation branch is used for respectively carrying out semantic segmentation on the remote sensing fusion features to obtain a remote sensing semantic segmentation result; and the segmentation result combination module is used for combining the remote sensing instance segmentation result and the remote sensing semantic segmentation result to obtain a remote sensing image panoramic segmentation result.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image detection, in particular to a remote sensing image panoramic segmentation network and a panoramic segmentation method. BACKGROUND

[0002] Satellite remote sensing technology has gradually shown the advantages of fast response, wide coverage and rich information due to its macro coverage and efficient observation characteristics, and under the development of interpretation technology. Especially under the support of intelligent interpretation technology, it has realized the intelligent and automatic classification of ground objects, target judgment and other remote sensing information acquisition. Remote sensing image panoramic segmentation is committed to segmenting countable foreground targets (aircraft, ships, vehicles, etc.) and uncountable background ground objects (water surface, road, vegetation, etc.) at the same time, and dividing the instances of countable foreground targets. Remote sensing image panoramic segmentation can meet the tasks of target detection, instance segmentation and semantic segmentation at one time, and has good application prospects in national defense, rescue and disaster relief, and ecological management.

[0003] Due to the complexity of panoramic segmentation tasks and the challenges of difficult-to-distinguish and easy-to-confuse remote sensing images, remote sensing image panoramic segmentation has always been difficult to achieve high precision and accuracy. On the one hand, panoramic segmentation needs to realize target-level detection and recognition, and pixel-level classification and segmentation, and the complexity of the mixed task limits the effective improvement of the precision. On the other hand, compared with natural images, due to the difference in imaging principle and angle, there are difficult-to-distinguish and easy-to-confuse problems between targets and ground objects.

[0004] And the existing research on remote sensing image panoramic segmentation technology is mainly based on full supervision training method. Due to the limitation of labeling cost, the training data is often limited, and the characteristics of remote sensing big data cannot be fully utilized. And the full supervision method is affected by the sample distribution of the training data, and has weak scene migration ability and serious cross-domain problem.

[0005] Although the unsupervised method can rely on a large amount of unlabeled data for pre-training to solve the problem of labeling cost and model generalization ability, due to the characteristics of the imaging angle of remote sensing images, the actual proportion of important remote sensing targets in the image is very small, which leads to the problems of semantic information loss and target feature degradation. And the pre-training model has a large number of parameters, which is difficult to migrate, affecting the flexibility and timeliness. In addition, the previous remote sensing image panoramic segmentation technology often follows the processing idea of natural images, focusing on ground object segmentation, and failing to focus on important remote sensing targets. SUMMARY

[0006] In order to solve the above technical problems in the prior art, the purpose of the present application is to provide a remote sensing image panoramic segmentation network and a panoramic segmentation method, which can realize high-accuracy panoramic segmentation of high-resolution remote sensing images.

[0007] To achieve the above-mentioned purposes of the application, the application provides a remote sensing image panoramic segmentation network, comprising:

[0008] An image feature extraction backbone network is used to obtain remote sensing image features based on a remote sensing image to be panoramically segmented.

[0009] A pre-trained remote sensing characteristic knowledge extraction module is used to obtain remote sensing characteristic knowledge features based on the remote sensing image to be panoramically segmented.

[0010] A registration and fusion module is used to register and fuse the remote sensing image features and the remote sensing characteristic knowledge features to obtain remote sensing fusion features.

[0011] An instance segmentation branch is used to perform instance segmentation on the remote sensing fusion features respectively to obtain remote sensing instance segmentation results.

[0012] A semantic segmentation branch is used to perform semantic segmentation on the remote sensing fusion features respectively to obtain remote sensing semantic segmentation results.

[0013] A segmentation result combination module is used to combine the remote sensing instance segmentation results and the remote sensing semantic segmentation results to obtain remote sensing image panoramic segmentation results.

[0014] The application further provides a remote sensing image panoramic segmentation method, which is realized based on the above-mentioned remote sensing image panoramic segmentation network.

[0015] The steps are as follows:

[0016] Remote sensing image features and remote sensing characteristic knowledge features are obtained based on a remote sensing image to be panoramically segmented.

[0017] The remote sensing image features and the remote sensing characteristic knowledge features are registered and fused to obtain remote sensing fusion features.

[0018] The remote sensing fusion features are respectively subjected to instance segmentation and semantic segmentation to obtain remote sensing instance segmentation results and remote sensing semantic segmentation results.

[0019] The remote sensing instance segmentation results and the remote sensing semantic segmentation results are combined to obtain remote sensing image panoramic segmentation results.

[0020] According to one technical solution of the application, the registration and fusion of the remote sensing image features and the remote sensing characteristic knowledge features include the following processes:

[0021] The size of the remote sensing characteristic knowledge features is registered to the remote sensing image features to obtain spatially registered remote sensing characteristic knowledge features.

[0022] The spatially registered remote sensing characteristic knowledge features are normalized to obtain semantically registered remote sensing characteristic knowledge features.

[0023] The semantic registration remote sensing characteristic knowledge feature and the remote sensing image feature weighted sum are calculated to obtain the remote sensing fusion feature.

[0024] According to one of the technical solutions of the present application,

[0025] The instance segmentation is instance segmentation of a target in a remote sensing image.

[0026] The semantic segmentation is semantic segmentation of a background in the remote sensing image.

[0027] According to one of the technical solutions of the present application, the training process of the remote sensing characteristic knowledge extraction module includes:

[0028] A remote sensing image used for remote sensing characteristic knowledge extraction is obtained, a target in the remote sensing image is labeled by a weak labeling method, and a training data set used for remote sensing characteristic knowledge extraction is constructed.

[0029] The weak labeling is a partial manual labeling of the target in the remote sensing image by points or scribbles.

[0030] Based on the training data set, a self-supervised learning strategy is used to pre-train a knowledge extraction first network based on a Transformer architecture to obtain a pre-trained knowledge extraction first network.

[0031] The pre-trained knowledge extraction first network is used as a teacher branch, a knowledge extraction second network based on a Transformer architecture is used as a student branch, and a parameter sharing knowledge transfer network is constructed.

[0032] By minimizing the teacher-student output difference, the remote sensing characteristic knowledge in the encoder of the knowledge extraction first network is transferred to the feature extraction module of the knowledge extraction second network to obtain a pre-trained feature extraction module of the knowledge extraction second network, which is used as a final remote sensing characteristic knowledge extraction module.

[0033] According to one of the technical solutions of the present application, the remote sensing characteristic knowledge extraction second network is a Transformer architecture or a CNN architecture; and the number of layers of the knowledge extraction second network is less than the number of layers of the knowledge extraction first network.

[0034] According to one of the technical solutions of the present application, the self-supervised learning strategy is a mask autoencoder (MAE) strategy.

[0035] The loss function of the MAE strategy is:

[0036]

[0037] Among them, and are weights of a manually annotated region reconstruction loss function and a non-manually annotated region reconstruction loss function , and , an initial ;

[0038] ;

[0039] wherein, is an original manually annotated region, is a manually annotated region reconstructed after being randomly masked, is a total number of pixels or features in a neighborhood of a randomly masked region in the manually annotated region, is any of the pixels or features;

[0040]

[0041] wherein, is an original non-manually annotated region, is a non-manually annotated region reconstructed after being randomly masked, is a total number of pixels or features in a neighborhood of a randomly masked region in the non-manually annotated region, is any of the pixels or features.

[0042] According to one technical solution of the present application, the parameter sharing knowledge transfer network comprises an input embedding module, a position encoding module, a teacher branch and a student branch.

[0043] The output of the input embedding module and the output of the position encoding module are added together, and the result is used as the common input of the teacher branch and the student branch.

[0044] The teacher branch comprises a pre-trained knowledge extraction first network, and the student branch comprises a knowledge extraction second network of a Transformer architecture.

[0045] The decoder of the knowledge extraction first network and the decoder of the knowledge extraction second network each comprise three convolutional layers and one linear layer connected in sequence.

[0046] The present application also provides an electronic device comprising one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, and the one or more computer programs are stored in the memory; when the electronic device is running, the processor executes the one or more computer programs stored in the memory, so that the electronic device executes the remote sensing image panoramic segmentation method described above.

[0047] ​The application further provides a computer readable storage medium for storing computer instructions, which, when executed by a processor, implement the remote sensing image panoramic segmentation method.

[0048] The application provides a remote sensing image panoramic segmentation network and a panoramic segmentation method, which have the following beneficial effects:

[0049] By adopting the weakly supervised mask pre-training method focusing on the target and the knowledge distillation technology, a remote sensing characteristic knowledge extraction module containing common knowledge of remote sensing features is obtained, and then the characteristic knowledge is introduced through feature registration and fusion, so that deep feature mining of remote sensing important targets and features is realized and high-accuracy panoramic segmentation is realized.

[0050] (1) By designing the weakly supervised mask pre-training method focusing on the target, the labeling cost and the training data volume are greatly balanced. Compared with the full supervision method, the labeling cost is saved and the training data volume is improved; compared with the unsupervised method, the attention degree to the important target in the pre-training process is enhanced, and the target semantic loss and feature degradation are relieved.

[0051] (2) By the knowledge distillation method based on parameter sharing, the modularization embedding of the pre-training knowledge is realized, and the flexibility and timeliness of the model are increased. Knowledge transfer and model parameter compression are performed, and the generalization performance, flexibility and scalability of the module are improved.

[0052] (3) By extracting and registering and fusing the characteristic knowledge of the remote sensing important target in the pre-training and knowledge introduction stage, the attention degree to the remote sensing important target in the whole process is enhanced. BRIEF DESCRIPTION OF DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0054] Figure 1 a flow chart of the remote sensing image panoramic segmentation method based on characteristic knowledge introduction according to an embodiment of the present application is schematically represented;

[0055] Figure 2 a structure diagram of the remote sensing image panoramic segmentation network in the remote sensing image panoramic segmentation method based on characteristic knowledge introduction according to an embodiment of the present application is schematically represented;

[0056] Figure 3Schematic representation of the principle of point annotation and scribble annotation in the method of panoramic segmentation of remote sensing images based on the introduction of characteristic knowledge according to an embodiment of the present application;

[0057] Figure 4 Schematic representation of the structure of the knowledge extraction first network in the method of panoramic segmentation of remote sensing images based on the introduction of characteristic knowledge according to an embodiment of the present application;

[0058] Figure 5 Schematic representation of the structure of the parameter sharing knowledge transfer network in the method of panoramic segmentation of remote sensing images based on the introduction of characteristic knowledge according to an embodiment of the present application. DETAILED DESCRIPTION

[0059] The description of the embodiments of the present application should be combined with the corresponding drawings, which should be considered as a part of the complete description of the present application. In the drawings, the shape or thickness of the embodiments can be exaggerated and simplified or convenient for illustration. Moreover, the parts of the structures in the drawings will be described separately, and it should be noted that the elements not shown or not described in the drawings are in the form known by those skilled in the art.

[0060] The description of the embodiments herein, any reference to direction and position, is only for the convenience of description, and cannot be understood as any limitation on the scope of protection of the present application. The following description of the preferred embodiments will involve combinations of features, which can exist independently or in combination, and the present application is not particularly limited to the preferred embodiments. The scope of the present application is defined by the claims. DETAILED DESCRIPTION

[0062] The panoramic segmentation network of remote sensing images of the present embodiment comprises:

[0063] The image feature extraction backbone network is used to obtain remote sensing image features based on remote sensing images to be segmented panoramically;

[0064] The pre-trained remote sensing characteristic knowledge extraction module is used to obtain remote sensing characteristic knowledge features based on remote sensing images to be segmented panoramically;

[0065] The registration and fusion module is used to register and fuse the remote sensing image features and the remote sensing characteristic knowledge features to obtain remote sensing fusion features;

[0066] The instance segmentation branch is used to perform instance segmentation on the remote sensing fusion features respectively to obtain remote sensing instance segmentation results;

[0067] The semantic segmentation branch is used to perform semantic segmentation on the remote sensing fusion features respectively to obtain remote sensing semantic segmentation results;

[0068] The segmentation result combination module is configured to combine the remote sensing instance segmentation result and the remote sensing semantic segmentation result to obtain a remote sensing image panorama segmentation result.

[0069] In this embodiment, the specific architecture of the remote sensing image panorama segmentation network is involved. The remote sensing image panorama segmentation network can introduce remote sensing characteristic knowledge features for panorama segmentation. By constructing the remote sensing image panorama segmentation network and designing the feature registration and feature fusion method, the remote sensing characteristic knowledge feature introduction from the remote sensing characteristic knowledge extraction module to the remote sensing image panorama segmentation network is realized.

[0070] In the remote sensing image panorama segmentation network, a ResNet50 network is used as an image feature extraction backbone network, and an instance segmentation branch and a semantic segmentation branch are constructed.

[0071] The remote sensing image panorama segmentation network in this embodiment is mainly used for panorama segmentation of remote sensing images with high spatial resolution, complex and diverse ground object categories, fuzzy ground object boundaries, and multi-scale target mixed distribution characteristics. Panorama segmentation is to perform instance segmentation on countable foreground targets (such as discrete areas of buildings, vehicles, trees, bridges, etc.) in remote sensing images to identify the number, contour, and position of the targets (such as airplanes, ships, vehicles, etc.); and to perform semantic segmentation on the background ground objects (such as water surface, road, grassland, etc.) in the same remote sensing image to identify the category, range, and boundary of the background.

[0072] The instance segmentation branch is configured to receive the remote sensing fusion features, locate candidate target regions through a detection sub-network containing preset remote sensing target anchor boxes, generate masks for the remote sensing fusion features of each candidate region (such as through a 3x3 convolution layer and a sigmoid activation function to output pixel-level masks), and finally combine the target category classification results and the masks to obtain remote sensing instance segmentation results containing target individual distinguishing information.

[0073] The semantic segmentation branch is configured to receive the remote sensing fusion features, pass the remote sensing fusion features through a multi-scale feature fusion module (such as reducing the dimension of the remote sensing fusion features through a 1x1 convolution and then splicing with shallow detail features (such as features obtained by 2 times down-sampling of the input image)), perform feature up-sampling to the original image resolution through a transposed convolution layer, and finally output pixel-level category remote sensing semantic segmentation results through a classification head (such as a 1x1 convolution mapping to a preset number of semantic categories).

[0074] The remote sensing image panorama segmentation network is as shown in Figure 2The ResNet50 network is used as an image feature extraction backbone network to extract remote sensing image features of the remote sensing image, and the remote sensing image features are multi-scale deep semantic features and local detail features of the remote sensing image. The remote sensing characteristic knowledge extraction module is used to extract remote sensing characteristic knowledge features (of important targets in the remote sensing image) in the remote sensing image.

[0075] After the remote sensing image features and the remote sensing characteristic knowledge features are registered and fused, semantic segmentation and instance segmentation are performed. That is, the instance segmentation branch performs instance segmentation on countable foreground targets, and the semantic segmentation branch performs semantic segmentation on uncountable background objects. Finally, deep feature mining of important remote sensing targets and objects can be realized, and panoramic segmentation with high accuracy can be performed.

[0076] The "result" in the embodiment is generally a graph containing various segmentation results. Specific implementation method two

[0078] A panoramic segmentation method of a remote sensing image in the embodiment is realized based on the panoramic segmentation network of the remote sensing image described above.

[0079] The steps are as follows:

[0080] Remote sensing image features and remote sensing characteristic knowledge features are obtained based on the remote sensing image to be segmented.

[0081] The remote sensing image features and the remote sensing characteristic knowledge features are registered and fused to obtain remote sensing fusion features.

[0082] The remote sensing fusion features are respectively subjected to instance segmentation and semantic segmentation to obtain remote sensing instance segmentation results and remote sensing semantic segmentation results.

[0083] The remote sensing instance segmentation results and the remote sensing semantic segmentation results are combined to obtain remote sensing image panoramic segmentation results.

[0084] In the embodiment, the panoramic segmentation of the remote sensing image is realized by the following steps.

[0085] S1, remote sensing image features and remote sensing characteristic knowledge features are obtained based on the remote sensing image to be segmented.

[0086] Remote sensing image features of the remote sensing image are extracted based on an image feature extraction backbone network (ResNet50 network), including multi-scale deep semantic features and local detail features of the remote sensing image.

[0087] Remote sensing characteristic knowledge features in the remote sensing image, that is, important targets in the remote sensing image, are extracted based on a remote sensing characteristic knowledge extraction module.

[0088] S2, register and fuse the remote sensing image features and the remote sensing characteristic knowledge features to obtain remote sensing fusion features;

[0089] After registering and fusing the remote sensing image features and the remote sensing characteristic knowledge features, remote sensing fusion features are obtained, which include both multi-scale deep semantic and local detail fusion features and remote sensing characteristic knowledge features.

[0090] S3, perform instance segmentation and semantic segmentation on the remote sensing fusion features respectively to obtain remote sensing instance segmentation results and remote sensing semantic segmentation results;

[0091] The instance segmentation and semantic segmentation described above are combined as panoramic segmentation. In the panoramic segmentation stage, the registered and fused remote sensing fusion features are taken as input to perform instance segmentation on target individuals and semantic segmentation on background regions. The specific steps are as follows:

[0092] After feature registration and fusion of the remote sensing image features extracted by the image feature extraction backbone network and the features extracted by the remote sensing characteristic knowledge extraction module, the features are input to the instance segmentation branch and the semantic segmentation branch respectively;

[0093] Then, the instance segmentation branch performs instance segmentation on countable foreground targets, and the semantic segmentation branch performs semantic segmentation on uncountable background objects.

[0094] S4, combine the remote sensing instance segmentation results and the remote sensing semantic segmentation results to obtain remote sensing image panoramic segmentation results.

[0095] After combination, a remote sensing atlas is obtained, which not only recognizes the number, contour and position of the target, but also recognizes the category, range and boundary of the background.

[0096] The remote sensing image panoramic segmentation method of the embodiment mainly extracts remote sensing characteristic knowledge features of important remote sensing targets through the pre-trained remote sensing characteristic knowledge extraction module, and then performs registration and fusion, thereby enhancing the attention degree to important remote sensing targets.

[0097] The remote sensing image panoramic segmentation method of the embodiment takes characteristic knowledge introduction as the core and combines pre-training, knowledge distillation and panoramic segmentation technology.

[0098] Under the premise of including the pre-training step of the remote sensing characteristic knowledge extraction module, the remote sensing image panoramic segmentation method mainly includes six stages: constructing a characteristic knowledge extraction network, designing a weakly supervised mask pre-training method for focusing on targets, designing a remote sensing characteristic knowledge extraction module and a parameter sharing knowledge transfer network, using a knowledge distillation method for knowledge transfer, designing a characteristic knowledge introduction method, and high-resolution remote sensing image panoramic segmentation. THREE SPECIFIC IMPLEMENTATIONS

[0100] The embodiment is a further illustration of embodiment two, in which the remote sensing image features and remote sensing characteristic knowledge features are registered and fused, and the process includes:

[0101] The size of the remote sensing characteristic knowledge features is registered to the remote sensing image features to obtain spatially registered remote sensing characteristic knowledge features;

[0102] The spatially registered remote sensing characteristic knowledge features are normalized to obtain semantically registered remote sensing characteristic knowledge features;

[0103] The weighted sum of the semantically registered remote sensing characteristic knowledge features and the remote sensing image features is calculated to obtain remote sensing fusion features.

[0104] In this embodiment, the feature registration and fusion in the remote sensing characteristic knowledge feature introduction stage are mainly involved.

[0105] In the remote sensing characteristic knowledge feature introduction stage, by designing the registration and fusion method of the remote sensing characteristic knowledge features and the remote sensing image features, the remote sensing characteristic knowledge feature introduction from the remote sensing characteristic knowledge extraction module to the remote sensing image panoramic segmentation network can be realized, and the specific steps of registration and fusion are as follows:

[0106] The input remote sensing image data is obtained via the image feature extraction backbone network (ResNet50 network) and the remote sensing characteristic knowledge extraction module to obtain remote sensing image features and remote sensing characteristic knowledge features .

[0107] After registration, the remote sensing image features and the remote sensing characteristic knowledge features are fused, and the fused result is denoted as , and the processing flow can be represented as:

[0108]

[0109]

[0110]

[0111] wherein, is the result after spatial registration, is the size of , is an interpolation function used to register the size of to , is the result after semantic registration, is a normalization function, and are learnable weight coefficients. Specific embodiment four

[0113] The embodiment is a further illustration of the embodiment two or three, in the embodiment, instance segmentation is performed on the target in the remote sensing image;

[0114] Semantic segmentation is performed on the background in the remote sensing image. Specific implementation five

[0116] The embodiment is a further illustration of the embodiment four, in the embodiment, the training process of the remote sensing characteristic knowledge extraction module includes:

[0117] The remote sensing image used for remote sensing characteristic knowledge extraction is obtained, the target in the remote sensing image is labeled by a weak labeling method, and a training data set used for remote sensing characteristic knowledge extraction is constructed;

[0118] Weak labeling is a partial manual labeling of the target in the remote sensing image by points or scribbles;

[0119] Based on the training data set, a self-supervised learning strategy is used to pre-train the knowledge extraction first network based on the Transformer architecture, and a pre-trained knowledge extraction first network is obtained;

[0120] The pre-trained knowledge extraction first network is used as a teacher branch; the knowledge extraction second network based on the Transformer architecture is used as a student branch, and a parameter sharing knowledge transfer network is constructed;

[0121] By minimizing the difference between the outputs of the teacher and the student, the remote sensing characteristic knowledge in the encoder of the knowledge extraction first network is transferred to the feature extraction module of the knowledge extraction second network, and the feature extraction module of the pre-trained knowledge extraction second network is obtained and used as the final remote sensing characteristic knowledge extraction module.

[0122] In the embodiment, the training process of the remote sensing characteristic knowledge extraction module is involved.

[0123] S1, obtain a remote sensing image used for remote sensing characteristic knowledge extraction, label the target in the remote sensing image by a weak labeling method, and construct a training data set used for remote sensing characteristic knowledge extraction;

[0124] Weak labeling is a partial manual labeling of the target in the remote sensing image by points or scribbles;

[0125] The remote sensing image used for remote sensing characteristic knowledge extraction is a remote sensing image collected and sorted in advance. The labeling of the training data set is specifically a weak labeling of the target of interest in the remote sensing image, such as an airplane, a ship, a vehicle, etc. by using points, scribbles and the like. Among them, the labeling method of points and scribbles is as shown in Figure 3 The target is partially labeled instead of fine labeling, which simplifies the labeling process.

[0126] S2, based on the training data set, a self-supervised learning strategy is adopted to pre-train the knowledge extraction first network based on the Transformer architecture, and a pre-trained knowledge extraction first network is obtained;

[0127] Among them, for the high-resolution remote sensing image pre-training task, a knowledge extraction first network based on the Transformers architecture is constructed, mainly including an input embedding module, a position encoding module, a Transformer encoder containing 24 layers, a Transformer decoder containing 24 layers and an output layer, as shown in Figure 4 The specific principle is as follows:

[0128] The input embedding module is used for sequence generation of input data. Let the input image data be , the input weak annotation data be , and the sequence feature output by the input embedding module be The processing flow of this module can be represented as:

[0129]

[0130]

[0131]

[0132] Among them, is the original image data embedding, is the weak annotation data embedding, and are trainable fully connected layers, is dimensionally spliced.

[0133] The position encoding module adds the input data sequence characteristics to the network, helping the network quickly learn the spatial position relationship in the image. The absolute position encoding method is adopted, let be the position index, be the dimension index, and the output position encoding matrix of this module can be represented as:

[0134]

[0135]

[0136] Then, the position encoding matrix is added to the input embedding matrix to obtain the embedding representation containing position information :

[0137]

[0138] The transformer encoder is used to extract features from the input data, and the multi-head attention mechanism is the core algorithm for capturing global dependencies in the data. The calculation of each attention head is as follows:

[0139]

[0140] wherein, are the query, key and value matrices respectively, is the dimension of the matrix.

[0141] The output of the multi-head attention is the result of concatenating and linearly transforming the outputs of all attention heads:

[0142]

[0143] wherein, is the number of attention heads, is the weight matrix of the output linear transformation.

[0144] The structure of the transformer decoder is similar to that of the transformer encoder, relying on the input image features to build the relevance between the masked part and the remaining part in the pre-training stage, and generating the reconstructed masked part by an output layer . This step can be represented as:

[0145]

[0146] wherein, is the output feature processed by the transformer decoder, is the weight matrix of the output linear layer, is the bias vector.

[0147] S3, extract the encoder of the pre-trained knowledge extraction first network as a teacher model; extract the knowledge of the second network as a student model, and construct a parameter sharing knowledge transfer network;

[0148] In the design of the remote sensing characteristic knowledge extraction module and the parameter sharing knowledge transfer network, a transformer structure with fewer layers (containing 6 encoder layers and 6 decoder layers) is designed as the knowledge extraction second network, and a parameter sharing knowledge transfer network is realized in a double-branch structure.

[0149] wherein, the pre-trained knowledge extraction first network is used as the teacher branch.

[0150] The knowledge extraction second network is a student branch. The remote sensing characteristic knowledge extraction module is composed of a feature extraction module structure with fewer layers. Compared with the Transformer encoder in the knowledge extraction first network which has been pre-trained, the module has the advantages of fewer parameters and being suitable for embedding in other models as a module. Moreover, in order to be able to use the knowledge distillation method, the remote sensing characteristic knowledge extraction module is designed to have similar characteristic extraction capability and generalization performance as the Transformer encoder in the knowledge extraction first network after pre-training, and a parameter sharing knowledge transfer network is designed.

[0151] S4, by minimizing the teacher-student output difference, the remote sensing characteristic knowledge in the encoder of the knowledge extraction first network is transferred to the feature extraction module (such as the encoder of the Transformer or the convolutional layer of the CNN) of the knowledge extraction second network, and the feature extraction module of the pre-trained knowledge extraction second network is obtained as the final remote sensing characteristic knowledge extraction module.

[0152] In the knowledge distillation knowledge transfer stage, a training method with partial weight freezing and a distillation loss function are designed, and training is performed to gradually make the student branch reach the accuracy of the teacher branch.

[0153] Among them, only the remote sensing characteristic knowledge extraction module is trained, and the parameters of other network structures are in a frozen state.

[0154] In order to make the output of the distilled knowledge module as close as possible to the output of the original network, the distillation loss function is defined as follows:

[0155]

[0156] Among them, and are the outputs of the knowledge extraction first network and the distilled knowledge extraction second network, respectively;

[0157] Finally, the parameter sharing knowledge transfer network is trained by using high-resolution remote sensing image data through remote sensing image scene and target classification tasks until the student branch of the remote sensing characteristic knowledge extraction module reaches similar performance to the teacher branch, and the teacher-student output difference is minimized.

[0158] Using pre-training and knowledge distillation technology, the characteristic knowledge of typical remote sensing targets is extracted from massive remote sensing data, and important remote sensing targets are focused on. Subsequently, feature registration and fusion are realized through the introduction of characteristic knowledge, and high-accuracy remote sensing image panoramic segmentation is realized, which has important theoretical research and application value. Specific implementation method six

[0160] ​In this embodiment, the remote sensing characteristic knowledge extraction second network is a Transformer architecture or a CNN architecture; and the number of layers of the knowledge extraction second network is less than the number of layers of the knowledge extraction first network. Specific implementation method seven

[0162] In this embodiment, the self-supervised learning strategy is a mask autoencoder (MAE) strategy.

[0163] The loss function of the MAE strategy is:

[0164]

[0165] wherein, and are the weights of the artificial annotation area reconstruction loss function and the non-artificial annotation area reconstruction loss function , and , the initial ;

[0166] ;

[0167] wherein, is the original artificial annotation area, is the reconstructed artificial annotation area after random masking, is the total number of pixels or features in the neighborhood of the randomly masked area in the artificial annotation area, is any of the above pixels or features.

[0168]

[0169] wherein, is the original non-artificial annotation area, is the reconstructed non-artificial annotation area after random masking, is the total number of pixels or features in the neighborhood of the randomly masked area in the non-artificial annotation area, is any of the above pixels or features.

[0170] In this embodiment, the process of weakly supervised mask pre-training of the attention target by using the self-supervised learning strategy for the knowledge extraction first network is specifically disclosed.

[0171] In the design of weakly supervised mask pre-training method of attention target, a weakly labeled improved MAE method is designed, that is, the attention targets in the training data, such as aircraft, ship, vehicle, etc., are weakly labeled by using points, graffiti, etc., and input into the network for input coding during pre-training. It can improve the attention degree of the first network to typical remote sensing targets and improve the generalization performance of the network after pre-training.

[0172] The above-mentioned MAE (Masked Autoencoders) is a self-supervised learning method used for pre-training deep neural networks. The core idea is to mask a part of the input data, and then train the model to reconstruct these masked parts.

[0173] By mixing loss , increase the attention of the target area, for the labeled part of the field range, use mean square error to calculate the mean square error between the reconstructed randomly masked area in the artificial labeled area and the original randomly masked area in the artificial labeled area:

[0174]

[0175] Wherein, is the original artificial labeled area, is the reconstructed artificial labeled area after random masking, is the total number of pixels or features in the neighborhood of the randomly masked area in the artificial labeled area;

[0176] For other unlabeled parts, use absolute error loss to calculate the absolute difference between the reconstructed randomly masked area in the non-artificial labeled area and the original randomly masked area in the non-artificial labeled area:

[0177]

[0178] Wherein, is the original non-artificial labeled area, is the reconstructed non-artificial labeled area after random masking, is the total number of pixels or features in the neighborhood of the randomly masked area in the non-artificial labeled area;

[0179] In summary, the total loss is:

[0180]

[0181] Wherein, and are hyperparameters used to balance the weights of the two losses. DETAILED DESCRIPTION EIGHT

[0183] This embodiment is a further explanation of embodiment seven. In this embodiment, the parameter sharing knowledge transfer network includes an input embedding module, a position encoding module, a teacher branch, and a student branch.

[0184] The output of the input embedding module is added to the output of the position encoding module, and the result is used as the common input for the teacher branch and the student branch.

[0185] The teacher branch includes a pre-trained knowledge extraction first network; the student branch includes a knowledge extraction second network based on the Transformer architecture.

[0186] Both the decoder of the first knowledge extraction network and the decoder of the second knowledge extraction network consist of three sequentially connected... A convolutional layer and a linear layer.

[0187] This embodiment involves the specific architecture of the parameter-sharing knowledge transfer network.

[0188] like Figure 5 As shown, in the parameter-sharing knowledge transfer network, the input data is processed by the input embedding module and the position encoding module, and then connected to the first knowledge extraction network as the teacher branch and the second knowledge extraction network as the student branch, respectively. The decoders in the teacher and student branches have the same structure, consisting of three layers. It consists of a convolutional layer and a linear layer.

[0189] The present invention discloses a remote sensing image panoramic segmentation network and method. The remote sensing image panoramic segmentation network comprises: an image feature extraction backbone network for obtaining remote sensing image features based on the remote sensing image to be segmented panoramically; a pre-trained remote sensing characteristic knowledge extraction module for obtaining remote sensing characteristic knowledge features based on the remote sensing image to be segmented panoramically; a registration and fusion module for registering and fusing the remote sensing image features and the remote sensing characteristic knowledge features to obtain remote sensing fused features; an instance segmentation branch for performing instance segmentation on the remote sensing fused features to obtain remote sensing instance segmentation results; a semantic segmentation branch for performing semantic segmentation on the remote sensing fused features to obtain remote sensing semantic segmentation results; and a segmentation result combination module for combining the remote sensing instance segmentation results and the remote sensing semantic segmentation results to obtain the remote sensing image panoramic segmentation result.

[0190] Furthermore, it should be noted that the present invention can be provided as a method, apparatus, or computer program product. Therefore, embodiments of the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, embodiments of the present invention can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0191] The embodiments of the present application are described with reference to the flowchart illustrations and / or block diagrams of the methods, terminal devices (systems) and computer program products according to embodiments of the present application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing terminal devices to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal devices, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0192] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal devices to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means that implement the function specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. These computer program instructions can also be loaded onto a computer or other programmable data processing terminal devices, such that a series of operational steps are performed on the computer or other programmable terminal devices to create a computer implemented process so that the instructions executed on the computer or other programmable terminal devices provide steps for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0193] It should also be noted that, in the present text, the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusion, so that processes, methods, articles or terminal devices that include a series of elements not only include those elements, but also include other elements that are not explicitly listed, or other elements inherent to such processes, methods, articles or terminal devices. Without more limitations, the element defined by the phrase "including a" does not exclude the presence of additional identical elements in the process, method, article or terminal device that includes the element.

[0194] Finally, it should be noted that the above description is of preferred embodiments of the application, and that although preferred embodiments of the application have been described, numerous changes and modifications can be made to the preferred embodiments without departing from the principles of the application, and that such changes and modifications are contemplated as falling within the scope of the application. Accordingly, the appended claims are intended to embrace all such changes and modifications.

Claims

1. A panoramic segmentation network for remote sensing images, characterized in that, include: An image feature extraction backbone network is used to obtain remote sensing image features based on the remote sensing image to be segmented in a panorama. A pre-trained remote sensing feature knowledge extraction module is used to obtain remote sensing feature knowledge characteristics based on the remote sensing image to be segmented in a panoramic view. The registration and fusion module is used to register and fuse the remote sensing image features and remote sensing characteristic knowledge features to obtain remote sensing fused features; The instance segmentation branch is used to segment the remote sensing fusion features into instances to obtain remote sensing instance segmentation results. The semantic segmentation branch is used to perform semantic segmentation on the remote sensing fusion features to obtain remote sensing semantic segmentation results. The segmentation result combination module is used to combine the remote sensing instance segmentation result and the remote sensing semantic segmentation result to obtain the panoramic segmentation result of the remote sensing image.

2. A method for panoramic segmentation of remote sensing images, characterized in that, Implementation based on the remote sensing image panoramic segmentation network described in claim 1; The steps are as follows: Based on the remote sensing image to be segmented into a panorama, remote sensing image features and remote sensing characteristic knowledge features are obtained respectively; The remote sensing image features and remote sensing characteristic knowledge features are registered and fused to obtain remote sensing fusion features; The remote sensing fusion features are subjected to instance segmentation and semantic segmentation respectively to obtain remote sensing instance segmentation results and remote sensing semantic segmentation results; The remote sensing instance segmentation results and the remote sensing semantic segmentation results are combined to obtain the panoramic segmentation results of the remote sensing image.

3. The remote sensing image panoramic segmentation method according to claim 2, characterized in that, The process of registering and fusing the remote sensing image features and remote sensing characteristic knowledge features includes: The size of the remote sensing characteristic knowledge features is registered to the remote sensing image features to obtain spatially registered remote sensing characteristic knowledge features; The remote sensing characteristic knowledge features of spatial registration are normalized to obtain the remote sensing characteristic knowledge features of semantic registration. The remote sensing fusion feature is obtained by calculating the weighted sum of the remote sensing characteristic knowledge features of the semantic registration and the remote sensing image features.

4. The remote sensing image panoramic segmentation method according to claim 2 or 3, characterized in that, The instance segmentation refers to the instance segmentation of targets in remote sensing images; The semantic segmentation refers to performing semantic segmentation on the background in the remote sensing image.

5. The remote sensing image panoramic segmentation method according to claim 4, characterized in that, The training process for the remote sensing characteristic knowledge extraction module includes: Acquire remote sensing images for remote sensing feature knowledge extraction, label targets in the remote sensing images using a weak labeling method, and construct a training dataset for remote sensing feature knowledge extraction. The weak annotation refers to the partial manual annotation of targets in the remote sensing image using dots or doodles. Based on the training dataset, a self-supervised learning strategy is used to pre-train the first knowledge extraction network based on the Transformer architecture to obtain the pre-trained first knowledge extraction network. The pre-trained knowledge extraction first network is used as the teacher branch; the knowledge extraction second network based on the Transformer architecture is used as the student branch to construct a parameter-sharing knowledge transfer network. By minimizing the difference between teacher and student outputs, the remote sensing characteristic knowledge in the encoder of the first knowledge extraction network is transferred to the feature extraction module of the second knowledge extraction network to obtain the pre-trained feature extraction module of the second knowledge extraction network, which serves as the final remote sensing characteristic knowledge extraction module.

6. The remote sensing image panoramic segmentation method according to claim 5, characterized in that, The second network for remote sensing feature knowledge extraction is a Transformer architecture or a CNN architecture; and the number of layers in the second network for remote sensing feature knowledge extraction is less than the number of layers in the first network for remote sensing feature knowledge extraction.

7. The remote sensing image panoramic segmentation method according to claim 5 or 6, characterized in that, The self-supervised learning strategy is the Masked Autoencoder (MAE) strategy. The loss function of the MAE strategy is: in, and These are the loss functions for reconstructing manually labeled regions. Non-manually labeled region reconstruction loss function The weights, and ,initial ; ； in, It is the original manually labeled area. These are manually labeled regions that have been reconstructed after being randomly occluded. It is the total number of pixels or features in the neighborhood of the randomly occluded area within the manually labeled region. It is any of the aforementioned pixels or features; in, It is the original, non-manually labeled area. These are non-manually labeled regions that have been reconstructed after being randomly occluded. It is the total number of pixels or features in the neighborhood of randomly occluded areas within a non-manually labeled region. It is any of the aforementioned pixels or features.

8. The remote sensing image panoramic segmentation method according to claim 7, characterized in that, The parameter-sharing knowledge transfer network includes an input embedding module, a position encoding module, a teacher branch, and a student branch; The result of adding the output of the input embedding module to the output of the position encoding module is used as the common input of the teacher branch and the student branch. The teacher branch includes a pre-trained knowledge extraction first network; the student branch includes a knowledge extraction second network based on the Transformer architecture. The decoders of both the first and second knowledge extraction networks consist of three sequentially connected decoders. A convolutional layer and a linear layer.

9. An electronic device, characterized in that, include: One or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, and the one or more computer programs are stored in the memory, and when the electronic device is running, the processor executes the one or more computer programs stored in the memory to cause the electronic device to perform the remote sensing image panoramic segmentation method as described in any one of claims 2 to 8.

10. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, implement the remote sensing image panoramic segmentation method as described in any one of claims 2 to 8.

Images