Remote sensing ground feature information extraction method and device based on pixel contrast learning

Abstract

The invention provides a remote sensing ground feature information extraction method and device based on pixel contrast learning, and the method comprises the steps: obtaining an original remote sensing image, constructing an input image matrix according to the attribute information of the original remote sensing image, and carrying out the dimension conversion of the input image matrix; a SegFormer pre-training model is used to extract multi-scale features of the input image matrix after the dimension conversion, the multi-scale features are fused, a feature matrix is output, and the feature matrix comprises a ground feature classification result; and forming supervised contrast segmentation loss by the pixel contrast loss and the cross entropy loss, and constraining the ground feature classification result by using the supervised contrast segmentation loss. According to the scheme, the feature information extraction efficiency and extraction precision are improved.

Description

Technical Field

[0001] The invention belongs to the field of remote sensing image recognition, and in particular relates to a remote sensing ground object information extraction method and device based on pixel contrast learning. Background Art

[0002] Remote sensing data has the characteristics of large scale, short update cycle and convenient acquisition, which can provide a rich data source for information extraction, making deep learning methods well matched in remote sensing ground object information extraction tasks. However, the complex scenes, rich ground object categories and huge coverage in remote sensing images make it difficult to construct data sets, which leads to the fact that public data set resources for remote sensing are still in a relatively scarce state. In the field of computer vision, many natural image data sets have been constructed, and traditional schemes generate models based on them and use applicable pre-trained models. Therefore, pre-trained models based on natural images are widely used in remote sensing ground object information extraction tasks.

[0003] In the pre-trained model, contrastive learning can train the model by learning the similarities and differences between samples, which can improve the model's ability to judge the category features and further improve the model's generalization ability and performance. However, it only takes advantage of the self-supervised method in data utilization and does not take advantage of the features of image details. Therefore, the traditional solution still lacks in the integrity and accuracy of ground object information extraction. Summary of the invention

[0004] The purpose of the present invention is to provide a method and device for extracting remote sensing ground object information based on pixel contrast learning, aiming to achieve efficient, intelligent and high-precision road information extraction with smaller parameters.

[0005] According to a first aspect of the present invention, a method for extracting remote sensing ground object information based on pixel contrast learning is provided, comprising:

[0006] Acquire an original remote sensing image, construct an input image matrix according to attribute information of the original remote sensing image, and perform dimension conversion on the input image matrix;

[0007] The SegFormer pre-trained model is used to extract multi-scale features of the input image matrix after dimension conversion, and the multi-scale features are fused to output a feature matrix, which contains the ground object classification results;

[0008] The pixel contrast loss and the cross entropy loss are combined into a supervised contrast segmentation loss, and the supervised contrast segmentation loss is used to constrain the object classification result.

[0009] Preferably, constructing an input image matrix according to the attribute information of the original remote sensing image and performing dimension conversion on the input image matrix further comprises:

[0010] Set the batch size to 3, load the original remote sensing image, and construct the input image matrix as (3, H, W, C), where H and W are the height and width of the original remote sensing image, and C is the dimension of the original remote sensing image;

[0011] The channel conversion module is used to convert the dimension of the loaded original remote sensing image, that is, to convert the input matrix (3, H, W, C) into (3, H, W, 3).

[0012] Preferably, the extracting multi-scale features of the input image matrix after dimension conversion by using the SegFormer pre-training model and fusing the multi-scale features further comprises:

[0013] In the encoder stage, a four-layer hierarchical feature extraction module is used to extract multi-scale features of the original remote sensing image at scales of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 respectively;

[0014] In the decoder stage, a multi-layer perceptron (MLP) is used to fuse the multi-scale features extracted during the encoding process. During the fusion process, all feature layers are upsampled to H / 4, W / 4, and the image feature matrix (3, H / 4, W / 4, Ncls) is output, where Ncls is the preset number of target land categories.

[0015] Preferably, the cross entropy loss is used to compare the segmentation results of each original remote sensing image in a batch with the true surface value, and the pixel contrast loss is used to directly perform category judgment on all features in the batch, that is, to perform category aggregation operations on the three-layer feature map.

[0016] According to a second aspect of the present invention, there is provided a remote sensing ground object information extraction device based on pixel contrast learning, comprising:

[0017] A remote sensing image preprocessing module is used to obtain the original remote sensing image, construct an input image matrix according to the attribute information of the original remote sensing image, and perform dimension conversion on the input image matrix;

[0018] A pre-training module is used to extract multi-scale features of the input image matrix after dimension conversion using the SegFormer pre-training model, fuse the multi-scale features, and output a feature matrix, wherein the feature matrix includes a ground object classification result;

[0019] The classification result constraint module is used to combine pixel contrast loss and cross entropy loss into supervised contrast segmentation loss, and use the supervised contrast segmentation loss to constrain the ground object classification result.

[0020] The present invention uses CNN to perform shallow feature extraction on local features during the feature extraction process, providing richer image information for further road feature extraction, and uses the Transformer framework based on the self-attention mechanism to fully extract the global information of the road in the image, so that the model can perform more efficient feature learning and improve the model recognition accuracy. Through pixel comparison learning, the category association in the target object feature mapping space across images is strengthened to better aggregate similar information, thereby improving the flexibility and feature expression ability of the model, enhancing the integrity of the model in extracting object information, and realizing high-precision remote sensing object information extraction under limited data resource conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a flow chart of a remote sensing ground object information extraction method based on pixel contrast learning according to the present invention;

[0022] Figure 2 is a diagram of the SegFormer network structure based on pixel contrast learning according to the present invention;

[0023] Figure 3 is a Massachusetts image road extraction result map according to the present invention;

[0024] Figure 4 is a comparison diagram of road extraction results of Massachusetts images including the multi-model of the present invention;

[0025] Figure 5 A comparison diagram of ground object extraction results of a multi-model Potsdam image including the present invention;

[0026] Figure 6 It is a structural diagram of the channel convert of the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments.

[0028] The method provided by the present invention can be implemented in the following device environment, and the device may include one or more of the following components: a processor, a memory, and a display screen. The memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the method described in the following embodiment.

[0029] In addition, those skilled in the art will appreciate that the structure of the above device does not limit the device, and the device may include more or fewer components, or a combination of certain components, or different component arrangements. For example, the device also includes components such as a radio frequency circuit, an input unit, a sensor, an audio circuit, and a power supply, which will not be described in detail here.

[0030] Pixel contrast learning is to compare images at the pixel level, and to increase the model's ability to discern land object information by maximizing the similarity between the same land types and minimizing the similarity between different land types. It can not only learn more fine-grained feature representations, but also make full use of the rich information characteristics of images. Based on the above analysis, the present invention proposes a lightweight Transformer remote sensing land object information extraction model Pc-Seg that integrates pixel contrast learning to achieve high-precision remote sensing land object information extraction under limited data resources. The lightweight pre-trained model SegFormer is used as the basic model. As a pre-trained model, SegFormer is trained from a large amount of natural image data. It itself has rich feature parameters and good generalization ability. Under the current conditions of scarce data resources and limited hardware resources, the use of SegFormer can improve the model's applicability and land object information extraction accuracy at a lower cost. The Pc-Seg model of the present invention aims at the problem that SegFormer models RGB images and has certain limitations in its application in multi-dimensional remote sensing images. The CNN-based dimension conversion module is combined with SegFormer, and CNN is used to perform shallow feature extraction and data dimension conversion on the image, which overcomes the application limitations of SegFormer in multi-dimensional remote sensing images and enhances the local information expression of shallow features of input data. Finally, in order to enhance the effective use of remote sensing image information by the model and improve the ability to aggregate cross-image land class features, the Pc-Seg model introduces a pixel comparison learning method to further improve the generalization ability of the model and the accuracy of ground object information extraction.

[0031] See also Figure 1 The remote sensing ground object information extraction method based on pixel contrast learning provided by the present invention comprises:

[0032] Step 101: Acquire an original remote sensing image, construct an input image matrix according to attribute information of the original remote sensing image, and perform dimension conversion on the input image matrix.

[0033] like Figure 2As shown, first select a suitable batch size batchsize, load the training data, and set it to 3 in this embodiment, so the input image matrix is ​​(3, H, W, C), which is used to divide the image into multiple small parts, namely batches, and input these batches into the neural network in sequence for batch processing. Where H and W are the height and width of the original remote sensing image, and C is the dimension of the original remote sensing image, that is, the number of bands of the remote sensing image. For example, if the image contains three bands of red, green, and blue, the image is three-dimensional, and if it is four bands of red, green, blue, and near infrared, it is four-dimensional.

[0034] use Figure 6 The channel conversion module Channel Convert shown performs dimensionality conversion on the loaded remote sensing image, converts the input matrix (3, H, W, C) into (3, H, W, 3) to make the true color display consistent with the RGB image, conform to the input data format of the TransFormer model, and meet the input requirements of the SegFormer pre-trained model.

[0035] Step 102: Use the SegFormer pre-trained model to extract multi-scale features of the input image matrix after dimension conversion, fuse the multi-scale features, and output a feature matrix, which contains the ground object classification results.

[0036] The input data feature layer is extracted through the SegFormer pre-trained model. SegFormer consists of two stages: encoder and decoder. In the encoder stage, SegFormer uses a four-layer hierarchical feature extraction module to extract multi-scale features of the original remote sensing image at scales of 1 / 4, 1 / 8, 1 / 16, and 1 / 32. In the decoder stage, SegFormer uses a lightweight multi-layer perceptron MLP to fuse the multi-scale features extracted during the encoding process. During the fusion process, all feature layers are upsampled to H / 4, W / 4. The final output image feature matrix is ​​(3, H / 4, W / 4, Ncls), where Ncls is the number of target land categories preset in the extraction task.

[0037] In the encoder stage, the model first encodes the input layer in blocks (i.e., Transformerblock). The Transformer Block module corresponds to the self-attention mechanism, which allows the model to interact globally between different positions to capture long-distance dependencies. Then, the encoded data is subjected to 4-layer hierarchical feature extraction. In each layer, the feature layer is extracted using efficient self-attention, hybrid feature fusion, and overlapping feature fusion methods. In the efficient self-attention module, the self-attention mechanism is the core of encoder feature extraction, and self-attention is implemented by multi-head Q, K, and V matrices. The specific formula is as follows:

[0038]

[0039] Efficient self-attention transforms the dimension of the matrix feature K in the attention by introducing the conversion ratio R to achieve the purpose of model complexity compression. The conversion formula is as follows:

[0040]

[0041]

[0042] From the formula, we can see that for the matrix K(N, C), the conversion ratio R is used to reshape the matrix K into a shape (N / R, C×R), and then the transformed K is linearly transformed through the linear layer (Linear), and the C×R tensor is used as input to generate a C-dimensional tensor. Therefore, the new matrix of K is (N / R, C). This reduces the complexity of the self-attention mechanism from O(N 2 ) is reduced to O(N 2 / R).

[0043] The present invention further sets the conversion rate R of layers 1 to 4 to [64, 16, 4, 1] in the hierarchical Transformer encoder of the SegFormer model. At the same time, SegFormer uses a hybrid feature fusion network to improve the degradation of the model performance when the test resolution of the position code interpolation is different from that of the training. The hybrid feature fusion network process is:

[0044] xout=MLP(GELU(Conv3×3(MLP(xin))))+xin

[0045] MLP is the multi-layer perceptron vector mapping function; GELU is the activation function; xin is the input of the hybrid feature fusion network, and xout is the output of the hybrid feature fusion network.

[0046] For the output of hierarchical features, overlapping feature blocks with surrounding features are used to replace non-overlapping feature blocks in VIT to further enhance the model feature extraction capability.

[0047] The SegFormer decoder process is as follows:

[0048] F^ i =Linear(C i , C)(F i )

[0049] F^ i =Upsample(H / 4×W / 4)(F^ i )

[0050] F = Linear(4C, C)(Concat(F^ i ))

[0051] M = Linear(C, Ncls)(F).

[0052] Among them: Linear is the linear transformation operation of the input data through the weight matrix, Concat is the connection operator, F i is the hierarchical feature, C i is the hierarchical feature dimension i level, C is the merge dimension, H is the input image height, W is the input image width, Ncls is the number of target categories, and M is the segmentation result.

[0053] In the specific process of the decoder, first, the multi-level features F obtained by the encoder are i The channel dimension is unified through an MLP layer. Then, the features are upsampled to 1 / 4 of the input image and concatenated together. An MLP layer is used to fuse the concatenated features F. Finally, another MLP layer uses the fused features to predict the segmentation result M with a resolution of H / 4×W / 4×Ncls.

[0054] Step 103: The pixel contrast loss and the cross entropy loss are combined into a supervised contrast segmentation loss, and the supervised contrast segmentation loss is used to constrain the object classification result.

[0055] The image feature matrix is ​​restored to (3, H, W, Ncls) using the upsampling process, and the supervised contrast segmentation loss function is used to constrain the model classification results. In the pixel contrast learning paradigm, the cross entropy function is combined with the pixel contrast loss function to form a supervised contrast segmentation loss function.

[0056] Exemplarily, the supervised contrastive segmentation loss function is It is expressed as:

[0057]

[0058] in: is the cross entropy loss function, λ is the weight coefficient; is the pixel contrast loss function.

[0059] Cross Entropy Loss Function It is expressed as:

[0060]

[0061] Where: Ω is the number of categories, y ic is a sign function (0 or 1). If the true category of sample i is equal to category c, then y ic Take 1, otherwise take 0, p icis the predicted probability that the observed sample i belongs to category c.

[0062] Pixel contrast loss function as follows:

[0063]

[0064] Where: i is the pixel category anchor point, and Respectively represent the positive sample and negative sample pixel feature sets of pixel i. + and i - They refer to positive eigenvectors (similar) and negative eigenvectors (different), respectively. τ is a hyperparameter greater than zero that controls the sharpness of the exponential function. By adjusting the value of τ, the sensitivity of the loss function to similarity and difference can be changed. The operator "·" represents the inner (dot) product.

[0065] In the supervised contrast segmentation loss function, the cross entropy loss is used to compare the segmentation results of each original remote sensing image in the batch with the true surface value, that is, the cross entropy loss is calculated three times for one training. The pixel contrast learning loss does not make category judgments on the features, but directly makes category judgments on all the features in the batch, that is, category aggregation operations are performed on the three-layer feature layers. Therefore, the pixel contrast learning loss is only calculated once for one training.

[0066] The supervised contrast segmentation loss function combines the pixel contrast loss function with the cross entropy loss function, which not only makes up for the problem that the relationship between pixels cannot be expressed due to the fact that only the prediction of each pixel is independently constrained in the cross entropy loss function, but also because the pixel contrast loss function can directly bring pixels of the same category closer in the feature space and make pixels of different categories move away from each other, further strengthening the constraints on pixel features.

[0067] The datasets used in the present invention include: (1) an open source Massachusetts road dataset, which contains 1,171 aerial images with a size of 1,500 × 1,500 pixels, covering various types of geographical environments such as cities, suburbs, and rural areas; (2) the Potsdam public dataset of multidimensional remote sensing image data provided by ISPRS-Commission III, which consists of 38 high-resolution aerial images, each of which includes 4 bands and an image size of 6,000 × 6,000 pixels, and contains a total of 6 types of land features (impermeable layer, building, low vegetation, tree, car and other land), with a spatial resolution of 5 cm.

[0068] The image information extraction evaluation indicators selected by the present invention include: precision, which indicates the ratio of the number of pixels correctly classified for each type of objects in the object extraction results to the total number of pixels of that type; recall, which indicates the ratio of the number of pixels predicted to be correctly classified for each type of objects in the object extraction results to the total number of pixels of that type of objects in the label image; overall accuracy (OA), which indicates the percentage of all correctly classified objects in the extraction results; F1 score (F1Score), the harmonic mean of Precision and Recall can comprehensively evaluate the accuracy and comprehensiveness of the segmentation results; Intersection over Union (IoU), which indicates the degree of overlap between the pixel area occupied by the target object in the prediction result and the area occupied by the object in the label. The higher the values ​​of these indicators, the higher the extraction accuracy and the better the effect. In addition to the above performance evaluation method, the model calculation amount (FLOPs) is also introduced to measure the complexity of the model; the model parameter amount (Params) indicates the total number of parameters that need to be trained in model training. Lower values ​​of these indicators indicate smaller models, lower complexity, and faster inference speed.

[0069] like Figure 3As shown, in order to better demonstrate the applicability and robustness of SegFormer and Pc-Seg in road extraction tasks, the present invention uses remote sensing images in different situations to perform road extraction, namely, images of areas with high vegetation ratios and images of high-density buildings containing more spectral information that is consistent with the road. As can be seen from the figure, the use of SegFormer and Pc-Seg can better overcome the influence of the surrounding environment on road extraction work, and relatively completely extract road network information. In terms of using deep neural networks for road extraction tasks, it can be seen from the result figures (1) and (2) that SegFormer and Pc-Seg can overcome the influence of vegetation and its shadows on road extraction work, and better extract coherent road information. From the result (3), it can be seen that SegFormer and Pc-Seg can achieve different types of road extraction tasks. The result figures (4) and (5) show that SegFormer and Pc-Seg can overcome the more complex surface influences and better extract road information in images with more serious building influences by using deep learning network models to establish high-dimensional surface category mapping. At the same time, in result (2), the extracted information is compared with the image, and it can be found that both SegFormer and Pc-Seg extract information for road areas that are not marked in the ground truth. This shows that the use of deep learning to build an intelligent and automated ground feature extraction model can focus on some features that are ignored by visual interpretation, thereby obtaining more complete ground feature information. In terms of model comparison, it can be seen from results (1), (3), and (4) that misclassification and noise appear in SegFormer, while Pc-Seg effectively improves the above problems. The extraction results maintain the extraction accuracy and road network integrity. Although there are certain errors in the SegFormer and Pc-Seg models, they also extract information that is not presented in the real surface road labels, and overall obtain better road information extraction results. Compared with the SegFormer extraction results, the Pc-Seg road extraction accuracy is higher. In summary, in the road extraction task with the same three-dimensional remote sensing image data source, Pc-Seg performs better than SegFormer. This is because Pc-Seg can strengthen the category association in the target ground feature mapping space across images by introducing pixel contrast learning, better aggregate similar information, and enable the model to better construct the target ground feature map, thereby enhancing the performance of the model in the ground object extraction task.

[0070] like Figure 4As shown, from top to bottom, there are two scenes of urban road images affected by buildings, two scenes of urban roads affected by vegetation, and two scenes of rural road images. In order to verify the performance of the model in the image object extraction task, the present invention compares the relatively mature UNet (a typical encoder-decoder structure, often used as a comparison benchmark for semantic segmentation algorithms), DeepLab V3+ and HRNet (which realizes feature fusion at different resolutions, used to compare the performance of Pc-Seg in feature fusion), LMFFNet (a multi-feature fusion model), LWAFNet (a lightweight model for road extraction), and five models based on convolutional neural networks, as well as the Efficient Transformer based on the Transformer framework (a deep lightweight learning framework for remote sensing images proposed based on the Transformer framework) and Swin Transformer (a deep learning information extraction model for the classic Transformer framework) with the Pc-Seg model proposed in the present invention. The image size used in the comparative experiment is 512×512. By comprehensively comparing the road extraction results of the above models, it can be found that among all the models, Pc-Seg can better preserve the integrity and regularity of road information, and also has a good performance for inconspicuous road features. This is because the self-attention mechanism in Pc-Seg can automatically learn the dependencies between different positions in the image. The self-attention mechanism is not limited to local contextual information, but captures the long-distance dependencies between different positions in the input data to improve the expressiveness and prediction accuracy of the model. Moreover, the self-attention mechanism can pay different degrees of attention to the information at different positions according to the different characteristics of the input data, further improving the flexibility and feature expression ability of the model, and enhancing the integrity of the model's extraction of ground object information. At the same time, the pixel contrast learning used in Pc-Seg can make more full use of the spectral information in the image through cross-image feature analysis, enhance the model's performance in extracting ground object category features, and thus improve the accuracy of ground object information extraction.

[0071] As shown in Table 1, the F1-score of Pc-Seg is 82.54%, and the IoU is 69.61. Compared with the models Unet, DeeplabV3+, HRNet, LMFFNet, and LWAFNet under CNN, the F1-score is 6.66%, 4.02%, 7.43%, 1.64%, and 2.05% higher, respectively, and the IoU is 8.47%, 4.96%, 9.47%, 1.89%, and 2.32% higher, respectively. In the 3-channel dataset, the model proposed in the present invention can achieve higher accuracy improvement in road tasks compared with the CNN model. For Efficient Transformer and Swin Transformer, which are both under the Transformer framework, Pc-Seg improves by 1.12% and 0.92% in F1-score, and 1.35% and 1.18% in IoU, respectively, which indicates that Pc-Seg has higher model performance. In terms of FLOPs, it can be seen that Pc-Seg is higher than ET but lower than ST. However, in terms of comprehensive evaluation, the comprehensive performance of the model proposed in the present invention is the best. This is because CNN is used to extract shallow features of local features in the preliminary feature extraction process, providing richer image information for further road feature extraction. Then, the Transformer framework based on the self-attention mechanism can more fully extract the global information of the road in the image compared with CNN, so that the model can perform more efficient feature learning, thereby improving the model accuracy. Finally, Pc-Seg introduces contrastive learning. On the basis of giving full play to the advantages of the pre-trained model, by comparing sample pairs with similar properties, it emphasizes the similarity and difference between pixels, and by learning the relationship between pixels, it better captures the feature differences between roads and other objects, increases the adaptability of the model to road information extraction in remote sensing images, and improves the accuracy of road extraction. In summary, Pc-Seg can achieve good results in the task of intelligent road information extraction in 3D remote sensing images.

[0072] Table 1

[0073]

[0074] like Figure 5The comparison of the multi-classification information extraction results of the five models shown shows that Pc-Seg also performs well in multi-classification tasks. In the extraction of impermeable layers, buildings and trees where the category information is relatively obvious, the information extracted by Pc-Seg is relatively complete. Compared with other models, the "noise" phenomenon in the extraction results of Pc-Seg is significantly reduced. This is because Pc-Seg is a multi-level feature extraction model based on the self-attention mechanism. It can obtain the global ground object information in each slice image more completely through the self-attention mechanism, and then through multi-level fusion, the model captures the rough features of high resolution and the fine features of low resolution, further strengthening the feature expression in the image. In addition, for the expression of local information, Pc-Seg enhances the model's ability to extract local features of input data by using convolution kernels of different scales during the dimensional conversion process, which makes up for the deficiency of the self-attention mechanism in extracting local information to a certain extent. At the same time, the basic network of Pc-Seg is the pre-trained model SegFormer. Through the dimension conversion module and pixel comparison learning method, the existing various types of ground feature parameters in SegFormer are adjusted in a targeted manner for remote sensing images. On the premise of the original SegFormer feature parameters, the feature expression of ground object categories in remote sensing images is added, which further improves the applicability and extraction effect of the model in image information extraction tasks.

[0075] As shown in Table 2, the OA, MIoU and FWIoU comparisons of the five models show that Pc-Seg's ground feature extraction results are the highest, with OA, MIoU and FWIoU being 82.72%, 64.06% and 72.38% respectively. Among them: OA is 5.42%, 4.27%, 2.67% and 2.16% higher than DeeplabV3+, HRNet, UNet+++ and LMFFNet respectively, indicating that Pc-Seg has the best overall performance in multi-classification ground feature extraction tasks. MIoU is 7.34%, 7.22%, 1.57% and 0.85% higher than DeeplabV3+, HRNet, UNet+++ and LMFFNet respectively, indicating that Pc-Seg can perform well in all categories of ground feature information extraction. Compared with MIoU, the accuracy of FWIoU of all models has been improved to a certain extent. This is because FWIoU is an intersection-over-union ratio with land distribution weights, and the deep neural network used in the present invention has a higher extraction accuracy in land types such as unused land, buildings and trees, which account for a large proportion of information in the image, so FWIoU has a higher accuracy than MIoU. In terms of FWIoU, Pc-Seg is 6.20%, 7.80%, 2.97% and 1.39% higher than DeeplabV3+, HRNet, UNet+++ and LMFFNet, respectively, indicating that Pc-Seg's global information acquisition ability and feature clustering ability make it extremely suitable for the task of extracting land object information that accounts for a high proportion of images and has relatively obvious category information. Therefore, Pc-Seg shows a higher accuracy improvement effect in FWIoU. The experimental results show that Pc-Seg can excellently complete the task of extracting land object information in multi-dimensional images.

[0076] Table 2

[0077]

[0078]

[0079] It can be seen that the remote sensing ground object information extraction method based on pixel contrast learning proposed in the present invention has the following advantages compared with the related technologies: in the feature extraction process, CNN is used to perform shallow feature extraction on local features, which provides richer image information for further road feature extraction, and the Transformer framework based on the self-attention mechanism is used to fully extract the global information of the road in the image, so that the model can perform more efficient feature learning and improve the model recognition accuracy. Through pixel contrast learning, the category association in the target ground object feature mapping space is strengthened across images, and similar information is better aggregated, thereby improving the flexibility and feature expression ability of the model, enhancing the integrity of the model in ground object information extraction, and realizing high-precision remote sensing ground object information extraction under limited data resource conditions.

[0080] Accordingly, in a second aspect, the present invention provides a remote sensing object information extraction device based on pixel contrast learning, comprising: a remote sensing image preprocessing module, used to obtain the original remote sensing image, construct an input image matrix according to the attribute information of the original remote sensing image, and perform dimension conversion on the input image matrix; a pre-training module, used to extract multi-scale features of the input image matrix after dimension conversion using the SegFormer pre-training model, and fuse the multi-scale features to output a feature matrix, wherein the feature matrix contains object classification results; a classification result constraint module, used to combine pixel contrast loss and cross entropy loss into a supervised contrast segmentation loss, and use the supervised contrast segmentation loss to constrain the object classification results.

[0081] The above-mentioned device can be implemented by the remote sensing object information extraction method based on pixel contrast learning provided by the embodiment of the first aspect above. The specific implementation method can be found in the description of the embodiment of the first aspect, which will not be repeated here.

[0082] It is understood that the model structures and parameters described in the above embodiments are only examples. Those skilled in the art can also easily combine and adjust the structural features of the above embodiments according to the use requirements, and should not limit the concept of the present invention to the specific details of the above examples.

[0083] The present invention also provides an electronic device, including a processor and a memory, wherein the memory stores a plurality of instructions, and the processor is used to read the instructions and execute any one of the methods in the foregoing embodiment one. The processor and the memory can be connected via a bus or other means, taking the bus connection as an example. The processor can be a central processing unit (CPU). The processor can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components and other chips, or a combination of the above-mentioned various chips.

[0084] As a non-transient storage medium, the memory can be used to store non-transient software programs, non-transient computer executable programs and modules, such as the program instructions / modules corresponding to the remote sensing object information extraction method based on pixel contrast learning in the embodiment of the present invention. The processor executes various functional applications and data processing of the processor by running the non-transient software programs, instructions and modules stored in the memory, that is, realizing the remote sensing object information extraction method based on pixel contrast learning in the above method embodiment.

[0085] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0086] The present invention also provides a storage medium, wherein the storage medium stores a plurality of instructions, wherein the instructions can be loaded and executed by a processor, so that the processor can execute any one of the methods in Embodiment 1. The storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0087] Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent substitutions for some of the technical features therein; and these modifications or substitutions do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A remote sensing ground object information extraction method based on pixel contrast learning, characterized in that: include: Acquire an original remote sensing image, construct an input image matrix according to attribute information of the original remote sensing image, and perform dimension conversion on the input image matrix; The SegFormer pre-trained model is used to extract multi-scale features of the input image matrix after dimension conversion, and the multi-scale features are fused to output a feature matrix, which contains the ground object classification results; The pixel contrast loss and the cross entropy loss are combined into a supervised contrast segmentation loss, and the supervised contrast segmentation loss is used to constrain the object classification result.

2. The remote sensing object information extraction method based on pixel contrast learning according to claim 1 is characterized in that: The step of constructing an input image matrix according to the attribute information of the original remote sensing image and performing dimension conversion on the input image matrix further includes: Set the batch size to 3, load the original remote sensing image, and construct the input image matrix as (3, H, W, C), where H and W are the height and width of the original remote sensing image, and C is the dimension of the original remote sensing image; The channel conversion module is used to convert the dimension of the loaded original remote sensing image, that is, to convert the input matrix (3, H, W, C) into (3, H, W, 3).

3. The remote sensing object information extraction method based on pixel contrast learning according to claim 1 is characterized in that: The method of extracting multi-scale features of the input image matrix after dimension conversion by using the SegFormer pre-trained model and fusing the multi-scale features further includes: In the encoder stage, a four-layer hierarchical feature extraction module is used to extract multi-scale features of the original remote sensing image at scales of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 respectively; In the decoder stage, a multi-layer perceptron (MLP) is used to fuse the multi-scale features extracted during the encoding process. During the fusion process, all feature layers are upsampled to H / 4, W / 4, and the image feature matrix (3, H / 4, W / 4, Ncls) is output, where Ncls is the preset number of target land categories.

4. The remote sensing object information extraction method based on pixel contrast learning according to claim 3 is characterized in that: The cross entropy loss is used to compare the segmentation results of each original remote sensing image in a batch with the true value of the surface, and the pixel contrast loss is used to directly classify all features in the batch, that is, to perform category aggregation operations on the three-layer feature map.

5. A remote sensing ground object information extraction device based on pixel contrast learning, characterized in that: include: A remote sensing image preprocessing module is used to obtain the original remote sensing image, construct an input image matrix according to the attribute information of the original remote sensing image, and perform dimension conversion on the input image matrix; A pre-training module is used to extract multi-scale features of the input image matrix after dimension conversion using the SegFormer pre-training model, fuse the multi-scale features, and output a feature matrix, wherein the feature matrix includes a ground object classification result; The classification result constraint module is used to combine pixel contrast loss and cross entropy loss into supervised contrast segmentation loss, and use the supervised contrast segmentation loss to constrain the ground object classification result.

6. The remote sensing object information extraction device based on pixel contrast learning according to claim 5 is characterized in that: The remote sensing image preprocessing module is further used for: Set the batch size to 3, load the original remote sensing image, and construct the input image matrix as (3, H, W, C), where H and W are the height and width of the original remote sensing image, and C is the dimension of the original remote sensing image; The channel conversion module is used to convert the dimension of the loaded original remote sensing image, that is, to convert the input matrix (3, H, W, C) into (3, H, W, 3).

7. The remote sensing object information extraction device based on pixel contrast learning according to claim 5 is characterized in that: The pre-training module is further used to: In the encoder stage, a four-layer hierarchical feature extraction module is used to extract multi-scale features of the original remote sensing image at scales of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 respectively; In the decoder stage, a multi-layer perceptron (MLP) is used to fuse the multi-scale features extracted during the encoding process. During the fusion process, all feature layers are upsampled to H / 4, W / 4, and the image feature matrix (3, H / 4, W / 4, Ncls) is output, where Ncls is the preset number of target land categories.

8. The remote sensing ground object information extraction device based on pixel contrast learning according to claim 7 is characterized in that: The cross entropy loss is used to compare the segmentation results of each original remote sensing image in a batch with the true value of the surface, and the pixel contrast loss is used to directly classify all features in the batch, that is, to perform category aggregation operations on the three-layer feature map.

9. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores a plurality of instructions, and the processor is used to read the instructions and execute the remote sensing ground object information extraction method based on pixel contrast learning as described in any one of claims 1 to 4.

10. A storage medium, characterized in that: The storage medium stores a plurality of instructions, and the plurality of instructions can be read by a processor and executed by the remote sensing ground object information extraction method based on pixel contrast learning as described in any one of claims 1 to 4.

Images