Image processing method, device, equipment and storage medium

Abstract

The present application discloses an image processing method, device, equipment and storage medium. The method includes: obtaining a remote sensing image for a target area; if it is determined that the size data of the remote sensing image is greater than or equal to a set threshold, slicing the remote sensing image to generate at least one remote sensing image slice; for each remote sensing image slice in the at least one remote sensing image slice, performing an object query on the remote sensing image slice based on a pre-trained first image processing model, determining the first building instance in the remote sensing image slice, and generating a first mask image representing the outline of the first building; based on each first mask image, generating a target mask image representing the outline of the target building in the target area. In this way, by utilizing the idea of ​​object query to locate a single building as an instance, single-building extraction can be performed directly, effectively alleviating the problem of building adhesion.

Description

Technical Field

[0001] The present application relates to the technical field of remote sensing image processing, and in particular to an image processing method, device, equipment and storage medium. Background Art

[0002] With the acceleration of urbanization, buildings are an important part of the urban structure, and the rapid acquisition of their information is crucial in urban planning, environmental monitoring, and disaster management. Remote sensing images have become the main means of obtaining building information because they can provide large-scale, high-time data. At present, the mainstream technology relies on the semantic segmentation method in computer vision, which performs pixel-by-pixel analysis on remote sensing images to achieve accurate pixel-level classification to extract building outlines.

[0003] Affected by observation conditions and scenes, remote sensing images have complex and diverse backgrounds and blurred boundaries compared to natural images, which naturally limits their ability to capture texture details, resulting in blurred edges of buildings. On the other hand, due to the influence of sunlight angle and shooting angle, shadows and occlusions are easily generated between adjacent buildings. These factors not only increase the complexity of edge pixel classification, making the contour inconsistent with the actual situation, but also easily lead to the adhesion of adjacent building contours, making it difficult to achieve clear, single-story building top contour segmentation. Summary of the invention

[0004] In view of this, the embodiments of the present application provide an image processing method, apparatus, device and storage medium, aiming to solve the problem of how to effectively achieve accurate single-building segmentation of the top contour of a building and avoid building adhesion.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides an image processing method, the method comprising:

[0007] Acquire remote sensing images of target areas;

[0008] If it is determined that the size data of the remote sensing image is greater than or equal to the set threshold, slicing the remote sensing image to generate at least one remote sensing image slice;

[0009] For each remote sensing image slice of the at least one remote sensing image slice, performing an object query on the remote sensing image slice based on a pre-trained first image processing model, determining a first building instance in the remote sensing image slice, and generating a first mask image representing an outline of the first building;

[0010] Based on each of the first mask images, a target mask image representing the contour of the target building in the target area is generated.

[0011] In some embodiments, the method further comprises:

[0012] Performing corner point detection on the first building segmentation mask corresponding to the first mask image based on the second image processing model to generate a plurality of corner points corresponding to the first building outline and position information of each corner point;

[0013] Based on the position information of each corner point and the set arrangement rules, the corner points are arranged to construct a vector polygon;

[0014] Determine first building vector outline data based on the vector data of the vector polygon;

[0015] Based on each of the first building vector outline data, target building vector outline data representing the target building outline is generated.

[0016] In some embodiments, the method further comprises:

[0017] If it is determined that the size data of the remote sensing image is smaller than the set threshold, an object query is performed on the remote sensing image based on the first image processing model to determine the target building instance corresponding to the target area, and a target mask image representing the outline of the target building is generated.

[0018] In some embodiments, the pre-trained first image processing model includes a pre-trained feature extractor and a pre-trained third image processing model, the pre-trained third image processing model includes a pixel decoder and a transform decoder, and the method further includes:

[0019] Extracting features from the remote sensing image slices based on the rotation variable window attention layer of the pre-trained feature extractor to generate multi-scale image feature data;

[0020] Upsampling the multi-scale image feature data based on the pixel decoder to generate a multi-scale high-resolution pixel embedding vector;

[0021] Performing an object query on the multi-scale high-resolution pixel embedding vector based on the self-attention model and the cross-attention model of the transform decoder to determine the first building instance; and generating a pixel category label of the remote sensing image slice and a first building segmentation mask of the first building instance;

[0022] The first mask image is generated based on the transform decoder, the pixel class labels and the first building segmentation mask.

[0023] In some embodiments, the method further comprises:

[0024] Based on multiple attention layers, a backbone network of the feature extractor is constructed, wherein the attention layer adopts a rotation variable window attention model;

[0025] The backbone network is pre-trained based on the pre-processed first remote sensing image sample set until the pre-trained feature extractor is generated.

[0026] In some embodiments, the method further comprises:

[0027] Acquire a second remote sensing image sample set after labeling, wherein the second remote sensing image sample set includes a training sample set;

[0028] Predicting the transform decoder based on the training sample set to generate prediction mask data;

[0029] The transformer decoder is pre-trained based on the predicted mask data, the real labeled data corresponding to the training sample set and the first loss function until a pre-trained third image processing model is generated.

[0030] In some embodiments, the second image processing model includes a pre-trained adversarial neural network generator and a corner point detection model, and performing corner point detection on the first mask image based on the second image processing model to generate multiple corner points corresponding to the first building outline and position information of each corner point includes:

[0031] modifying the first building segmentation mask based on the generator;

[0032] Corner point detection is performed on the corrected first building segmentation mask based on the corner point detection model to generate the multiple corner points and position information corresponding to each corner point.

[0033] In some embodiments, the adversarial neural network further includes: an initial generator and an initial discriminator, and the method further includes:

[0034] Based on the initial generator, the mask sample data representing the contours of each building is corrected to generate initial corrected mask data;

[0035] Based on the initial discriminator, determining a discrimination result between the initial revised mask data and the real revised sample data corresponding to the mask sample data;

[0036] Determine a second loss function based on the discrimination result, the adversarial loss, the reconstruction loss and the regularization loss;

[0037] Based on the second loss function, the initial discriminator and the initial generator are trained iteratively until a trained generator is generated.

[0038] In a second aspect, an embodiment of the present application provides an image processing device, the image processing device comprising:

[0039] An acquisition module, used for acquiring remote sensing images of a target area;

[0040] A slicing module, configured to slice the remote sensing image to generate at least one remote sensing image slice if it is determined that the size data of the remote sensing image is greater than or equal to a set threshold;

[0041] an object query module, configured to perform an object query on each remote sensing image slice of the at least one remote sensing image slice based on a pre-trained first image processing model, determine a first building instance in the remote sensing image slice, and generate a first mask image representing an outline of the first building;

[0042] A generating module is used to generate a target mask image representing the contour of the target building in the target area based on each of the first mask images.

[0043] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory for storing a computer program that can be run on the processor, wherein the processor, when used to run the computer program, performs the steps of the method described in the first aspect of the embodiment of the present application.

[0044] In a fourth aspect, an embodiment of the present application provides a computer storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in the first aspect of the embodiment of the present application are implemented.

[0045] The technical solution provided by the embodiment of the present application is an image processing method, which also includes: acquiring a remote sensing image of a target area; if it is determined that the size data of the remote sensing image is greater than or equal to a set threshold, slicing the remote sensing image to generate at least one remote sensing image slice; for each remote sensing image slice in the at least one remote sensing image slice, performing an object query on the remote sensing image slice based on a pre-trained first image processing model, determining a first building instance in the remote sensing image slice, and generating a first mask image representing the outline of the first building; based on each first mask image, generating a target mask image representing the outline of the target building in the target area.

[0046] In this way, this application introduces a pre-trained first image processing model by utilizing the idea of ​​object query. The model can perform object query and locate a single building as an instance, thereby directly extracting single-family buildings, effectively alleviating the problem of building adhesion. At the same time, before extracting the building contour, the first image processing model is pre-trained, which also improves the accuracy of building contour extraction and effectively alleviates the problem of building adhesion. At the same time, compared with the prior art, the first image processing model in this application can be applied to landform scenes in different regions and remote sensing images in different seasons, and has strong generalization and adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A schematic diagram of the process of an image processing method provided in an embodiment of the present application;

[0048] Figure 2 A network architecture diagram for extracting buildings from remote sensing images provided in an embodiment of the present application;

[0049] Figure 3 A schematic diagram of the secondary optimization of the adversarial neural network provided in the embodiment of the present application;

[0050] Figure 4 A flowchart of a method for accurately extracting the top contour of a building from a remote sensing image provided as an application example of this application;

[0051] Figure 5 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0052] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The present application will be described in further detail below in conjunction with the accompanying drawings and examples. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present application. The terms used herein in the specification of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application so that the above-mentioned objects, features and advantages of the present application can be more clearly understood. The present application will be described in further detail below in conjunction with the accompanying drawings.

[0054] The embodiment of the present application provides an image processing method. Here, the image processing method may be in the process of extracting the outline of a building, and further, may be in the process of extracting the outline of the top of a building.

[0055] Here, the method comprises the following steps:

[0056] Step 110: Acquire a remote sensing image of the target area.

[0057] It is understood that remote sensing images refer to digital representations of information on the earth's surface obtained by satellites, sensors, or other sensors. For example, Figure 2 The input image on the left is a remote sensing image of a certain area. The target area can include: urban planning areas, specific building complexes, agricultural land, forest areas or any other geographical range that needs to be analyzed through remote sensing images.

[0058] In an optional embodiment, the remote sensing image can be taken by a satellite or a drone and acquired by the image processing device based on network transmission. In practical applications, in order to better display the details of the target area, high-resolution remote sensing images are usually used. High-resolution remote sensing images can provide clearer details, especially in the process of extracting building outlines.

[0059] Step 120: If it is determined that the size data of the remote sensing image is greater than or equal to the set threshold, the remote sensing image is sliced ​​to generate at least one remote sensing image slice.

[0060] It is understandable that when processing large remote sensing images, directly inputting the entire image into the image processing model may lead to excessive consumption of computing resources and affect the accuracy of the image processing model. Based on this, in order to avoid these problems, the image slicing method is usually used to reduce the amount of data processed at a single time.

[0061] Here, a threshold of the image size (eg, the number of pixels of width and height) is set. If the size of the remote sensing image exceeds the set threshold, it needs to be sliced ​​to generate at least one remote sensing image slice.

[0062] Exemplarily, the size of the remote sensing image can be sliced ​​by sliding window slicing to generate at least one remote sensing image slice. Sliding window slicing refers to moving a window on the image, and each movement captures a portion of the image as a slice.

[0063] Step 130: For each remote sensing image slice in at least one remote sensing image slice, perform an object query on the remote sensing image slice based on a pre-trained first image processing model, determine a first building instance in the remote sensing image slice, and generate a first mask image representing the outline of the first building.

[0064] It is understandable that after slicing the remote sensing image, each remote sensing slice is separately input into the first image processing model for processing, that is, for each remote sensing image slice in at least one remote sensing image slice, an object query is performed based on the pre-trained first image processing model to determine the first building instance in the remote sensing image slice, and generate a first mask image representing the outline of the first building, so that multiple slices can be processed in parallel to speed up the processing speed. Moreover, based on the object query method, single-building extraction can be performed directly, and the problem of building adhesion can be effectively alleviated.

[0065] It can be understood that object query refers to finding and identifying a specific object type in an image. In the embodiment of the present application, the object type is a building.

[0066] Here, the building outline refers to the actual edge or boundary of the building, which is the projected shape of the building in the two-dimensional image.

[0067] Here, the first mask image refers to a binary image or a probability map, which is used to identify and highlight a specific building in a remote sensing image slice. In an embodiment of the present application, in the mask image, the pixels where the building is located are usually marked with a specific value (such as 1 or a high probability value), and these pixels constitute the outline or internal area of ​​the building. The pixels in the non-building area will be marked with another value (such as 0 or a low probability value). For example, Figure 2 As shown in the output image on the right, the red part in the mask image marks the outline of the building.

[0068] It can be understood that the remote sensing image slice includes at least one first building instance, and the first mask image represents at least one first building outline of the remote sensing image slice.

[0069] Step 140: Based on each of the first mask images, generate a target mask image representing the contour of the target building in the target area.

[0070] It can be understood that after determining the first mask image of the first building in each remote sensing slice, based on each first mask image, the target building instance data of the target area can be determined, and the target mask image of the target building outline can be determined.

[0071] For example, if the sliding window technique is used for slicing, after obtaining the first mask image of each remote sensing image slice, these local first mask images are integrated to form a complete mask covering the entire target area, namely, a target mask image.

[0072] It can be understood that after the remote sensing image is sliced, each remote sensing slice is individually input into the first image processing model for processing, that is, for each remote sensing image slice in at least one remote sensing image slice, an object query is performed based on the pre-trained first image processing model to determine the first building instance in the remote sensing image slice, and generate a first mask image representing the outline of the first building, so that multiple slices can be processed in parallel to speed up the processing speed.

[0073] In this way, this application introduces a pre-trained first image processing model by utilizing the idea of ​​object query. The model can perform object query and locate a single building as an instance, thereby directly extracting single-family buildings, effectively alleviating the problem of building adhesion. At the same time, before extracting the building contour, the first image processing model is pre-trained, which also improves the accuracy of building contour extraction and effectively alleviates the problem of building adhesion. At the same time, compared with the prior art, the first image processing model in this application can be applied to landform scenes in different regions and remote sensing images in different seasons, and has strong generalization and adaptability.

[0074] In some embodiments, the method further comprises:

[0075] Performing corner point detection on the first building segmentation mask corresponding to the first mask image based on the second image processing model to generate multiple corner points corresponding to the first building outline and position information of each corner point;

[0076] Based on the position information of each corner point and the set arrangement rules, the corner points are arranged to construct a vector polygon;

[0077] Determine first building vector outline data based on the vector data of the vector polygon;

[0078] Based on each of the first building vector outline data, target building vector outline data representing the outline of the target building is generated.

[0079] It should be noted that the above object query of remote sensing image slices based on the first image processing model treats a single building as an instance for positioning, thereby avoiding the problem of building adhesion caused by directly using semantic segmentation. On this basis, further, in order to improve the accuracy of building contour extraction, especially in the process of extracting the top contour of the building, the first mask image is optimized again based on the second image processing model, which can not only realize the automatic extraction of the top contour of a single building accurately, but also effectively alleviate the problem of blurred building extraction boundaries caused by the resolution of remote sensing images.

[0080] It can be understood that in order to more accurately extract the building outline, the embodiment of the present application performs corner point detection on the first building segmentation mask of the first mask image based on the second image processing model, generates multiple corner points corresponding to the first building outline and the position information of each corner point; arranges each corner point based on the position information of each corner point and the set arrangement rules to construct a vector polygon. The vector polygon further reconstructs the first building outline, and based on the vector data of the vector polygon, a more refined building outline representation, namely the first building vector outline data, can be obtained.

[0081] Exemplarily, the detected corner points can be arranged in a clockwise direction, and redundant points close to the edge can be filtered out by line segment simplification, and finally an instance representation is generated for each building outline. Each instance is represented by vector data consisting of the coordinates of several constituent polygons, and the vector data is the first building vector outline data.

[0082] Finally, all the first building vector outline data are integrated to generate target building vector outline data representing the entire target building outline.

[0083] In this way, by combining the object query of the first image processing model and the corner point detection of the second image processing model, the embodiment of the present application not only solves the problem of building adhesion, but also significantly improves the accuracy of building contour extraction, and realizes automatic and high-precision building vector contouring. It is suitable for landform scenes in different regions and remote sensing images in different seasons, and has strong generalization and adaptability.

[0084] In some embodiments, the method further comprises:

[0085] If it is determined that the size data of the remote sensing image is less than a set threshold, an object query is performed on the remote sensing image based on the first image processing model to determine the target building instance corresponding to the target area, and a target mask image representing the outline of the target building is generated.

[0086] It can be understood that if it is determined that the size data of the remote sensing image is smaller than the set threshold, it means that the size of the remote sensing image can be directly processed without slicing, that is, the remote sensing image can be directly queried for an object based on the first image processing model, the target building instance corresponding to the target area can be determined, and a target mask image representing the outline of the target building can be generated.

[0087] In some embodiments, the pre-trained first image processing model includes a pre-trained feature extractor and a pre-trained third image processing model, and the pre-trained third image processing model includes a pixel decoder and a transform decoder.

[0088] Here, the feature extractor may be a feature extractor ViTAE (Vision Transformer with AutoEncoder), and the third image processing model may be a building extraction base model Mask2Former, which includes a pixel decoder (Pixel Decoder) and a transform decoder (Transformer Decoder).

[0089] In some embodiments, the method further comprises:

[0090] Based on the rotation variable window attention layer of the pre-trained feature extractor, feature extraction is performed on remote sensing image slices to generate multi-scale image feature data;

[0091] Upsample multi-scale image feature data based on a pixel decoder to generate multi-scale high-resolution pixel embedding vectors;

[0092] Performing object query on the multi-scale high-resolution pixel embedding vector based on the self-attention model and the cross-attention model of the transform decoder to determine the first building instance; and generating the pixel category label of the remote sensing image slice and the first building segmentation mask of the first building instance;

[0093] A first mask image is generated based on the transform decoder, the pixel class labels, and the first building segmentation mask.

[0094] For example, Figure 2 As shown, Figure 2 The network architecture diagram of building extraction from remote sensing images shows two satellite images on the left and right sides of the figure, representing the input image and the output image respectively. The input image is usually an unprocessed remote sensing image, and the output image is the result obtained after processing by the first image processing model, that is, the first mask image of the first building outline.

[0095] For example, in Figure 2 Backbone Network: Responsible for extracting features from the input image. The backbone network used in the embodiment of the present application is the feature extractor ViTAE.

[0096] Pixel Decoder: The pixel decoder receives the multi-scale feature maps from the backbone network and restores them to the resolution of the original image through a series of upsampling and deconvolution layers. In this process, the pixel decoder uses skip connections to introduce high-resolution contextual information from the backbone feature extractor to enhance the semantic information and spatial details of the features, thereby generating multi-scale high-resolution pixel embedding vectors.

[0097] Transformer Decoder is Transformer Decoder: It contains multiple mask attention layers, such as Lx in the figure, Lx includes: Layer1, Layer2, Layer3, ..., LayerL. These layers use the mask attention mechanism to process object queries, and combine the local feature information in the pixel embedding to assign corresponding category labels and instance masks to each pixel, that is, the instance category and segmentation mask in the figure. The calculation formula of the mask attention layer is as follows:

[0098]

[0099] in: Indicates The query features of the layer, represents the query features of the previous layer, represents the mask matrix of the previous layer, represents the query matrix, represents the transpose of the key matrix, Represents the normalized scaling factor.

[0100] Output: The result after processing by the Transformer decoder is a segmentation map containing the category to which each pixel belongs and the corresponding instance mask. This information can be used to generate accurate building outlines.

[0101] In practical applications, the feature extractor is the feature extractor ViTAE corresponding to the backbone network. The third image processing model first obtains a multi-scale feature map through the backbone feature extractor ViTAE; then the multi-scale feature map is used as the input of the pixel decoder; finally, the Transformer transform decoder is used to operate the pixel decoder to process object queries.

[0102] In some embodiments, the method further comprises:

[0103] Based on multiple attention layers, a backbone network of the feature extractor is constructed, and the attention layer adopts a rotating variable window attention model;

[0104] The backbone network is pre-trained based on the pre-processed first remote sensing image sample set until a pre-trained feature extractor is generated.

[0105] It is understandable that before extracting features from a remote sensing image based on a feature extractor, the feature extractor needs to be pre-trained to ensure the accuracy of feature extraction.

[0106] Here, the backbone network of the feature extractor consists of multiple attention layers that adopt the Rotation Variable Window Attention model (RVWA).

[0107] It is understandable that the feature extractor ViTAE is improved from Vision Transformer and consists of several repeatedly stacked attention layers. Based on the characteristics of remote sensing images, the rotating variable window attention mechanism is used to replace the full attention mechanism in the original attention layer, so that rich contextual information can be extracted from the generated different windows to learn better object representation, which can effectively cope with the large size characteristics of remote sensing images and the arbitrary orientation of buildings. The calculation formula of the rotating variable window attention mechanism is as follows:

[0108] First, given the input features in window w , prediction window transformation parameters: scaling factor , offset and rotation angle :

[0109]

[0110] in is a linear layer, is a nonlinear activation layer, is the global average pooling layer.

[0111] Second, according to the above predicted scaling factor , offset and rotation angle , transform the coordinates of the window to obtain the transformed window size.

[0112] Here, set and The coordinates of the upper left corner and lower right corner of the window, respectively, and the transformed coordinates and The calculation is as follows:

[0113]

[0114]

[0115] in is the coordinate of the window center point before transformation, and is the offset in the x and y directions, are the scaling factors in the x and y directions, , , , They are the horizontal and vertical distances from the upper left corner and lower right corner to the center of the window, , , , are the transformed coordinates.

[0116] Finally, the self-attention calculation is performed within the transformed window, and the calculation formula is:

[0117]

[0118] in is the calculated feature, is the self-attention layer, , , Respectively represent the query, key, and value features within the window. is the scaling factor used to adjust The sensitivity of the function.

[0119] In the embodiment of the present application, the feature extractor includes multiple attention layers, which use the rotation variable window attention model (RVWA). RVWA allows the model to have higher flexibility in processing information in different directions, especially when processing remote sensing images, it can accurately and efficiently extract features from remote sensing images, which helps to improve the accuracy and reliability of remote sensing image analysis.

[0120] In some embodiments, the method further comprises:

[0121] Acquire a second remote sensing image sample set after labeling, where the second remote sensing image sample set includes a training sample set;

[0122] Predicting the transform decoder based on the training sample set to generate prediction mask data;

[0123] The transformer decoder is pre-trained based on the predicted mask data, the real labeled data corresponding to the training sample set, and the first loss function until a pre-trained third image processing model is generated.

[0124] It is understandable that in order to improve the accuracy of the image processing model, the third image processing model needs to be pre-trained. Specifically, the third image processing model can be pre-trained by combining the label guidance strategy to accelerate network optimization and construct a third image processing model based on object query.

[0125] Here, the label-guided strategy means that in addition to using learnable object queries during training, class embeddings of known categories are introduced as additional queries, and the true annotated masks are input into the Transformer decoder layer as attention masks. The purpose of this is to allow the model to better learn how to reconstruct the true masks and labels.

[0126] Here, before pre-training, the second remote sensing image samples need to be labeled. The labeled second remote sensing image sample set can be divided into a training sample set and a verification set according to a preset ratio. For example, the ratio of the training set to the verification set can be 8 to 2.

[0127] For example, remote sensing image data is collected for building sample annotation and processing to obtain a high-quality building sample data set, namely, the aforementioned second remote sensing image sample set, including: (1) collecting high-resolution satellite image data, and combining expert experience and knowledge to construct building sample collection specifications and interpretation sign manuals to assist sample collection; (2) based on auxiliary information such as building sample collection specifications and interpretation sign manuals, a manual method is used to annotate the building outlines in the image in the form of polygons, and a preliminary quality inspection is performed; (3) the annotated images and label information are sliced ​​according to the COCO standard data format, the slice size is set to 512*512, and 50% of the overlapping area is retained between slices, thereby initially forming a building sample data set; (4) the sample data set is cleaned and quality inspected again, and invalid polygons, samples that do not meet the requirements such as out-of-bounds annotation contours, etc. are deleted, and problems such as inaccurate annotation boxes and incorrect annotation attributes are corrected. (5) The cleaned sample data set is randomly divided into a training set and a validation set in a ratio of 8:2.

[0128] In an embodiment of the present application, the third image processing model is pre-trained based on the training sample set, and the predicted mask data is generated, and the transformer decoder is pre-trained based on the predicted mask data, the real labeled data corresponding to the training sample set and the first loss function, and the parameters of the transformer decoder are updated until the pre-trained third image processing model is generated. Specifically, the first loss function calculates the gap between the predicted mask data and the real labeled data of the training sample set, and according to the loss output by the first loss function, the parameters of the transformer decoder are adjusted to reduce the error until the pre-trained third image processing model is generated.

[0129] Exemplarily, firstly, the predicted segmentation mask generated by the third image processing model based on the training set is matched using bipartite graph matching. With real annotation Matching is performed to calculate the loss and update the model parameters. Specifically, the cost matrix is ​​calculated based on the intersection over union (IoU), and then the Hungarian algorithm is used to find the matching solution that minimizes the cost. The specific steps are as follows:

[0130] First, based on the intersection over union (IoU), the predicted mask is calculated With real annotation The cost matrix between , the calculation formula is as follows:

[0131]

[0132] in represents the computational cost, represents the i-th prediction mask, represents the jth true annotation .

[0133] Use the Hungarian algorithm to find the best match and get the predicted mask With real annotation The optimal matching result between them, the optimization objective function is as follows:

[0134]

[0135] in, is the matching loss function, is the ith true value, It is arranged The i-th predicted value after is the set of all possible permutations.

[0136] According to the best matching result, the predicted mask Assigned to the corresponding ground truth , used to calculate the first loss function, which uses binary cross entropy loss and The loss combination is calculated as follows:

[0137]

[0138]

[0139]

[0140] in, is the total loss, and is the weight coefficient used to balance the two losses, is the binary cross entropy loss, yes loss, represents the true label of the i-th pixel, represents the probability that the i-th pixel is predicted to be a building.

[0141] Thus, the first image processing model of the embodiment of the present application directly extracts single-family buildings using the idea of ​​object query, and adds a label guidance strategy for training during the pre-training process to improve the model convergence speed, which not only improves the accuracy of building contour extraction, but also effectively alleviates the problem of building adhesion.

[0142] In some embodiments, the second image processing model includes a pre-trained adversarial neural network generator and a corner detection model.

[0143] Here, the second image processing model includes a pre-trained adversarial neural network (GAN) generator and a corner detection model. Adversarial neural networks usually consist of two main parts: a generator and a discriminator. It mainly trains the generative model through two competing neural networks - the generator and the discriminator - so that the generator can generate realistic samples.

[0144] Here, the pre-trained first image processing model is used for inference to obtain the mask extraction result of the building contour of the remote sensing image, that is, the first mask image, and then the adversarial neural network is used to perform secondary optimization on the mask contour to obtain a more accurate contour that conforms to the building morphology.

[0145] In some embodiments, performing corner point detection on the first mask image based on the second image processing model to generate multiple corner points corresponding to the first building outline and position information of each corner point includes:

[0146] Modifying the first building segmentation mask based on the generator;

[0147] Corner point detection is performed on the corrected first building segmentation mask based on the corner point detection model to generate multiple corner points and position information corresponding to each corner point.

[0148] Here, the corner detection model can adopt a convolutional neural network, which has five layers.

[0149] In practical applications, a convolutional neural network is used to detect corners of the first building segmentation mask modified by the generator of the adversarial neural network. The convolutional neural network model consists of five convolutional layers, which receives the first building segmentation mask as input and outputs a corner probability map. Pixels with a value higher than a threshold in the probability map can be considered as valid corners of the building polygon, thereby generating multiple corner points and determining their position information.

[0150] In this way, the embodiment of the present application can perform secondary optimization on the initial building contours extracted from the remote sensing image by the first image model by adopting a second image model based on an adversarial neural network, which can effectively alleviate the problem of blurred building extraction boundaries caused by the resolution of the remote sensing image. It is thus suitable for landform scenes in different regions and remote sensing images in different seasons, and has strong generalization and adaptability.

[0151] In some embodiments, the adversarial neural network further includes: an initial generator and an initial discriminator, and the method further includes:

[0152] Based on the initial generator, the mask sample data representing the building outline is corrected to generate initial corrected mask data;

[0153] Based on the initial discriminator, determining a discrimination result between the initial corrected mask data and the real corrected sample data corresponding to the masked sample data;

[0154] Determine the second loss function based on the discrimination result, adversarial loss, reconstruction loss and regularization loss;

[0155] Based on the second loss function, the initial discriminator and the initial generator are trained iteratively until a trained generator is generated.

[0156] For example, Figure 3 As shown in the figure, the adversarial neural network consists of two networks, a generator (EG) and a discriminator (D). The generator modifies the first mask image to generate a regularized version of the segmentation mask, and the discriminator is used to test the similarity between the generated regularized contour and the actual contour. With the training iterations, the two networks compete with each other in optimization, and finally the generator is able to generate a regular contour that is closer to the actual shape of the building.

[0157] In the actual training process, Figure 3 As shown in Figure 1, the adversarial neural network also includes a reconstructor, which receives the input labeled mask (y) and tries to reconstruct the original mask data. Its goal is to restore the original mask data as much as possible to verify the effect of the generator.

[0158] For example, Figure 3 As shown, Figure 3 This is a schematic diagram of the secondary optimization outline architecture of the adversarial neural network according to an embodiment of the present application. Taking the pre-training process as an example, the various modules and functions of the architecture are introduced below.

[0159] Here, the adversarial network consists of three main parts: Generator, Reconstructor and Discriminator.

[0160] Generator: The generator (EG) receives the input mask (x) corresponding to the input remote sensing image z and converts it into a revised mask data. The main function of the generator is to generate mask data that is closer to the actual building form.

[0161] Reconstructor: The reconstructor (ER) receives the input labeled mask (y) and tries to reconstruct the original mask data. Its goal is to restore the original mask data as much as possible to verify the effect of the generator.

[0162] exist Figure 3 In , the latent variable space is located between the generator and the reconstructor, which means that the generator and the reconstructor share the same latent space. This design allows the generator and the reconstructor to interact in the same latent space, thereby realizing the functions of image generation and reconstruction.

[0163] Regularized Decoder: The regularized decoder F connects the generator and the discriminator. It takes as input the vector in the latent space and outputs the generated image. This decoder contains convolutional layers, batch normalization and ReLU activation functions, and finally the sigmoid activation function (conv 1×1, sigmoid).

[0164] Discriminator: The discriminator (D) is responsible for distinguishing between the modified mask data generated by the generator and the real mask data. It evaluates whether the mask data generated by the generator is realistic enough. Finally, it outputs a probability value between 0 and 1, indicating the authenticity of the image (true or false as shown in the figure).

[0165] Training process: There is an adversarial relationship between the generator and the discriminator, which are jointly optimized to achieve the best results. The generator tries to generate more realistic mask data, while the discriminator tries to distinguish the generated data from the real data.

[0166] Loss function: The loss function consists of adversarial loss, reconstruction loss, and regularization loss. Adversarial loss measures the adversarial relationship between the generator and the discriminator. Reconstruction loss measures the performance of the generator and the reconstructor. Regularization loss is used to encourage the generator to generate more reasonable contours.

[0167] Specifically, the calculation formula is as follows:

[0168]

[0169] in For the total loss, , , , , are weight coefficients used to balance various losses. For adversarial loss, and To rebuild the losses, and is the regularization loss, , , They represent the generator, reconstruction path and discriminator respectively.

[0170] Specifically, the adversarial loss is composed of the generator and the discriminator The adversarial loss is composed of:

[0171]

[0172]

[0173]

[0174] in represents adversarial loss, is the loss from the generator, is the loss from the discriminator, Represents all true labels expectations, Represents all input images and noise joint expectations, Representation Discriminator For the generator Generated regularized mask The discrimination result of .

[0175] The reconstruction loss is composed of and generator The cross entropy loss is composed of, and the calculation formula is as follows:

[0176]

[0177]

[0178] in To reconstruct the path of loss, For the generator of loss, represents the pixel value of the true mask, represents the pixel value of the input mask, Represents the reconstruction path In pixels The predicted value at Represents the regularized mask generated by the generator G at pixel The predicted value at .

[0179] The regularized loss consists of Potts loss and normalized clipping loss, and the calculation formula is as follows:

[0180]

[0181]

[0182] in Potts loss, used to encourage the generator Produces building outlines that align with building boundaries observed in the intensity image, is the normalized clipping loss function, used to further refine the generator As a result, Representation Generator Produced kind The vectorized representation of the segmentation mask after represents the affinity matrix, Normalized affinity matrix.

[0183] Below, the embodiment of the present application is described in detail with reference to an application example.

[0184] Aiming at the problems of inaccurate boundaries and contour adhesion in the extraction of building contours from high-resolution remote sensing images, this application example proposes a remote sensing image processing method based on object query.

[0185] Taking high-resolution optical remote sensing satellite images as input, the outline of buildings is represented by masks. First, feature information is obtained based on a large-scale pre-trained backbone feature extractor, and a single building is located as an instance based on object query. Then, pixel-by-pixel classification is performed to obtain outline information, and finally, the building outline is optimized twice through an adversarial neural network. This solution can accurately realize the automatic extraction of the top outline of a single building.

[0186] like Figure 4 As shown, Figure 4 This is a flow chart of the method for accurately extracting the top contour of a building from a remote sensing image, which specifically includes the following steps:

[0187] Step 401: Large-scale remote sensing image data set, image slices and backbone feature extraction network pre-training.

[0188] Specifically, it mainly includes collecting large-scale remote sensing image data sets, image slicing, and pre-training the backbone feature extraction network. The details are as follows:

[0189] Step 1-1: The feature extractor ViTAE is improved from Vision Transformer and consists of several repeatedly stacked attention layers. Based on the characteristics of remote sensing images, a rotating variable window attention mechanism is used to replace the full attention mechanism in the original attention layer, so that rich contextual information can be extracted from different generated windows to learn better object representations, which can effectively cope with the large size characteristics of remote sensing images and the arbitrary orientation of buildings. The calculation formula of the rotating variable window attention mechanism is as follows:

[0190] Step 1-1-1: First predict the window transformation parameters, given the input features in window w , the scaling factor of the prediction window , offset and rotation angle :

[0191]

[0192] in is a linear layer, is a nonlinear activation layer, is the global average pooling layer.

[0193] Step 1-1-2: According to the predicted scaling factor, offset and rotation angle, transform the coordinates of the window to obtain the transformed window size. and The coordinates of the upper left corner and lower right corner of the window, respectively, and the transformed coordinates and The calculation is as follows:

[0194]

[0195]

[0196] in is the coordinate of the window center point before transformation, and is the offset in the x and y directions, are the scaling factors in the x and y directions, , , , They are the horizontal and vertical distances from the upper left corner and lower right corner to the center of the window, , , , are the transformed coordinates.

[0197] Step 1-1-3: Perform self-attention calculation in the transformed window, and the calculation formula is:

[0198]

[0199] in is the calculated feature, is the self-attention layer, , , Respectively represent the query, key, and value features within the window. is the scaling factor used to adjust The sensitivity of the function.

[0200] Step 1-2: Pre-train ViTAE. First, collect and prepare a large-scale remote sensing image dataset. Ensure that the dataset contains diverse remote sensing images, covering various scenes and objects. According to the input requirements of the model, divide the images into small patches of fixed size, and perform necessary data enhancement on the images, such as scaling, normalization, and enhancement.

[0201] Step 1-3: Input the enhanced training sample set into the backbone feature extractor ViTAE for pre-training. The specific steps are as follows:

[0202] Step 1-3-1: Split the input image into non-overlapping patches and convert each patch into a vision token through an embedding layer.

[0203] Step 1-3-2: According to the predefined mask ratio, a certain number of vision tokens are randomly selected and masked. These masked tokens will be used as targets to be predicted later.

[0204] Step 1-3-3: Input the unmasked tokens into the encoder part of the ViTAE model for feature extraction.

[0205] Step 1-3-4: Use the decoder part of the model to reconstruct the masked patch area by combining the learnable mask tokens and the features extracted by the encoder, and optimize the model by minimizing the difference between the reconstructed area and the true masked area in pixel space.

[0206] Step 402: Labeling and quality inspection of building samples, sample standardization processing, and building sample data set.

[0207] Specifically, the details are as follows:

[0208] Step 2-1: Collect high-resolution satellite image data and combine expert experience and knowledge to construct building sample collection specifications and interpretation sign manuals to assist sample collection.

[0209] Step 2-2: Based on auxiliary information such as the building sample collection specifications and the sign interpretation manual, a manual method is used to annotate the building outlines in the image in the form of polygons and perform preliminary quality inspection.

[0210] Step 2-3: Slice the annotated images and label information according to the COCO standard data format, set the slice size to 512*512, and retain 50% overlapping areas between slices to initially form a building sample dataset.

[0211] Step 2-4: Clean and quality check the sample data set again, delete invalid polygons, samples with out-of-bounds annotation contours, and other samples that do not meet the requirements, and modify problems such as inaccurate annotation boxes and incorrect annotation attributes.

[0212] Step 2-5: Randomly divide the cleaned sample data set into training set and validation set in a ratio of 8:2.

[0213] Step 403: constructing a building extraction model (ie, the aforementioned first image processing model).

[0214] Specifically, the construction of the building extraction model is to build the basic building extraction model Mask2Former.

[0215] like Figure 2 As shown, firstly, a multi-scale feature map is obtained through the backbone feature extractor ViTAE; then the multi-scale feature map is used as the input of the pixel decoder (Pixel Decoder), and the pixel decoder gradually upsamples the low-resolution deep features from the output of the backbone feature extractor, and gradually restores the resolution of the feature map to the size of the original image through deconvolution, and obtains multi-scale high-resolution pixel embedding (Pixel Embedding); finally, the Transformer decoder is used to operate the multi-scale pixel embedding to process object queries, and finally predict the segmentation map and category of each object based on the object query. Specifically, the details are as follows:

[0216] Step 3-1: Feature extraction.

[0217] The backbone feature extractor ViTAE is used to extract image features of the input data through a rotating variable window attention mechanism.

[0218] Step 3-2: Pixel decoder decoding.

[0219] Here, the feature map processed by the rotated variable window attention mechanism is input to the pixel decoder. The pixel decoder uses a series of upsampling and deconvolution layers to gradually restore the low-resolution feature map to the same resolution as the original input image. In this process, the pixel decoder uses skip connections to introduce high-resolution contextual information from the backbone feature extractor to enhance the semantic information and spatial details of the features.

[0220] Step 3-3: Mask localization and classification.

[0221] Here, based on the output of the pixel decoder, the high-resolution pixel embedding is further processed using a Transformer decoder based on a masked attention mechanism. The decoder is composed of several stacked masked attention layers. The calculation formula of the masked attention layer is as follows:

[0222]

[0223] in: Indicates The query features of the layer, represents the query features of the previous layer, represents the mask matrix of the previous layer, represents the query matrix, represents the transpose of the key matrix, Represents the normalized scaling factor.

[0224] Step 3-4: The Transformer decoder effectively processes object queries through self-attention and cross-attention mechanisms, and combines the local feature information in the pixel embedding to assign corresponding category labels and instance masks to each pixel. Finally, based on the predicted segmentation mask and category label, the Mask2Former model generates an accurate top outline for each building.

[0225] Step 404: Model training and optimization.

[0226] Here, the ViTAE obtained by pre-training in step 1 is used as the initial weight of the backbone feature extractor part, the sample data set in step 402 is used to train the basic model Mask2Former in step 403, and the label-guided strategy is combined to accelerate network optimization to construct a building contour fine extraction network based on object query. Specifically, the details are as follows:

[0227] Step 4-1: Data enhancement.

[0228] Using the label-guided strategy, in addition to the learnable object query, the class embedding of the real category is introduced as the query during training, and the real annotated mask is input into the Transformer decoder layer as the attention mask, and the model is trained to reconstruct the real mask and label.

[0229] Step 4-2: Loss calculation.

[0230] Use bipartite graph matching to match the predicted segmentation mask with the true annotation in order to calculate the loss and update the model parameters. The specific steps are as follows:

[0231] Step 4-2-1: First, calculate the predicted mask based on the intersection over union (IoU) With real annotation The cost matrix between , the calculation formula is as follows:

[0232]

[0233] in represents the computational cost, represents the i-th prediction mask, represents the jth true annotation

[0234] Step 4-2-2: Use the Hungarian algorithm to find the best match and obtain the predicted mask With real annotation The optimal matching result between them, the optimization objective function is as follows:

[0235]

[0236] in, is the matching loss function, is the ith true value, It is arranged The i-th predicted value after is the set of all possible permutations.

[0237] Step 4-2-3: Based on the best matching result, the predicted mask Assigned to the corresponding ground truth , used to calculate the loss function, the loss function uses binary cross entropy loss and The loss combination is calculated as follows:

[0238]

[0239]

[0240]

[0241] in, is the total loss, and is the weight coefficient used to balance the two losses, is the binary cross entropy loss, yes loss, represents the true label of the i-th pixel, represents the probability that the i-th pixel is predicted to be a building.

[0242] Step 405: Secondary optimization of the adversarial neural network profile (ie, the aforementioned second image processing model).

[0243] Here, if Figure 3As shown, the model trained in step 404 is used for reasoning to obtain the mask extraction result of the building outline of the remote sensing image, and the mask outline is optimized twice using an adversarial neural network to obtain an accurate outline that is more consistent with the building shape. The adversarial neural network consists of two networks, a generator and a discriminator, in which the generator attempts to generate a regularized version of the segmentation mask, and the discriminator is used to test the similarity between the generated regularized outline and the actual outline. With the training iteration, the two networks compete with each other in optimization, and finally the generator is able to generate a regular outline that is closer to the actual shape of the building. Specifically, the details are as follows:

[0244] Step 5-1: Calculation of adversarial loss, reconstruction loss and regularization loss.

[0245] The remote sensing image and the corresponding building outline prediction mask obtained based on the model inference in step 404 are used as input, and the real annotation is used as a label to perform adversarial neural network training to obtain a better building prediction mask. The optimization objective function consists of three types of losses: adversarial loss, reconstruction loss and regularization loss, and the calculation formula is as follows:

[0246]

[0247] in For the total loss, , , , , are weight coefficients used to balance various losses. For adversarial loss, and To rebuild the losses, and is the regularization loss, , , They represent the generator, reconstruction path and discriminator respectively.

[0248] Specifically, the adversarial loss is composed of the generator and the discriminator The adversarial loss is composed of:

[0249]

[0250]

[0251]

[0252] in represents adversarial loss, is the loss from the generator, is the loss from the discriminator, Represents all true labels expectations, Represents all input images and noise joint expectations, Representation Discriminator For the generator Generated regularized mask The discrimination result of .

[0253] The reconstruction loss is composed of and generator The cross entropy loss is composed of, and the calculation formula is as follows:

[0254]

[0255]

[0256] in To reconstruct the path of loss, For the generator of loss, represents the pixel value of the true mask, represents the pixel value of the input mask, Represents the reconstruction path In pixels The predicted value at Represents the regularized mask generated by the generator G at pixel The predicted value at .

[0257] The regularized loss consists of Potts loss and normalized clipping loss, and the calculation formula is as follows:

[0258]

[0259]

[0260] in Potts loss, used to encourage the generator Produces building outlines that align with building boundaries observed in the intensity image, is the normalized clipping loss function, used to further refine the generator As a result, Representation Generator Produced kind The vectorized representation of the segmentation mask after represents the affinity matrix, Normalized affinity matrix.

[0261] Step 5-2: Use a convolutional neural network to detect corners on the building segmentation mask that has been regularized by the adversarial neural network. The network model consists of five convolutional layers, which receives the regularized building mask as input and outputs a corner probability map. Pixels with a value higher than a certain threshold in the probability map can be considered as valid corners of the building polygon.

[0262] During the inference process, several corner points of each regularized contour are obtained through the corner detection network, and then the detected points are arranged in a clockwise direction, and the redundant points close to the edge are filtered out through line segment simplification. Finally, an instance representation is generated for each building contour, and each instance consists of the coordinates of several polygons.

[0263] Step 406: Model reasoning and application.

[0264] Here, the above building extraction model and the above secondary optimization model are connected in series to obtain the building extraction results of remote sensing images. Specifically, the details are as follows:

[0265] Step 6-1: The remote sensing image dataset to be processed is used as input and passed into the trained building extraction model in step 404 to extract the building mask, and then the adversarial neural network in step 405 is used to perform secondary optimization on the extracted contour.

[0266] Step 6-2: Use a sliding window approach to perform slice reasoning on large-scale remote sensing images, and merge the results to obtain the vector outline of the building in the entire image.

[0267] Thus, this application example can achieve the following advantages through the above solution:

[0268] (1) An efficient and accurate method for accurately and automatically extracting building contours from remote sensing images was established. Based on a large-scale pre-trained feature extraction model, the idea of ​​object query was used to directly extract single buildings, and a label-guided strategy was added for training to improve the model convergence speed. This method improved the accuracy of building contour extraction and effectively alleviated the problem of building adhesion.

[0269] (2) A building contour optimization method based on adversarial neural network is proposed, which can perform secondary optimization on the building contours extracted from remote sensing images and effectively alleviate the problem of blurred building extraction boundaries caused by the resolution of remote sensing images.

[0270] (3) This application example is applicable to landform scenes in different regions and remote sensing images in different seasons, and has strong generalization and adaptability.

[0271] In order to implement the method of the embodiment of the present application, the embodiment of the present application also provides an image processing device, which corresponds to the above-mentioned image processing method, and each step in the above-mentioned image processing method embodiment is also fully applicable to the embodiment of the present image processing device.

[0272] like Figure 5 As shown, the image processing device 500 includes: an acquisition module 501, a slicing module 502, an object query module 503 and a generation module 504. The acquisition module 501 is used to acquire a remote sensing image for a target area; the slicing module 502 is used to slice the remote sensing image if it is determined that the size data of the remote sensing image is greater than or equal to a set threshold, and generate at least one remote sensing image slice; the object query module 503 is also used to perform an object query on the remote sensing image slice based on a pre-trained first image processing model for each remote sensing image slice in the at least one remote sensing image slice, determine the first building instance in the remote sensing image slice, and generate a first mask image representing the outline of the first building; the generation module 504 is used to generate a target mask image representing the outline of the target building in the target area based on each first mask image.

[0273] In some embodiments, the image processing module also includes a corner point detection module 505 and an arrangement module 506, which are used to perform corner point detection on the first building segmentation mask corresponding to the first mask image based on the second image processing model, and generate multiple corner points corresponding to the first building outline and the position information of each corner point; the arrangement module 506 is used to arrange each corner point based on the position information of each corner point and the set arrangement rules to construct a vector polygon; the image processing module also includes a determination module 507, which is used to determine the first building vector outline data based on the vector data of the vector polygon; the generation module 504 is also used to generate target building vector outline data representing the target building outline based on each first building vector outline data.

[0274] In some embodiments, if it is determined that the size data of the remote sensing image is less than a set threshold, the object query module 503 is also used to perform an object query on the remote sensing image based on the first image processing model, determine the target building instance corresponding to the target area, and generate a target mask image representing the outline of the target building.

[0275] In some embodiments, the generation module 504 is used to perform feature extraction on remote sensing image slices based on the rotating variable window attention layer of the pre-trained feature extractor to generate multi-scale image feature data; upsample the multi-scale image feature data based on the pixel decoder to generate a multi-scale high-resolution pixel embedding vector; the object query module 503 is also used to perform object query on the multi-scale high-resolution pixel embedding vector based on the self-attention model and cross-attention model of the transform decoder to determine the first building instance; and generate the pixel category label of the remote sensing image slice and the first building segmentation mask of the first building instance; the generation module 504 is also used to generate a first mask image based on the transform decoder, the pixel category label and the first building segmentation mask.

[0276] In some embodiments, the image processing module also includes a pre-training module 508, which is used to construct a backbone network of a feature extractor based on multiple attention layers, and the attention layer adopts a rotating variable window attention model; the backbone network is pre-trained based on the preprocessed first remote sensing image sample set until a pre-trained feature extractor is generated.

[0277] In some embodiments, the acquisition module 501 is used to acquire a second remote sensing image sample set after annotation processing, and the second remote sensing image sample set includes a training sample set; the pre-training module 508 is also used to predict the transform decoder based on the training sample set to generate prediction mask data; the transformer decoder is pre-trained based on the prediction mask data, the real annotated data corresponding to the training sample set and the first loss function until a pre-trained third image processing model is generated.

[0278] In some embodiments, the image processing model also includes a correction module 509, which is used to correct the first building segmentation mask based on the generator; the corner point detection module 505 is used to perform corner point detection on the corrected first building segmentation mask based on the corner point detection model to generate multiple corner points and position information corresponding to each corner point.

[0279] In some embodiments, the generation module 504 is also used to correct the mask sample data representing the contours of each building based on the initial generator to generate initial corrected mask data; the determination module 507 is also used to determine the discrimination result between the initial corrected mask data and the real corrected sample data corresponding to the mask sample data based on the initial discriminator; based on the discrimination result, adversarial loss, reconstruction loss and regularization loss, determine the second loss function; the pre-training module 508 is also used to iteratively train the initial discriminator and the initial generator based on the second loss function until a trained generator is generated.

[0280] In actual application, the acquisition module 501, the slicing module 502, the object query module 503, the generation module 504, the corner detection module 505, the arrangement module 506, the determination module 507, the pre-training module 508 and the correction module 509 can be implemented by a processor in the image processing device. Of course, the processor needs to run the computer program in the memory to realize its function.

[0281] It should be noted that: when the image processing device provided in the above embodiment performs image processing, only the division of the above program modules is used as an example. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device is divided into different program modules to complete all or part of the above-described processing. In addition, the image processing device provided in the above embodiment and the image processing method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0282] Based on the hardware implementation of the above program modules and in order to implement the method of the embodiment of the present application, the embodiment of the present application also provides an electronic device. Figure 6 Only an exemplary structure of the electronic device is shown, not the entire structure. Part or all of the structures shown in FIG. 6 may be implemented as needed. As shown in FIG. 6 , the electronic device 600 provided in an embodiment of the present application includes: at least one processor 601, a memory 602, a user interface 603, and at least one network interface 604. The various components in the electronic device 600 are coupled together via a bus system 605. It can be understood that the bus system 605 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 605 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, various buses are labeled as bus systems 605 in FIG. 6 .

[0283] The user interface 603 may include a display, a keyboard, a mouse, a trackball, a click wheel, keys, buttons, a touch pad or a touch screen.

[0284] The memory 602 in the embodiment of the present application is used to store various types of data to support the operation of the electronic device. Examples of such data include: any computer program used to operate on the electronic device.

[0285] The control method of the electronic device disclosed in the embodiment of the present application can be applied to the processor 601, or implemented by the processor 601. The processor 601 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the control method of the electronic device can be completed by the hardware integrated logic circuit or software instructions in the processor 601. The above-mentioned processor 601 may be a general-purpose processor, a digital signal processor (DSP, DigitalSignal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 601 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiment of the present application. The general-purpose processor may be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiment of the present application, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in the memory 602. The processor 601 reads the information in the memory 602 and completes the steps of the control method of the electronic device provided in the embodiment of the present application in combination with its hardware.

[0286] In an exemplary embodiment, the electronic device may be implemented by one or more application specific integrated circuits (ASIC), DSP, programmable logic device (PLD), complex programmable logic device (CPLD), field programmable gate array (FPGA), general processor, controller, microcontroller (MCU), microprocessor, or other electronic components to execute the aforementioned method.

[0287] It can be understood that the memory 602 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), and direct RAM bus random access memory (DRRAM, Direct Rambus Random Access Memory).The memories described in the embodiments of the present application are intended to include, but are not limited to, these and any other suitable types of memories.

[0288] In an exemplary embodiment, the present application also provides a computer storage medium, which can be a computer-readable storage medium, on which a computer program is stored, and the computer program can be executed by a processor to complete the steps of the method of the present application. The computer-readable storage medium can be a memory such as ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface storage, optical disk, or CD-ROM.

[0289] It should be noted that: "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0290] In addition, the technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict.

[0291] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: The method further comprises: Acquire remote sensing images of target areas; If it is determined that the size data of the remote sensing image is greater than or equal to the set threshold, slicing the remote sensing image to generate at least one remote sensing image slice; For each remote sensing image slice of the at least one remote sensing image slice, performing an object query on the remote sensing image slice based on a pre-trained first image processing model, determining a first building instance in the remote sensing image slice, and generating a first mask image representing an outline of the first building; Based on each of the first mask images, generating a target mask image representing an outline of a target building in the target area; The method further comprises: Performing corner point detection on the first building segmentation mask corresponding to the first mask image based on the second image processing model to generate a plurality of corner points corresponding to the first building outline and position information of each corner point; Based on the position information of each corner point and the set arrangement rules, the corner points are arranged to construct a vector polygon; Determine first building vector outline data based on the vector data of the vector polygon; Based on each of the first building vector outline data, generating target building vector outline data representing the outline of the target building; The pre-trained first image processing model includes a pre-trained feature extractor and a pre-trained third image processing model, the pre-trained third image processing model includes a pixel decoder and a transform decoder, and the method further includes: Extracting features from the remote sensing image slices based on the rotation variable window attention layer of the pre-trained feature extractor to generate multi-scale image feature data; Upsampling the multi-scale image feature data based on the pixel decoder to generate a multi-scale high-resolution pixel embedding vector; Performing an object query on the multi-scale high-resolution pixel embedding vector based on the self-attention model and the cross-attention model of the transform decoder to determine the first building instance; and generating a pixel category label of the remote sensing image slice and a first building segmentation mask of the first building instance; generating the first mask image based on the transform decoder, the pixel class label and the first building segmentation mask; The second image processing model includes a pre-trained adversarial neural network generator and a corner point detection model. The corner point detection is performed on the first mask image based on the second image processing model to generate a plurality of corner points corresponding to the first building outline and position information of each corner point, including: modifying the first building segmentation mask based on the generator; Corner point detection is performed on the corrected first building segmentation mask based on the corner point detection model to generate the multiple corner points and position information corresponding to each corner point.

2. The method according to claim 1, characterized in that The method further comprises: If it is determined that the size data of the remote sensing image is smaller than the set threshold, an object query is performed on the remote sensing image based on the first image processing model to determine the target building instance corresponding to the target area, and a target mask image representing the outline of the target building is generated.

3. The method according to claim 2, characterized in that The method further comprises: Based on multiple attention layers, a backbone network of the feature extractor is constructed, wherein the attention layer adopts a rotation variable window attention model; The backbone network is pre-trained based on the pre-processed first remote sensing image sample set until the pre-trained feature extractor is generated.

4. The method according to claim 2, characterized in that: The method further comprises: Acquire a second remote sensing image sample set after labeling, wherein the second remote sensing image sample set includes a training sample set; Predicting the transform decoder based on the training sample set to generate prediction mask data; The transformer decoder is pre-trained based on the predicted mask data, the real labeled data corresponding to the training sample set and the first loss function until a pre-trained third image processing model is generated.

5. The method according to claim 4, characterized in that The adversarial neural network further includes: an initial generator and an initial discriminator, and the method further includes: Based on the initial generator, the mask sample data representing the contours of each building is corrected to generate initial corrected mask data; Based on the initial discriminator, determining a discrimination result between the initial revised mask data and the real revised sample data corresponding to the mask sample data; Determine a second loss function based on the discrimination result, the adversarial loss, the reconstruction loss and the regularization loss; Based on the second loss function, the initial discriminator and the initial generator are trained iteratively until a trained generator is generated.

6. An image processing device, characterized in that: The image processing device comprises: An acquisition module, used for acquiring remote sensing images of a target area; A slicing module, configured to slice the remote sensing image to generate at least one remote sensing image slice if it is determined that the size data of the remote sensing image is greater than or equal to a set threshold; an object query module, configured to perform an object query on each remote sensing image slice of the at least one remote sensing image slice based on a pre-trained first image processing model, determine a first building instance in the remote sensing image slice, and generate a first mask image representing an outline of the first building; The pre-trained first image processing model includes a pre-trained feature extractor and a pre-trained third image processing model, the pre-trained third image processing model includes a pixel decoder and a transform decoder, and further includes: Extracting features from the remote sensing image slices based on the rotation variable window attention layer of the pre-trained feature extractor to generate multi-scale image feature data; Upsampling the multi-scale image feature data based on the pixel decoder to generate a multi-scale high-resolution pixel embedding vector; Performing an object query on the multi-scale high-resolution pixel embedding vector based on the self-attention model and the cross-attention model of the transform decoder to determine the first building instance; and generating a pixel category label of the remote sensing image slice and a first building segmentation mask of the first building instance; generating the first mask image based on the transform decoder, the pixel class label and the first building segmentation mask; Also includes: Performing corner point detection on the first building segmentation mask corresponding to the first mask image based on the second image processing model to generate a plurality of corner points corresponding to the first building outline and position information of each corner point; Based on the position information of each corner point and the set arrangement rules, the corner points are arranged to construct a vector polygon; Determine first building vector outline data based on the vector data of the vector polygon; Based on each of the first building vector outline data, generating target building vector outline data representing the outline of the target building; The second image processing model includes a pre-trained adversarial neural network generator and a corner point detection model. The corner point detection is performed on the first mask image based on the second image processing model to generate a plurality of corner points corresponding to the first building outline and position information of each corner point, including: modifying the first building segmentation mask based on the generator; Performing corner point detection on the modified first building segmentation mask based on the corner point detection model to generate the plurality of corner points and position information corresponding to each corner point; A generating module is used to generate a target mask image representing the contour of the target building in the target area based on each of the first mask images.

7. An electronic device, characterized in that: include: A processor and a memory for storing a computer program that can be run on the processor, wherein the processor, when running the computer program, executes the steps of the method according to any one of claims 1 to 5.

Images