Satellite image processing method, device, electronic device and computer storage medium

Abstract

The present application provides a satellite image processing method, device, electronic device and computer storage medium, the method comprising: acquiring a satellite image; processing the satellite image using a semantic branch, a pixel feature branch, a height branch and an offset branch of a first building detection model to obtain a first image feature indicating a building area in the satellite image, a second image feature composed of pixel features of each pixel of the satellite image, a third image feature including a height feature of each pixel in the satellite image and a fourth image feature including an offset feature of each pixel; dividing the building area into sub-areas corresponding to the building according to the second image feature; determining the height and offset of the building according to the height features and offset features of the pixels in the building sub-areas, thereby determining the bottom area of ​​the building. In this solution, multiple branches can process satellite images in parallel, so this solution has higher processing efficiency than the prior art.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a satellite image processing method, device, electronic equipment and computer storage medium. Background Art

[0002] At present, digital maps have become a common navigation tool in people's daily lives. In order to make it easier for users to identify buildings on the map, existing digital maps often need to show users the corresponding three-dimensional models of buildings. To do this, it is necessary to detect the top and bottom of each building on the corresponding satellite image.

[0003] In the prior art, an instance detection model can be used to detect the top of each building in a satellite image, and then a height prediction model and an offset prediction model can be used to predict the height and offset of the building based on the identified top of each building (the offset refers to the offset of the bottom of the building relative to the top in the satellite image), thereby determining the bottom of the building.

[0004] In the prior art, the height prediction model and the offset prediction model need to rely on the output of the instance detection model, resulting in low overall processing efficiency. Summary of the invention

[0005] Based on the above-mentioned shortcomings of the prior art, the present application provides a satellite image processing method, device, electronic device and computer storage medium to improve the processing efficiency of satellite images.

[0006] The first aspect of the present application provides a satellite image processing method, comprising:

[0007] Access satellite imagery;

[0008] The satellite image is processed using a pre-constructed first building detection model to obtain a first image feature, a second image feature, a third image feature and a fourth image feature of the satellite image; wherein the first building detection model includes a semantic branch, a pixel feature branch, a height branch and an offset branch; the first image feature is output by the semantic branch and is used to indicate a building area in the satellite image; the second image feature is output by the pixel feature branch and includes a pixel feature of each pixel in the satellite image; the third image feature is output by the height branch and includes a height feature of each pixel in the satellite image; the fourth image feature is output by the offset branch and includes an offset feature of each pixel in the satellite image;

[0009] Determine the pixel features of the pixels in the building area indicated by the first image feature in the second image feature, and classify the pixels in the building area based on the determined pixel features of the pixels in the building area indicated by the first image feature to obtain a plurality of sub-areas into which the building area is divided; wherein each of the sub-areas represents the top of a building in the satellite image;

[0010] For each of the sub-regions, determining the height of a building corresponding to the sub-region according to a height feature of pixels located in the sub-region included in the third image feature, and determining the offset of the building corresponding to the sub-region according to an offset feature of pixels located in the sub-region included in the fourth image feature;

[0011] For each of the sub-areas, a bottom area of ​​the building corresponding to the sub-area is determined according to the sub-area, the height of the building corresponding to the sub-area, and the offset.

[0012] Optionally, after determining, for each of the sub-areas, the bottom area of ​​the building corresponding to the sub-area according to the sub-area and the height and offset of the building corresponding to the sub-area, the method further includes:

[0013] For each of the buildings, vectorization processing is performed on the sub-areas and the bottom area of ​​the building to obtain a set of contour vectors representing the contour of the building.

[0014] Optionally, the vectorization processing is performed on the sub-area and the bottom area of ​​the building to obtain a set of outline vectors representing the outline of the building, including:

[0015] Using a polygon fitting algorithm, a plurality of pixels at the edge of the sub-area and the bottom area of ​​the building are determined as key pixels;

[0016] Rectangular fitting is performed on key pixels located in the sub-area of ​​the building and key pixels located in the bottom area of ​​the building, respectively, to obtain a first rectangle representing the outline of the sub-area of ​​the building and a second rectangle representing the outline of the bottom area of ​​the building; wherein the vectors constituting the first rectangle and the vectors constituting the second rectangle are used as the outline vectors of the building.

[0017] Optionally, the vectorization processing is performed on the sub-area and the bottom area of ​​the building to obtain a set of outline vectors representing the outline of the building, including:

[0018] Using a polygon fitting algorithm, a plurality of pixels at the edge of the sub-area and the bottom area of ​​the building are determined as key pixels;

[0019] Connecting each two adjacent key pixels located in the sub-area of ​​the building to obtain a corresponding reference vector, and connecting each two adjacent key pixels located in the bottom area of ​​the building to obtain a corresponding reference vector;

[0020] For each two adjacent reference vectors, if the difference between the angle of the two reference vectors and 90° is less than a preset first threshold, adjust the two reference vectors so that the angle of the two reference vectors is equal to 90°;

[0021] For every two adjacent reference vectors, if the angle between the two reference vectors is less than a preset second threshold, the two reference vectors are merged into one reference vector; wherein all the reference vectors after adjustment and merging constitute the outline vector of the building.

[0022] Optionally, the using a pre-built first building detection model to process the satellite image to obtain a first image feature, a second image feature, a third image feature and a fourth image feature of the satellite image includes:

[0023] Downsampling the satellite image using a downsampling module of the first building detection model to obtain downsampling features of the satellite image;

[0024] The downsampled features of the satellite image are respectively input into the semantic branch, pixel feature branch, height branch and offset branch of the first building detection model to obtain a first image feature output by the semantic branch, a second image feature output by the pixel feature branch, a third image feature output by the height branch and a fourth image feature output by the offset branch.

[0025] Optionally, for each of the sub-regions, determining the height of a building corresponding to the sub-region according to height features of all pixels in the sub-region, and determining the offset of the building corresponding to the sub-region according to offset features of all pixels in the sub-region, includes:

[0026] For each of the sub-regions, the median of the height features of all pixels in the sub-region is determined as the height of the building corresponding to the sub-region, and the median of the offset features of all pixels in the sub-region is determined as the offset of the building corresponding to the sub-region.

[0027] Optionally, before determining, for each of the sub-regions, the height of the building corresponding to the sub-region according to the height features of all pixels in the sub-region, and determining the offset of the building corresponding to the sub-region according to the offset features of all pixels in the sub-region, the method further includes:

[0028] According to the scenes contained in the satellite image, the satellite image is divided into a plurality of blocks; wherein each of the blocks corresponds to one of the scenes;

[0029] For each pre-constructed second building detection model, the second building detection model is used to detect and obtain a sub-region of each building in the satellite image; wherein the number of the second building detection models is at least one; the first building detection model and each of the second building detection models correspond to a scene in the satellite image;

[0030] For each of the blocks, a sub-region of a building detected by the building detection model that does not correspond to the scene of the block is deleted from the block.

[0031] A second aspect of the present application provides a satellite image processing device, comprising:

[0032] An acquisition unit, used for acquiring satellite images;

[0033] A processing unit is used to process the satellite image using a pre-constructed first building detection model to obtain a first image feature, a second image feature, a third image feature and a fourth image feature of the satellite image; wherein the first building detection model includes a semantic branch, a pixel feature branch, a height branch and an offset branch; the first image feature is output by the semantic branch and is used to indicate a building area in the satellite image; the second image feature is output by the pixel feature branch and includes a pixel feature of each pixel in the satellite image; the third image feature is output by the height branch and includes a height feature of each pixel in the satellite image; the fourth image feature is output by the offset branch and includes an offset feature of each pixel in the satellite image;

[0034] a classification unit, configured to determine pixel features of pixels in the building area indicated by the first image feature in the second image feature, and classify the pixels in the building area based on the determined pixel features of pixels in the building area indicated by the first image feature, so as to obtain a plurality of sub-areas into which the building area is divided; wherein each of the sub-areas represents a top of a building in the satellite image;

[0035] a first determining unit, configured to determine, for each of the sub-regions, a height of a building corresponding to the sub-region according to a height feature of pixels located in the sub-region included in the third image feature, and to determine an offset of the building corresponding to the sub-region according to an offset feature of pixels located in the sub-region included in the fourth image feature;

[0036] The second determining unit is used to determine, for each of the sub-areas, a bottom area of ​​the building corresponding to the sub-area according to the sub-area, the height of the building corresponding to the sub-area, and the offset.

[0037] Optionally, the processing device further includes:

[0038] The vectorization unit is used for performing vectorization processing on the sub-area and the bottom area of ​​each building to obtain a set of outline vectors representing the outline of the building.

[0039] Optionally, when the vectorization unit performs vectorization processing on the sub-area and the bottom area of ​​the building to obtain a set of outline vectors representing the outline of the building, it is specifically used to:

[0040] Using a polygon fitting algorithm, a plurality of pixels at the edge of the sub-area and the bottom area of ​​the building are determined as key pixels;

[0041] Rectangular fitting is performed on key pixels located in the sub-area of ​​the building and key pixels located in the bottom area of ​​the building, respectively, to obtain a first rectangle representing the outline of the sub-area of ​​the building and a second rectangle representing the outline of the bottom area of ​​the building; wherein the vectors constituting the first rectangle and the vectors constituting the second rectangle are used as the outline vectors of the building.

[0042] A third aspect of the present application provides an electronic device, including a memory and a processor;

[0043] Wherein, the memory is used to store computer programs;

[0044] The processor is used to execute the computer program, specifically to implement the satellite image processing method provided in any one of the first aspects of the present application.

[0045] The fourth aspect of the present application provides a computer storage medium for storing a computer program. When the computer program is executed, it is specifically used to implement the satellite image processing method provided in any one of the first aspects of the present application.

[0046] The present application provides a satellite image processing method, device, electronic device and computer storage medium, the method comprising: acquiring a satellite image; processing the satellite image using a semantic branch, a pixel feature branch, a height branch and an offset branch of a first building detection model to obtain a first image feature indicating a building area in the satellite image, a second image feature composed of pixel features of each pixel of the satellite image, a third image feature including a height feature of each pixel in the satellite image and a fourth image feature including an offset feature of each pixel; dividing the building area into sub-areas corresponding to each building according to the second image feature; determining the height and offset of the building according to the height features and offset features of all pixels in the building sub-area, and then determining the bottom area of ​​the building. The multiple branches in the scheme can process satellite images in parallel, so the scheme has higher processing efficiency than the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0048] Figure 1 A flowchart of a satellite image processing method provided in an embodiment of the present application;

[0049] Figure 2 A schematic diagram of the structure and post-processing flow of a building detection model provided in an embodiment of the present application;

[0050] Figure 3 A schematic diagram of a top area and a bottom area of ​​a building provided in an embodiment of the present application;

[0051] Figure 4 A flowchart of another satellite image processing method provided in an embodiment of the present application;

[0052] Figure 5 A flowchart of another satellite image processing method provided in an embodiment of the present application;

[0053] Figure 6 A schematic diagram of the architecture of a multi-model fusion algorithm provided in an embodiment of the present application;

[0054] Figure 7 A schematic diagram of the detection effect of the satellite image processing method provided in an embodiment of the present application on buildings;

[0055] Figure 8A schematic diagram of the structure of a satellite image processing device provided in an embodiment of the present application;

[0056] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] The present application provides a method for processing satellite images, which is mainly used to detect the top and bottom of buildings displayed on satellite images, so as to subsequently display the corresponding three-dimensional model of the building to the user on a digital map generated based on the satellite image. In addition, in other embodiments of the present application, in order to reduce the amount of map data sent to the client, the detected building can also be vectorized to obtain an outline vector representing the outline of the building. When the client needs to display the three-dimensional model of the building, it only needs to send the outline vector to the client, and then the client draws the corresponding three-dimensional model on the display screen based on the outline vector.

[0059] The satellite image processing method provided in this application can be understood as the application of computer vision technology (CV) in the field of artificial intelligence to satellite images. Computer vision is a science that studies how to make machines "see", that is, using cameras and computers to replace human eyes to identify targets, trace traces, measure machine vision, and further perform graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instrument detection.

[0060] Broadly speaking, computer vision technology generally includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, etc.

[0061] First, please refer to Figure 1 The satellite image processing method provided in the embodiment of the present application may specifically include the following steps:

[0062] S101. Acquire satellite images.

[0063] Satellite images are used in the production and updating of digital maps. Satellite images can be taken by satellites over a certain area on the ground and then sent to server equipment on the ground through antennas.

[0064] S102: Process the satellite image using the pre-constructed first building detection model to obtain a first image feature, a second image feature, a third image feature, and a fourth image feature of the satellite image.

[0065] Please refer to Figure 2 The first building detection model used in the present application may include a downsampling module, and four branches located after the downsampling module, namely a semantic branch, a pixel feature branch, a height branch and an offset branch. The above four branches respectively output corresponding image features. After the output image features are processed by the clustering module, the detection results of the first building detection model are obtained.

[0066] Step S102 is equivalent to inputting the satellite image into the first building detection model of the present application, thereby obtaining the corresponding image features output by the four branches of the first building detection model. The specific processing flow is as follows:

[0067] The satellite image is first input into the downsampling module of the first building detection model, and the downsampling module of the first building detection model downsamples the satellite image to obtain the downsampling features of the satellite image.

[0068] The downsampled features of the satellite image are then input into the semantic branch, pixel feature branch, height branch and offset branch of the first building detection model respectively. After the downsampled features are processed by the above branches, the corresponding first image features, second image features, third image features and fourth image features are obtained.

[0069] Among them, the first image feature is output by the semantic branch, the second image feature is output by the pixel feature branch, the third image feature is output by the height branch, and the fourth image feature is output by the offset branch.

[0070] The semantic branch is used to distinguish between building areas (foreground areas) and non-building areas (background areas) in satellite images. Correspondingly, the first image feature is an image feature used to indicate building areas in satellite images. Specifically, the first image feature can be a matrix with the same size as the satellite image, each element in the matrix corresponds to a pixel of the satellite image, and the value of each element can be 1 or 0. If a pixel is determined to be located in a building area after the semantic branch is processed, the value of the element corresponding to the pixel in the first image feature is set to 1. Conversely, if the pixel is determined to be located in a non-building area, the value of the corresponding element is set to 0.

[0071] The pixel feature branch is used to extract the pixel features of each pixel in the satellite image, and the corresponding second image features include the pixel features of each pixel in the satellite image. The pixel feature of a pixel is a multidimensional vector that can reflect the pixel value of the pixel and the pixel values ​​of other pixels adjacent to the pixel.

[0072] The height branch is used to predict the relative height of each pixel in the satellite image (referring to the distance of the pixel in the satellite image as a position relative to the ground in the satellite image). The corresponding third image feature contains the height feature of each pixel in the satellite image. The height feature here can be directly equal to the relative height of the corresponding pixel, or it can be other vectors that can obtain the relative height of the corresponding pixel after simple calculation.

[0073] The offset branch is used to predict the offset of each pixel in the satellite image relative to the ground. The corresponding fourth image feature contains the offset feature of each pixel in the satellite image. When a satellite takes a satellite image, its viewing angle is often not perpendicular to the ground. Therefore, objects that are actually perpendicular to the ground will be tilted in the satellite image. Correspondingly, the pixels at the top of these objects will have a certain offset relative to the bottom of the object. The pixel offset is used to characterize the size of this offset. Similar to the height feature, the offset feature here can be directly equal to the offset of the corresponding pixel, or it can be other vectors that can obtain the offset of the corresponding pixel after simple calculation.

[0074] The above four branches and down-sampling module can all be convolutional network structures composed of several convolutional layers and other corresponding image processing layers.

[0075] S103: using the clustering module of the first building detection model to classify pixels located in the building area, to obtain a plurality of sub-areas into which the building area is divided.

[0076] Each sub-region represents the top of a building in the satellite image.

[0077] In step S103, the clustering module may perform the above classification in combination with the first image feature output by the semantic branch and the second image feature output by the pixel feature branch.

[0078] First, the clustering module can determine the building area and non-building area in the satellite image according to the indication of the first image feature, and then determine the pixels located in the building area that need to be classified in the satellite image, and then determine the pixel features of the pixels located in the building area indicated by the first image feature in the second image feature.

[0079] On this basis, the clustering module can classify the pixels in the building area based on the pixel features of the pixels in the building area indicated by the first image feature, and obtain multiple sub-areas into which the building area is divided by using a density clustering method.

[0080] Density clustering is an existing clustering algorithm. Its basic principle is to divide a plurality of samples to be classified (in this application, a large number of pixels located in the building area are the samples to be classified) into multiple categories according to the distance between the features of each two samples. Generally, a group of samples with closer feature distances will be classified into the same category, while samples with farther feature distances will be classified into different categories. The feature distance here refers to the distance between the feature vectors corresponding to the two samples (it can also be understood as the similarity of the two corresponding feature vectors). In this application, the feature distance between each two pixels located in the building area is the distance (or similarity) between the pixel features of the two pixels contained in the second image feature.

[0081] Through density clustering, the clustering module can divide the pixels in the building area into multiple categories. Finally, the area where a group of pixels in each category are located can be determined as a sub-area in the building area, and this sub-area is equivalent to the top area of ​​a building in the satellite image. In other words, after step S103, the clustering module can identify which pixels in the building area belong to one building and which pixels belong to another building.

[0082] S104 . For each sub-region, determine the height and offset of the building corresponding to the sub-region according to the third image feature and the fourth image feature.

[0083] Specifically, in step S104, the height features of the pixels located in a sub-region can be obtained from the third image features, and the height of the building corresponding to the sub-region can be determined based on these height features. At the same time, the offset features of the pixels located in the sub-region can be obtained from the fourth image features, and the offset of the building corresponding to the sub-region can be determined based on these offset features.

[0084] The height of a building refers to the relative distance between the top of the building and the bottom of the building in the satellite image, and the offset of the building refers to the offset of the top of the building relative to the bottom of the building in the satellite image.

[0085] Step S104 can also be implemented by the clustering module of the first building detection model of the present application. After dividing each sub-region, the clustering module can extract the height feature of all pixels in the sub-region from the third image feature for each sub-region, based on which the height of the building corresponding to the sub-region is determined, and extract the offset feature of all pixels in the sub-region from the fourth image feature, based on which the offset of the building corresponding to the sub-region is determined.

[0086] Optionally, when the height feature of each pixel in the third feature is equal to the relative height of the pixel in the satellite image, the specific process of determining the height of the building in step S104 may be:

[0087] Obtain the height features of all pixels in the top area of ​​the building, and determine the median of these height features as the height of the building;

[0088] Alternatively, the arithmetic mean of these height characteristics is determined as the height of the building.

[0089] Optionally, when the offset feature of each pixel in the fourth feature is equal to the offset of the pixel in the satellite image, the method for determining the offset of the building in step S104 may be:

[0090] Obtain the offset features of all pixels in the top area of ​​the building, and determine the median of the offset features as the offset of the building;

[0091] Alternatively, the arithmetic mean of the offsets is determined as the offset of the building.

[0092] S105 . For each sub-region, determine a bottom region of the building corresponding to the sub-region according to the sub-region and the height and offset of the building corresponding to the sub-region.

[0093] like Figure 3 As shown in the figure, when the satellite's viewing angle is completely perpendicular to the ground, the top area of ​​the building displayed on the satellite image will overlap with the bottom area. At this time, the offset of the building is 0. However, due to the offset of the satellite's viewing angle during shooting, the sub-area displayed on the satellite image (i.e., the top area of ​​the building) will be offset to a certain extent relative to the bottom area of ​​the building. In this case, you can refer to Figure 3 , the top area of ​​the building determined in step S103, the height and offset of the building, and the bottom area of ​​the building are determined.

[0094] After the top area and bottom area of ​​a building are determined on the satellite image, when the digital map of the corresponding area is subsequently displayed to the user, a three-dimensional model of the corresponding building can be generated and displayed on the digital map according to the top area and the bottom area, as well as the relative position of the bottom area on the map. This technology is an existing technical means in the field of digital maps and will not be described in detail here.

[0095] The following describes in detail the structures and principles of the downsampling module and each branch of the first building detection model of the present application.

[0096] The downsampling module may specifically include a plurality of sequentially arranged convolution layers for downsampling satellite images, a normalization layer Batch Normalization (BN) and an activation layer Relu located after the plurality of convolution layers, wherein the convolution layer abstracts and combines the basic texture features of the satellite image. The normalization layer performs normal distribution processing on the image features output by the convolution layer. The activation layer can perform nonlinear mapping on the output image features to enhance the generalization ability of the model. In the downsampling module, the input of the first convolution layer is the satellite image, the output of the last convolution layer is used as the input of the normalization layer, the input of the normalization layer is used as the output of the activation layer, and the output of the activation layer is used as the downsampling feature of the satellite image and is respectively input into the four branches of the first building detection model, and the input of each convolution layer except the first convolution layer is the output of the previous convolution layer. In addition, the size of the previous convolution layer in the downsampling module is larger than the size of the next convolution layer, and the satellite image passes through the convolution layer one by one, which is equivalent to being reduced step by step.

[0097] The downsampling module downsamples the satellite image, which is equivalent to reducing the size of the satellite image by several times. For example, assuming that the original size of the satellite image is M*N, the downsampling module downsamples the satellite image by S times, and the obtained downsampled features are equivalent to a satellite image of size (M / s)*(N / s). The downsampling multiple S is determined by the size of each convolutional layer in the above downsampling module.

[0098] The function of the downsampling module is to reduce the original large-sized satellite image to a smaller size, appropriately reduce the amount of data that needs to be processed by subsequent branches, so as to improve processing efficiency and reduce the computing resources consumed during processing.

[0099] Furthermore, in this solution, since the four branches of the first building detection model use the down-sampled features output by the down-sampling module together, only one down-sampling process is required to meet the demand for detecting buildings in satellite images. In the prior art, when multiple models are used to process satellite images, each model often needs to perform down-sampling processing on the satellite images separately. Therefore, this solution uses the first building detection model composed of the above four branches to process satellite images, which can consume less computing resources compared to the prior art.

[0100] As mentioned above, the semantic branch is mainly used to segment the foreground (building area) and background (non-building area) of the satellite image. The input of the semantic branch is the downsampled features of the satellite image output by the downsampling module. The semantic branch can adopt the feature pyramid structure (Feature Pyramid Networks, FPN). Specifically, the semantic branch includes a plurality of deconvolution layers arranged in sequence, each deconvolution layer is used to perform a deconvolution operation on the input image features, the size of the previous deconvolution layer is smaller than the size of the next deconvolution layer, and except for the first deconvolution layer, each deconvolution layer uses the output of the previous deconvolution layer as its own input. The input of the first deconvolution layer is the downsampled feature, and the output of the last deconvolution layer is the first image feature. When the downsampled feature passes through each deconvolution layer of the semantic branch, its size increases layer by layer, and the size of the first image feature output at the end is consistent with the original size of the satellite image. The above-mentioned semantic branch that is enlarged layer by layer and the previous downsampling module that is reduced layer by layer form a U-net network structure.

[0101] Optionally, in order to better integrate various image features, the size of each deconvolution layer of the semantic branch can be designed to be consistent with the size of a convolution layer in the downsampling module. For example, the size of the first deconvolution layer of the semantic branch and the last convolution layer of the downsampling module can be designed to be consistent, the size of the last deconvolution layer of the semantic branch and the size of the first convolution layer of the downsampling module can be consistent, and the same applies to other deconvolution layers and convolution layers.

[0102] In this way, the input of each deconvolution layer can include not only the output of the previous deconvolution layer, but also the output of the convolution layer with the same size as the previous deconvolution layer in the downsampling module. In other words, the deconvolution layer can add the above two types of inputs and realize the fusion of the two types of features through the deconvolution operation.

[0103] In this solution, a standard cross entropy loss function can be used when constructing a semantic branch. The construction process of the semantic branch is to use multiple sample images with pre-labeled foreground areas and background areas to input into the first building detection model, and then compare the foreground area and background area indicated by the first image feature input to the semantic branch with the annotations in the sample image, determine the function value of the cross entropy loss function based on the difference between the two, update the parameters of the semantic branch based on the function value, and repeat the above process until the difference between the two finally meets the convergence condition, at which time the construction of the semantic branch is completed.

[0104] The pixel feature branch is used to extract the pixel features of each pixel of the satellite image. As mentioned above, the pixel feature can be an N-dimensional vector. The value of the vector is determined by the pixel values ​​of the corresponding pixel and the adjacent pixels. The dimension N of the vector is determined by the structure of the pixel feature branch. Generally, N can be set to 8. The structure of the pixel feature branch can be consistent with the structure of the aforementioned semantic branch, that is, a feature pyramid structure is adopted. And, similarly, the input of each deconvolution layer in the pixel feature branch can also include the output of the previous deconvolution layer and the output of the convolution layer with the same size as the previous deconvolution layer in the downsampling module.

[0105] In this scheme, the pixel feature branch extracts the pixel features of each pixel in order to support the subsequent classification by the clustering module. Therefore, when constructing the pixel feature branch, the clustering loss function can be used as the loss function of the pixel feature branch.

[0106] The construction process of the pixel feature branch is similar to that of the semantic branch. First, multiple sample images with labeled top areas of buildings are generated. Then, these sample images are input into the first building detection model to obtain the second image features corresponding to each sample image output by the pixel feature branch. Finally, the intra-class aggregation loss and inter-class discrimination loss of the second image features corresponding to each sample image are evaluated according to the labeled top areas in the sample images. The value of the loss function of the pixel feature branch is obtained by combining the two losses. The pixel feature branch is updated based on this. The construction of the pixel feature branch is completed by repeating the above process.

[0107] The offset branch is used to extract the offset feature of each pixel. In this embodiment, the offset feature of a pixel is the offset of the pixel in the X direction (horizontal direction) and the Y direction (vertical direction). The structure of the offset branch and the structure type of the semantic branch both adopt a feature pyramid structure, and the deconvolution layer of the offset branch can also fuse the output of the previous deconvolution layer and the output of the corresponding convolution layer in the downsampling module for processing.

[0108] When building the offset branch, the sample image can be annotated as follows:

[0109] For pixels belonging to buildings in the sample image, the offset of the top of the corresponding building relative to the bottom is measured in the sample image, and the offset of the building is used as the offset of the corresponding pixel. For pixels that do not belong to buildings in the sample image, the offset is directly marked as 0.

[0110] After completing the above-mentioned annotation, the sample image can be input into the first building detection model to obtain the fourth image feature output by the offset branch. The function value of the loss function of the offset branch is determined by comparing the fourth image feature with the aforementioned annotation result of the sample image. Then, the parameters of the offset branch are updated according to the function value. The above process is repeated to complete the construction of the offset branch.

[0111] Among them, the image processing task implemented by the offset branch is equivalent to a regression task, so the loss function of the offset branch can be a two-norm square loss function (also called minimum square error).

[0112] The height branch is used to extract the height feature of each pixel. In this embodiment, the relative height of the pixel location in the satellite image can be directly used as the height feature. The structure of the height branch is consistent with the previous branches, and both adopt a feature pyramid structure. At the same time, the input of each deconvolution layer can include the output of the previous deconvolution layer and the output of the convolution layer of the corresponding size in the downsampling module.

[0113] The construction process of the height branch is similar to that of the offset branch. First, the sample image is highly labeled. Pixels that do not belong to buildings are labeled with a height of 0, and pixels that belong to buildings are labeled with the relative height of the building on the satellite image as the height of the pixel. After the labeling is completed, the value of the loss function of the height branch is determined based on the difference between the third image feature of the sample image output by the height branch and the labeling result, and the parameters of the height branch are updated accordingly. The construction of the height branch is completed by repeating this process.

[0114] The image processing task implemented by the height branch is also a regression task, and the loss function of the height branch can use a one-norm loss function (also called the minimum absolute deviation, or minimum absolute error).

[0115] The satellite image processing method provided in the embodiment of the present application has the following beneficial effects:

[0116] On the one hand, it can be seen from step S102 that in the first building detection model of the present scheme, the height branch and the offset branch for extracting the height features and the offset features can be executed simultaneously with the semantic branch and the pixel feature branch for detecting the top area of ​​the building, without relying on the image features output by the semantic branch and the pixel feature branch, nor on the top area divided in the subsequent step S103. In this way, if computing resources permit, the four branches in the first building detection model can be executed in parallel, thereby completing the main computing tasks in the detection process of buildings in satellite images at the same time, greatly shortening the time consumption of the entire detection process.

[0117] Secondly, when extracting the height features and offset features of each pixel, the height branch and the offset branch in this solution process the complete down-sampled features after the satellite image is down-sampled, that is, the processing process of the height branch and the offset branch combines the global features of the satellite image, especially considering the texture features of the side facades of each building in the satellite image (that is, the wall surface perpendicular to the ground except the bottom and top of the building), while the height model and the offset model in the prior art only analyze the detected top area of ​​the building, and do not consider the texture features of the side facades of the building. Therefore, the height features and offset features of each pixel output by the height branch and the offset branch in this solution can more accurately reflect the height and offset of the corresponding building than the prior art.

[0118] Optionally, after the server determines the top area and bottom area of ​​the building on the satellite image, in order for the client to display the three-dimensional model of the building, these areas need to be sent to the client through the network. If the top and bottom areas of the building composed of pixels are directly sent to the client, more network resources will be occupied, the transmission speed will be slow, and the user experience will be poor. Therefore, it is necessary to vectorize the detected top area and bottom area, and convert the area composed of pixels of the array into a set of contour vectors representing the contour of the area. When the client needs it, these contour vectors can be directly sent to the client. The client reconstructs the top area and bottom area of ​​the building on the display screen according to the contour vectors, and further constructs a three-dimensional model of the building.

[0119] In response to the above requirements, the present application embodiment provides a satellite image processing method, please refer to Figure 4 , the method may include the following steps:

[0120] S401. Acquire satellite images.

[0121] S402: Process the satellite image using the pre-constructed first building detection model to obtain a first image feature, a second image feature, a third image feature, and a fourth image feature of the satellite image.

[0122] S403: Classify pixels in the building area using the clustering module of the first building detection model to obtain a plurality of sub-areas into which the building area is divided.

[0123] S404: For each sub-region, determine the height and offset of the building corresponding to the sub-region according to the third image feature and the fourth image feature.

[0124] S405 . For each sub-region, determine the bottom region of the building corresponding to the sub-region according to the sub-region and the height and offset of the building corresponding to the sub-region.

[0125] The specific execution process of the above steps S401 to S405 is consistent with that of steps S101 to S105 in the aforementioned embodiment, and will not be described in detail here.

[0126] S406: Perform vectorization processing on the sub-areas and the bottom area of ​​the building to obtain a set of outline vectors representing the outline of the building.

[0127] It can be understood that step S406 can be performed for each building detected in the satellite image, so as to obtain the contour vector of each building in the satellite image.

[0128] The specific execution process of step S406 may include:

[0129] First, a polygon fitting algorithm is used to determine multiple pixels at the edge of the sub-area and the bottom area of ​​the building as key pixels. The polygon fitting algorithm here can be a mature algorithm in the relevant prior art. For example, the existing polygon fitting algorithm approxPolyDP() (an existing function in Opencv) in Opencv (a publicly released, cross-platform computer vision and machine learning software library) can be used to implement the above step of determining the key pixels;

[0130] Compared with writing and implementing polygon fitting algorithms by yourself, the existing polygon fitting algorithms have been used and tested by a large number of users, and their processing efficiency and accuracy are better than the self-developed polygon fitting algorithms. Therefore, in step S406, the above approxPolyDP() function can generally be used to determine the key pixels faster and better;

[0131] After determining the key pixels, every two adjacent key pixels can be directly connected into an edge (equivalent to a vector). After all the connections are completed, the edges located at the edges of the top and bottom areas of the building are equivalent to the contour vector of the building.

[0132] However, the vectors directly connected by the key pixels determined by the above polygon fitting algorithm often have certain deviations from the real building outline. For example, the real building outline is generally a rectangular or right-angled polygon outline, while the outline directly combined by the vectors connected by the key pixels may appear in other shapes. Therefore, after determining the key pixels, the key pixels can be optimized by the following method:

[0133] First, considering that the outlines of most buildings are generally standard rectangles (such as residential buildings in residential areas, large warehouses, etc.), rectangular fitting can be performed on the key pixels located in the top area of ​​the building and the key pixels located in the bottom area of ​​the building respectively to obtain a first rectangle representing the outline of the top area of ​​the building, and a second rectangle representing the outline of the bottom area of ​​the building. After the fitting is completed, the four sides of the first rectangle and the four sides of the second rectangle are equivalent to a set of outline vectors representing the outline of the building.

[0134] Secondly, in addition to the common simple buildings, there are some buildings whose outlines are not simple rectangular outlines, but more complex polygonal outlines, such as L-shaped, concave, and convex. However, the angles between adjacent sides of these polygonal outlines are generally right angles (that is, 90° angles), and acute angles with too small angles rarely appear.

[0135] For such buildings, each two adjacent key pixels located in the sub-area of ​​the building may be connected to obtain a corresponding reference vector, and each two adjacent key pixels located in the bottom area of ​​the building may be connected to obtain a corresponding reference vector;

[0136] After obtaining the reference vectors, considering that most angles are right angles, the angle between each two adjacent reference vectors can be identified one by one. If the angle between the two reference vectors is close to 90° (that is, the difference between the angle between the two reference vectors and 90° is less than the preset first threshold), for example, the angle between the two reference vectors is in the range of 80° to 100°, it can be considered that the real angle here is 90°, and the current angle is caused by the error of the polygon fitting algorithm. Therefore, the two reference vectors can be adjusted. Specifically, the pixels at the start and end points of the two reference vectors can be adjusted so that the angle of the two reference vectors after adjustment is equal to 90°. The above processing is called the square-angled orientation of the reference vectors.

[0137] After the square-angle conversion is completed, it can be further identified whether the angle between each two adjacent reference vectors is less than a preset second threshold. If the angle between two adjacent reference vectors is less than the preset second threshold (for example, less than 30°), it can be considered that the error is caused by the polygon fitting algorithm, and the two reference vectors can be directly merged into one reference vector. This process is called angle optimization.

[0138] An optional merging method is, for three sequentially connected reference vectors (denoted as vectors A, B, and C), after identifying that the angle between vector A and vector B is less than a second threshold, vector B can be deleted, and then the lengths and directions of vector A and vector C can be adjusted so that vector A and vector C, which were originally connected to vector B, are directly connected together.

[0139] After the above adjustments, all reference vectors that have been adjusted and merged (ie, after right-angle optimization and angle optimization) constitute the outline vector of the building.

[0140] A method for identifying buildings with complex polygonal outlines may be that, when performing rectangle fitting using the optimization method of the first aspect, if it is found that there is a large deviation between the rectangular outline fitted to a certain building and the positions of most key pixels, that is, the rectangular outline cannot be close to the distribution of key pixels within an acceptable error range, then it can be considered that the building has a complex polygonal outline.

[0141] In a third aspect, the optimization process of the third aspect is called a capping detection process, which can further optimize the contour vector after the optimization process of the first aspect or the second aspect is completed.

[0142] Specifically, after the key pixels are determined and optimized by polygon fitting, the building outlines depicted by the outline vectors may overlap. The overlapping problem refers to the intersection of the outline of the bottom area of ​​the building and the outline of the water area (including but not limited to rivers, lakes, etc.), and the intersection of the outline of the bottom area of ​​the building and the outline of the road.

[0143] When a map with overlapping problems is shown to users, it looks like part of a building is built on water or a road. Obviously, this phenomenon is wrong. Therefore, it is necessary to detect the contour vectors generated previously to determine whether each group of contour vectors has an overlapping problem. If a problem is detected in any group of contour vectors, the positions of the start and end points of each vector in this group of contour vectors need to be adjusted to eliminate the overlapping problem.

[0144] After the above polygon fitting and various aspects of optimization, the obtained contour vectors can be standardized according to the map data specifications of the corresponding enterprise, so that each set of contour vectors can be converted into vector data in a standard format. When the client needs to display the three-dimensional model of the building, it can obtain these vector data in a standard format from the server, and then draw the corresponding contour vectors on the display screen and fill them appropriately, so as to display the three-dimensional model of the building on the display screen.

[0145] On the one hand, this embodiment provides a complete satellite image processing flow from building detection to building vectorization, which can detect and extract the building's outline vector from any satellite image, and provide support for the subsequent sending to the client for display of the building's three-dimensional model. The entire process is fully automated, without the need for manual processing, and has high processing efficiency.

[0146] On the other hand, since the top and bottom areas representing the building are converted into corresponding contour vectors, when sending data to the client, only the coordinates of the starting point pixels and the end point pixels of the corresponding contour vector need to be transmitted, which greatly reduces the amount of map data sent by the server to the client and saves network resources between the server and the client.

[0147] Optionally, considering that a satellite image may contain a variety of different scenes (the scenes here include but are not limited to urban commercial centers with a large number of high-rise buildings, industrial parks consisting of a large number of warehouses and factories, villa areas with low buildings and regular distribution, villages with low buildings and irregular distribution, farmlands with almost no buildings, etc.), using only one building detection model to detect buildings in all scenes may result in a detection result with low accuracy. Therefore, the embodiment of the present application also provides a method for processing satellite images, please refer to Figure 5 , the method may include the following steps:

[0148] S501: Acquire satellite images.

[0149] S502: Process the satellite image using the pre-constructed first building detection model to obtain a first image feature, a second image feature, a third image feature, and a fourth image feature of the satellite image.

[0150] S503: Classify pixels located in the building area by using the clustering module of the first building detection model to obtain a plurality of sub-areas into which the building area is divided.

[0151] The specific execution process of step S501 to step S503 is consistent with the aforementioned step S101 to step S103 and will not be described in detail.

[0152] S504: Divide the satellite image into a plurality of blocks according to the scenes included in the satellite image.

[0153] Each block corresponds to a scene. For example, the city commercial center in the satellite image is divided into a block, the village outside the city is divided into a block, all the residential areas in the city are divided into a block, and so on.

[0154] Step S504 can be implemented using a pre-built block segmentation model. After the satellite image is input into the block segmentation model, the block segmentation model can draw the boundary of each block and configure corresponding labels, thereby implementing the division of blocks.

[0155] S505: For each pre-constructed second building detection model, use the second building detection model to detect and obtain a sub-region of each building in the satellite image.

[0156] There is at least one second building detection model, and generally there can be two or more. The first building detection model and each second building detection model correspond to a scene in the satellite image.

[0157] As in the previous text, the sub-area of ​​each building is the top area of ​​each building. The second building area detection model in step S505 refers to the sub-area of ​​each building. Figure 1 and Figure 4 The first building detection model in the corresponding embodiment is different from other building detection models.

[0158] The difference here refers to the different objects used for detection. For example, the aforementioned first building detection model can be a building detection model specially trained using sample images of the commercial center area, and is used to identify the top area of ​​the buildings in the commercial center. The second building detection model in step S505 can include building detection model 1 trained using sample images of the village area, building detection model 2 trained using sample images of the industrial park, and so on.

[0159] In addition to the different objects to be detected, the architecture of the second building detection model and the aforementioned first building detection model may be the same or different. For example, the second building detection model may also adopt the multi-branch and U-shaped network structures of the aforementioned first building detection model. In addition, the second building detection model may also adopt the existing mask-rcnn, centermask (both of which are existing neural network architectures) models, or may adopt traditional machine learning algorithms to build corresponding models, which are not limited here.

[0160] It can be seen that the first building detection model and each of the second building detection models are trained using sample images of specific scenes, so the recognition results of these building detection models will be more accurate when identifying buildings in blocks of corresponding scenes in satellite images.

[0161] S506 . For each block, delete from the block a sub-region of a building detected by the building detection model that does not correspond to the scene of the block.

[0162] Step S506 may be understood as a method of fusing sub-regions of buildings detected by a plurality of building detection models including the first building detection model and the second building detection model.

[0163] As mentioned above, each building detection model involved in this embodiment is a building detection model obtained by training with sample images of a specific scene. In other words, each building detection model corresponds to the scene to which the sample images used in the training phase belong. For example, the first building detection model trained with sample images of a commercial center corresponds to a commercial center, and the second building detection model trained with sample images of an industrial park corresponds to an industrial park. The detection results of any building detection model in the block corresponding to its scene are more accurate than the detection results of other building detection models.

[0164] For example, in the block of the industrial park scene, the top area of ​​the building detected by the detection model corresponding to the industrial park is obviously more accurate than the top area of ​​the building detected by the detection model corresponding to the village.

[0165] Therefore, in step S506, for each block, the sub-region detected by the detection model corresponding to the scene to which the block belongs can be retained, and the sub-regions detected by the detection models other than the detection model corresponding to the scene to which the block belongs can be deleted. In combination with the above example, for example, in a block belonging to an industrial park, the sub-region detected by the detection model corresponding to the industrial park is retained, while the sub-regions detected by the detection model corresponding to the village, the detection model corresponding to the commercial center, and the detection model not corresponding to the industrial park are deleted.

[0166] In this way, this solution can use specific detection models trained with different samples for targeted detection of blocks in different scenes, and finally integrate the detection results of multiple detection models through a preferential strategy.

[0167] After step S506, the top area of ​​each building can be determined on the satellite image. Thereafter, the height feature of each pixel in the third image feature output by the aforementioned first building detection model and the offset feature of each pixel in the fourth image feature can be used to determine the height and offset of each building, and then determine the corresponding bottom area, that is, execute subsequent steps S507 and S508.

[0168] Through step S504 to step S506, this embodiment substantially proposes a multi-model fusion algorithm based on the first building detection model in the aforementioned embodiment. The architecture of the algorithm is as follows: Figure 6 As shown,

[0169] like Figure 6 As shown, an optional multi-model fusion algorithm may include three detection models and one segmentation model, one of the three detection models being the first building detection model of the aforementioned embodiment. When the multi-model fusion algorithm is applied, the satellite image may be simultaneously input into each detection model and the segmentation model, the segmentation model being used to execute the aforementioned step S504, dividing a plurality of blocks of different scenes in the satellite image, and at the same time each detection model being used to execute step S505, respectively detecting the top area of ​​each building in the satellite image. Finally, the top area detection result of each detection model and the block segmentation result of the segmentation model are entered into the model fusion link (equivalent to step S506) of the multi-model fusion algorithm together, and in the model fusion link, the sub-area detected by the detection model corresponding to the block scene in each block is retained, while other sub-areas are deleted.

[0170] After the multi-model fusion phase is completed, the subsequent phases of determining the bottom area of ​​the building and vectorization can be entered to finally obtain the contour vectors of each building in the satellite image. The specific processing procedures of these phases are consistent with the corresponding steps in the aforementioned embodiment and will not be described in detail.

[0171] S507 . For each sub-region, determine the height and offset of the building corresponding to the sub-region according to the third image feature and the fourth image feature.

[0172] S508: For each sub-region, determine the bottom region of the building corresponding to the sub-region according to the sub-region, the height of the building corresponding to the sub-region, and the offset.

[0173] The specific execution process of step S507 and step S508 is consistent with the aforementioned step S104 and step S105 and will not be described in detail.

[0174] Optionally, after step S508 is completed, the following steps may also be performed: Figure 4 The specific process of the vectorization processing in the corresponding embodiment is consistent with step S406 and will not be described in detail.

[0175] Satellite images taken in different scenes have different background features and building features. Using a single building detection model to detect buildings in all scenes may result in the building detection model not being able to identify the features in all scenes well. This solution limits the functions of the building detection model so that each building detection model is only used to detect buildings in images of specific scenes. This can improve the accuracy of buildings detected by each building detection model in its corresponding specific scene. Finally, the blocks of different scenes segmented by the block segmentation model are used to merge the detection results of multiple building detection models with higher accuracy in the corresponding scenes. In this way, it can ensure that every building in the satellite images containing multiple scenes is fully detected, and the accuracy of the final detection results can be further improved.

[0176] Figure 7 A schematic diagram of the detection results of the satellite image processing method provided in the embodiment of the present application in the satellite image. The rectangular boxes indicated by the arrows on the left represent the top area of ​​the building detected in the satellite image, and the rectangular boxes indicated by the arrows on the right represent the bottom area of ​​the building determined by the height and offset in the satellite image.

[0177] In combination with the satellite image processing method provided in any embodiment of the present application, the present application embodiment also provides a satellite image processing device, please refer to Figure 8 , the device may include the following units:

[0178] The acquisition unit 801 is used to acquire satellite images.

[0179] The processing unit 802 is used to process the satellite image using the pre-constructed first building detection model to obtain the first image feature, the second image feature, the third image feature and the fourth image feature of the satellite image.

[0180] Among them, the first building detection model includes a semantic branch, a pixel feature branch, a height branch and an offset branch; the first image feature is output by the semantic branch and is used to indicate the building area in the satellite image; the second image feature is output by the pixel feature branch and includes the pixel feature of each pixel in the satellite image, the third image feature is output by the height branch and includes the height feature of each pixel in the satellite image, and the fourth image feature is output by the offset branch and includes the offset feature of each pixel in the satellite image.

[0181] The classification unit 803 is used to determine the pixel features of the pixels in the building area indicated by the first image feature in the second image feature, and classify the pixels in the building area based on the pixel features of the pixels in the building area indicated by the first image feature, so as to obtain multiple sub-areas into which the building area is divided.

[0182] Each sub-region represents the top of a building in the satellite image.

[0183] The first determination unit 804 is used to determine, for each sub-region, the height of the building corresponding to the sub-region based on the height feature of the pixels located in the sub-region contained in the third image feature, and to determine the offset of the building corresponding to the sub-region based on the offset feature of the pixels located in the sub-region contained in the fourth image feature.

[0184] The second determining unit 805 is configured to determine, for each sub-region, a bottom region of the building corresponding to the sub-region according to the sub-region, the height of the building corresponding to the sub-region, and the offset.

[0185] Optionally, the device further includes a vectorization unit 806, which is used to perform vectorization processing on the sub-areas and bottom areas of each building to obtain a set of outline vectors representing the outline of the building.

[0186] Optionally, when the vectorization unit 806 performs vectorization processing on the sub-area and the bottom area of ​​the building to obtain a set of outline vectors representing the outline of the building, it is specifically used to:

[0187] A plurality of pixels at the edge of the sub-area and the bottom area of ​​the building are determined as key pixels by using a polygon fitting algorithm;

[0188] Rectangular fitting is performed on key pixels located in the building sub-area and key pixels located in the building bottom area respectively to obtain a first rectangle representing the outline of the building sub-area and a second rectangle representing the outline of the building bottom area; wherein the vector constituting the first rectangle and the vector constituting the second rectangle are used as the outline vector of the building.

[0189] Optionally, when the vectorization unit 806 performs vectorization processing on the sub-area and the bottom area of ​​the building to obtain a set of outline vectors representing the outline of the building, it is specifically used to:

[0190] A plurality of pixels at the edge of the sub-area and the bottom area of ​​the building are determined as key pixels by using a polygon fitting algorithm;

[0191] Connecting every two adjacent key pixels located in the sub-area of ​​the building to obtain a corresponding reference vector, and connecting every two adjacent key pixels located in the bottom area of ​​the building to obtain a corresponding reference vector;

[0192] For each two adjacent reference vectors, if the difference between the angles of the two reference vectors and 90° is less than a preset first threshold, adjust the two reference vectors so that the angles of the two reference vectors are equal to 90°;

[0193] For every two adjacent reference vectors, if the angle between the two reference vectors is less than a preset second threshold, the two reference vectors are merged into one reference vector; wherein all the reference vectors after adjustment and merging constitute the outline vector of the building.

[0194] Optionally, when the processing unit 801 processes the satellite image using the pre-constructed first building detection model to obtain the first image feature, the second image feature, the third image feature and the fourth image feature of the satellite image, it is specifically used to:

[0195] Downsampling the satellite image using the downsampling module of the first building detection model to obtain downsampling features of the satellite image;

[0196] The downsampled features of the satellite image are respectively input into the semantic branch, pixel feature branch, height branch and offset branch of the first building detection model to obtain the first image feature output by the semantic branch, the second image feature output by the pixel feature branch, the third image feature output by the height branch and the fourth image feature output by the offset branch.

[0197] Optionally, the first determining unit 804 determines the height of the building corresponding to the sub-region according to the height characteristics of all pixels in the sub-region, and determines the offset of the building corresponding to the sub-region according to the offset characteristics of all pixels in the sub-region, specifically for:

[0198] For each sub-region, the median of the height features of all pixels in the sub-region is determined as the height of the building corresponding to the sub-region, and the median of the offset features of all pixels in the sub-region is determined as the offset of the building corresponding to the sub-region.

[0199] Optionally, the processing device further includes:

[0200] The division unit 807 is used to divide the satellite image into a plurality of blocks according to the scenes contained in the satellite image; wherein each block corresponds to a scene;

[0201] The second processing unit 808 is configured to detect each sub-region of each building in the satellite image using the second building detection model for each pre-constructed second building detection model.

[0202] There is at least one second building detection model; the first building detection model and each second building detection model correspond to a scene in the satellite image.

[0203] The deleting unit 809 is used to delete, for each block, from the block a sub-region of a building detected by the building detection model that does not correspond to the scene of the block.

[0204] The specific working principle of the satellite image processing device provided in this embodiment can refer to the corresponding steps of the satellite image processing method provided in any embodiment of the present application, and will not be repeated here.

[0205] The present application provides a satellite image processing device, wherein an acquisition unit 801 acquires a satellite image; a processing unit 802 processes the satellite image using a semantic branch, a pixel feature branch, a height branch, and an offset branch of a first building detection model to obtain a first image feature indicating a building area in the satellite image, a second image feature composed of pixel features of each pixel in the satellite image, a third image feature including a height feature of each pixel in the satellite image, and a fourth image feature including an offset feature of each pixel; a classification unit 803 divides the building area into sub-areas corresponding to each building according to the second image feature; a first determination unit 804 determines the height and offset of the building according to the height features and offset features of all pixels in the building sub-area, and then a second determination unit 805 determines the bottom area of ​​the building accordingly. Multiple branches in this solution can process satellite images in parallel, so this solution has higher processing efficiency than the prior art.

[0206] An embodiment of the present application also provides a computer storage medium for storing a computer program. When the stored computer program is executed, it is specifically used to implement the satellite image processing method provided in any embodiment of the present application.

[0207] The present application also provides an electronic device, such as Fig. 9 As shown, the electronic device includes a memory 901 and a processor 902 .

[0208] Wherein, the memory 901 is used to store computer programs;

[0209] The processor 902 is used to execute the above-mentioned computer program, specifically to implement the satellite image processing method provided in any embodiment of the present application.

[0210] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the satellite image processing method provided in the above-mentioned various optional implementations.

[0211] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0212] It should be noted that the concepts such as "first" and "second" mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0213] The present application can be implemented or used by professionals and technicians. Various modifications to these embodiments will be apparent to professionals and technicians in the field, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for processing satellite images, characterized in that: include: Access satellite imagery; Downsampling the satellite image using a downsampling module of a pre-constructed first building detection model to obtain downsampling features of the satellite image; The downsampled features of the satellite image are respectively input into the semantic branch, the pixel feature branch, the height branch and the offset branch of the first building detection model to obtain a first image feature output by the semantic branch and used to indicate the building area in the satellite image, a second image feature output by the pixel feature branch and including the pixel feature of each pixel in the satellite image, a third image feature output by the height branch and including the height feature of each pixel in the satellite image, and a fourth image feature output by the offset branch and including the offset feature of each pixel in the satellite image; when the height branch extracts the height feature of each pixel and the offset branch extracts the offset feature of each pixel, the complete downsampled features after the satellite image is downsampled are processed, and the downsampled features include the texture features of the side facades of the buildings in the satellite image; Determine the pixel features of the pixels in the building area indicated by the first image feature in the second image feature, and classify the pixels in the building area based on the determined pixel features of the pixels in the building area indicated by the first image feature to obtain a plurality of sub-areas into which the building area is divided; wherein each of the sub-areas represents the top of a building in the satellite image; For each of the sub-regions, the height of the building corresponding to the sub-region is determined according to the height feature of the pixels located in the sub-region included in the third image feature, and the offset of the building corresponding to the sub-region is determined according to the offset feature of the pixels located in the sub-region included in the fourth image feature; the offset of the building is the offset of the top of the building relative to the bottom of the building in the satellite image; For each of the sub-areas, a bottom area of ​​the building corresponding to the sub-area is determined according to the sub-area, the height of the building corresponding to the sub-area, and the offset.

2. The processing method according to claim 1, characterized in that: After determining the bottom area of ​​the building corresponding to the sub-area according to the sub-area, the height of the building corresponding to the sub-area and the offset for each sub-area, the method further includes: For each of the buildings, vectorization processing is performed on the sub-areas and the bottom area of ​​the building to obtain a set of contour vectors representing the contour of the building.

3. The processing method according to claim 2, characterized in that: The vectorization processing is performed on the sub-area and the bottom area of ​​the building to obtain a set of contour vectors representing the contour of the building, including: Using a polygon fitting algorithm, a plurality of pixels at the edge of the sub-area and the bottom area of ​​the building are determined as key pixels; Rectangular fitting is performed on key pixels located in the sub-area of ​​the building and key pixels located in the bottom area of ​​the building, respectively, to obtain a first rectangle representing the outline of the sub-area of ​​the building and a second rectangle representing the outline of the bottom area of ​​the building; wherein the vectors constituting the first rectangle and the vectors constituting the second rectangle are used as the outline vectors of the building.

4. The processing method according to claim 2, characterized in that: The vectorization processing is performed on the sub-area and the bottom area of ​​the building to obtain a set of contour vectors representing the contour of the building, including: Using a polygon fitting algorithm, a plurality of pixels at the edge of the sub-area and the bottom area of ​​the building are determined as key pixels; Connecting each two adjacent key pixels located in the sub-area of ​​the building to obtain a corresponding reference vector, and connecting each two adjacent key pixels located in the bottom area of ​​the building to obtain a corresponding reference vector; For each two adjacent reference vectors, if the difference between the angle of the two reference vectors and 90° is less than a preset first threshold, adjust the two reference vectors so that the angle of the two reference vectors is equal to 90°; For every two adjacent reference vectors, if the angle between the two reference vectors is less than a preset second threshold, the two reference vectors are merged into one reference vector; wherein all the reference vectors after adjustment and merging constitute the outline vector of the building.

5. The processing method according to claim 1, characterized in that: The step of determining, for each of the sub-regions, the height of a building corresponding to the sub-region according to a height feature of pixels located in the sub-region included in the third image feature, and determining the offset of the building corresponding to the sub-region according to an offset feature of pixels located in the sub-region included in the fourth image feature, includes: For each of the sub-regions, the median of the height features of all pixels in the sub-region is determined as the height of the building corresponding to the sub-region, and the median of the offset features of all pixels in the sub-region is determined as the offset of the building corresponding to the sub-region.

6. The processing method according to any one of claims 1 to 5, characterized in that: Before determining, for each of the sub-regions, the height of the building corresponding to the sub-region according to the height feature of the pixels located in the sub-region included in the third image feature, and determining the offset of the building corresponding to the sub-region according to the offset feature of the pixels located in the sub-region included in the fourth image feature, the method further includes: According to the scenes contained in the satellite image, the satellite image is divided into a plurality of blocks; wherein each of the blocks corresponds to one of the scenes; For each pre-constructed second building detection model, the second building detection model is used to detect and obtain a sub-region of each building in the satellite image; wherein the number of the second building detection models is at least one; the first building detection model and each of the second building detection models correspond to a scene in the satellite image; For each of the blocks, a sub-region of a building detected by the building detection model that does not correspond to the scene of the block is deleted from the block.

7. A satellite image processing device, characterized in that: include: An acquisition unit, used for acquiring satellite images; A processing unit, configured to downsample the satellite image using a downsampling module of a pre-constructed first building detection model to obtain downsampled features of the satellite image; The downsampled features of the satellite image are respectively input into the semantic branch, the pixel feature branch, the height branch and the offset branch of the first building detection model to obtain a first image feature output by the semantic branch and used to indicate the building area in the satellite image, a second image feature output by the pixel feature branch and including the pixel feature of each pixel in the satellite image, a third image feature output by the height branch and including the height feature of each pixel in the satellite image, and a fourth image feature output by the offset branch and including the offset feature of each pixel in the satellite image; when the height branch extracts the height feature of each pixel and the offset branch extracts the offset feature of each pixel, the complete downsampled features after the satellite image is downsampled are processed, and the downsampled features include the texture features of the side facades of the buildings in the satellite image; a classification unit, configured to determine pixel features of pixels in the building area indicated by the first image feature in the second image feature, and classify the pixels in the building area based on the determined pixel features of pixels in the building area indicated by the first image feature, so as to obtain a plurality of sub-areas into which the building area is divided; wherein each of the sub-areas represents a top of a building in the satellite image; a first determining unit, configured to determine, for each of the sub-regions, the height of a building corresponding to the sub-region according to a height feature of pixels located in the sub-region included in the third image feature, and determine an offset of the building corresponding to the sub-region according to an offset feature of pixels located in the sub-region included in the fourth image feature; the offset of the building is an offset of a top of the building relative to a bottom of the building in the satellite image; The second determining unit is used to determine, for each of the sub-areas, a bottom area of ​​the building corresponding to the sub-area according to the sub-area, the height of the building corresponding to the sub-area, and the offset.

8. The processing device according to claim 7, characterized in that The device further comprises: The vectorization unit is used for performing vectorization processing on the sub-area and the bottom area of ​​each building to obtain a set of outline vectors representing the outline of the building.

9. The processing device according to claim 8, characterized in that The vectorization unit is specifically used for: Using a polygon fitting algorithm, a plurality of pixels at the edge of the sub-area and the bottom area of ​​the building are determined as key pixels; Rectangular fitting is performed on key pixels located in the sub-area of ​​the building and key pixels located in the bottom area of ​​the building, respectively, to obtain a first rectangle representing the outline of the sub-area of ​​the building and a second rectangle representing the outline of the bottom area of ​​the building; wherein the vectors constituting the first rectangle and the vectors constituting the second rectangle are used as the outline vectors of the building.

10. The processing device according to claim 8, characterized in that The vectorization unit is specifically used for: Using a polygon fitting algorithm, a plurality of pixels at the edge of the sub-area and the bottom area of ​​the building are determined as key pixels; Connecting each two adjacent key pixels located in the sub-area of ​​the building to obtain a corresponding reference vector, and connecting each two adjacent key pixels located in the bottom area of ​​the building to obtain a corresponding reference vector; For each two adjacent reference vectors, if the difference between the angle of the two reference vectors and 90° is less than a preset first threshold, adjust the two reference vectors so that the angle of the two reference vectors is equal to 90°; For every two adjacent reference vectors, if the angle between the two reference vectors is less than a preset second threshold, the two reference vectors are merged into one reference vector; wherein all the reference vectors after adjustment and merging constitute the outline vector of the building.

11. The processing device according to claim 7, characterized in that The first determining unit is specifically configured to: For each of the sub-regions, the median of the height features of all pixels in the sub-region is determined as the height of the building corresponding to the sub-region, and the median of the offset features of all pixels in the sub-region is determined as the offset of the building corresponding to the sub-region.

12. The processing device according to any one of claims 7 to 11, characterized in that: The device further comprises: A division unit, configured to divide the satellite image into a plurality of blocks according to the scenes contained in the satellite image; wherein each of the blocks corresponds to one of the scenes; a second processing unit, configured to detect, for each pre-constructed second building detection model, a sub-region of each building in the satellite image using the second building detection model; wherein the number of the second building detection models is at least one; and the first building detection model and each of the second building detection models correspond to a scene in the satellite image; The deleting unit is used to delete, for each of the blocks, from the block a sub-region of a building detected by the building detection model that does not correspond to the scene of the block.

13. An electronic device, characterized in that: including memory and processor; Wherein, the memory is used to store computer programs; The processor is used to execute the computer program, specifically to implement the satellite image processing method according to any one of claims 1 to 6.

14. A computer storage medium, characterized in that: Used to store a computer program, which, when executed, is specifically used to implement the satellite image processing method according to any one of claims 1 to 6.

15. A computer program product, characterized in that The computer program product includes computer instructions; the computer instructions are stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the satellite image processing method as described in any one of claims 1 to 6.

Images