Hierarchical joint mapping method and device for community open space in remote sensing images

Abstract

The present invention discloses a hierarchical joint mapping method and device for community open space in remote sensing images, including: a sample generation module, a hierarchical mapping module and an occlusion modeling module; by developing an automated production process for hierarchical community open space samples, a spatially unified hierarchical community open space sample library is produced; a hierarchical occlusion perception model with a dual-branch hierarchical mapping architecture LMA is designed; a cascade map module CGM is designed to represent the occlusion process of ground elements by ground elements, and the upper branch features of LMA are used to guide the screening of lower branch features, thereby enhancing the information transmission between upper and lower branches of LMA. The present invention realizes high-precision extraction of community-scale hierarchical open space based on deep learning, and significantly improves the recognition and mapping capabilities of multi-level, mutually occluded open space elements.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing data processing, and relates to remote sensing data processing technology based on deep learning, and in particular to a hierarchical joint mapping method and device for community open spaces in remote sensing images. Background Art

[0002] Community open space refers to open areas in cities and communities for public use, including green spaces, water bodies, squares and trails. Community open space is not only an important part of the urban ecosystem, but also plays an important role in improving the quality of life of residents, promoting social interaction and enhancing urban resilience. Therefore, accurately obtaining and analyzing community-scale open space information is of great significance for urban planning and sustainable development assessment.

[0003] Remote sensing image data, with its macroscopic and real-time characteristics, has always been an important means of monitoring urban open spaces. At present, community open space classification methods based on remote sensing data are mainly divided into two categories: one is the traditional method based on visual interpretation of remote sensing original images, which uses the spectral characteristics of remote sensing images for manual interpretation and classification; the other is the intelligent classification method using deep learning, which performs automatic processing and classification based on remote sensing data products. These methods achieve refined identification and classification of community open spaces by constructing complex deep learning models, such as convolutional neural networks.

[0004] In order to obtain fine-scale community open space data products, the current single-level land cover mapping scheme using remote sensing data is generally used, which makes it difficult to capture the hierarchical relationship of open space elements and accurately map the obscured ground open space. The development of ultra-high-resolution remote sensing images and non-modal perception deep learning has made it possible to map community open spaces where multiple levels of elements obstruct each other. However, there is currently a lack of effective community open space hierarchical joint mapping technology that overcomes occlusion. Summary of the invention

[0005] In order to overcome the problems existing in the above-mentioned existing community open space mapping methods, the present invention provides a method and device for joint mapping of community open space hierarchies in remote sensing images, so as to solve the problem that elements in community open space in ultra-high-resolution remote sensing images are difficult to extract due to occlusion between each other.

[0006] A. According to a first aspect of an embodiment of the present invention, a hierarchical joint mapping method for community open spaces in remote sensing images is provided, comprising:

[0007] A1. Develop a hierarchical community open space sample automatic production process, design a processing scheme including cross-regional transfer learning, rule decision tree and overlapping cropping, and produce a spatially unified hierarchical community open space sample library. The existing open space sample library does not consider elements of different levels, while the sample library developed by the present invention has a set of ground and ground annotations for one image. In the specific implementation, the public annotation data of trees in local areas and ultra-high-resolution remote sensing images (VHR) are used to train the SegFormer-based transfer learning model, and tree annotation samples covering multiple cities (31 cities) are migrated and produced. Combined with the road, square and water body annotations from the OpenStreetMap (OSM) crowd-source geographic data, a spatially unified high-resolution hierarchical community open space sample library is produced by designing quality screening rules and overlapping cropping schemes, including 5,245 VHR remote sensing image blocks and their corresponding ground element annotation image blocks and ground element annotation image blocks. Each image has a size of 512x512, covering 31 cities, a total of 46,727.56 square kilometers, and a spatial resolution of 1.07 meters.

[0008] A2. A hierarchical joint mapping model with a dual-branch hierarchical mapping architecture (LMA) is designed, and its decoder part is designed as an upper branch and a lower branch to draw the above-ground open space elements and the ground open space elements, respectively.

[0009] Specifically, the design of the hierarchical joint mapping model adopts an encoding-decoding structure. The encoder part of the model adopts the feature encoder of the existing SegFormer model to encode the VHR remote sensing image blocks in the input hierarchical community open space sample library into the depth feature vector of the image; the decoder part of the model is designed as an LMA with an upper and lower branch structure, and its two branches are used to draw the ground and ground open space element maps respectively. After receiving the image depth feature vector output by the encoder, LMA first inputs the feature vector to the upper branch to obtain the ground element feature, and then fuses the ground element feature and the original depth feature output by the encoder through residual connection to obtain the fused feature, and further inputs the fused feature to the lower branch to obtain the ground element feature. In addition, LMA constructs a joint loss for the output features of the two branches to realize the joint mapping of the ground elements and the ground elements.

[0010] A3. Design a cascade graph module (CGM) to model the occlusion process of ground elements by ground elements, use the upper branch features of LMA (ground element features) to guide the screening of lower branch features (ground element features), and enhance the information transmission between the upper and lower branches of LMA.

[0011] Specifically, CGM consists of two graph convolutional networks, a cross-graph attention mechanism, and a gating mechanism. The two graph convolutions are located in the upper and lower branches of LMA, respectively, to extract the spatial relationship of features within a single branch. The cross-graph attention mechanism selects the feature space of the upper and lower graph convolutions, constructs the cross attention between the features of the ground elements and the ground elements, and mines the attention features that characterize the occlusion relationship between the two layers of elements. The gating mechanism obtains the attention feature and uses the occlusion relationship information contained in the feature to filter the features of the lower branch.

[0012] B. According to a second aspect of an embodiment of the present invention, a community open space mapping device is provided, comprising:

[0013] B1. Sample generation module, used to integrate VHR remote sensing data, OSM crowd-source geographic data and public annotation data of the target area into a spatially consistent hierarchical community open space sample library.

[0014] B2. A hierarchical mapping module is used to input the deep features output by the feature encoder into the hierarchical mapping architecture of the hierarchical joint mapping model, perform dual-branch decoding of above-ground open space elements and ground open space elements, and realize joint mapping of above-ground open space elements and ground open space elements through dual-branch joint training.

[0015] B2. Hierarchical mapping module, used to extract the depth features of the image from the image block, input the depth features into the hierarchical mapping architecture of the hierarchical joint mapping model, perform dual-branch decoding of the ground open space elements and the ground open space elements, and realize the joint mapping of the ground open space elements and the ground open space elements through dual-branch joint training. That is, the decoder is designed to be a two-layer LMA architecture to form a new hierarchical joint mapping model.

[0016] B3. Occlusion modeling module, used to identify potential occlusion relationships between the two-branch decoders and transfer occluder features between the two branches. Through graph convolutional networks and cross-graph attention mechanisms, the occlusion relationship of upper-layer elements on lower-layer elements is explicitly modeled to improve mapping accuracy.

[0017] C. According to the third aspect of an embodiment of the present invention, there is also provided an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor calls the program instructions to execute a community open space mapping method provided by any possible implementation of the various possible implementations of the first aspect.

[0018] D. According to the fourth aspect of an embodiment of the present invention, a non-transitory computer-readable storage medium is also provided, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions enable the computer to execute the community open space mapping method provided by any possible implementation of the various possible implementations of the first aspect.

[0019] As a preferred solution, in step A3, CGM includes a graph convolution network, a cross-graph attention mechanism and a gating mechanism. The graph convolution network is used to capture long-range associations in the graph. The cross-graph attention mechanism selects the feature spaces of the two branches in the upper and lower graph convolutions respectively, calculates and forms a cross-adjacency matrix, obtains the occlusion relationship between the ground elements and the ground elements, and forms an inter-level attention feature. The gating mechanism uses the attention feature to guide the selective output of the lower layer features, thereby realizing the transmission of occlusion information from the upper layer to the lower layer.

[0020] As a preferred solution, in step B2, the hierarchical mapping module adopts a dual-branch design, the upper branch is used to decode the above-ground green open space elements, and the lower branch is used to decode the ground open space elements. The features decoded by the upper branch are transmitted to the lower branch through the residual connection, and combined with the original depth features to enhance the recognition ability of the occluded elements.

[0021] As a preferred solution, in step B3, the occlusion modeling module captures the relationship between the occluder and the occluded object by calculating the cross adjacency matrix between the two branches, generates occlusion attention features, and controls the selection and filtering of lower-level features through a gating unit.

[0022] Compared with the prior art, the beneficial technical effects of the present invention are:

[0023] The present invention provides a method and device for hierarchical joint mapping of community open spaces in remote sensing images. By improving the existing deep learning network model, a hierarchical joint mapping model including LMA and CGM is designed, which is suitable for extracting multi-layered, mutually occluded open space elements from remote sensing images. The network has a natural two-layer decoding architecture to adapt to the multi-level spatial structure of the elements, and realizes high-precision extraction of community-scale hierarchical open spaces based on deep learning, significantly improving the recognition and mapping capabilities of multi-level, mutually occluded open space elements. This method solves the problem of overlapping objects that cannot be represented by traditional remote sensing plane mapping schemes, not only improves the classification accuracy of occluded elements, but also realizes automated hierarchical sample generation and large-scale mapping schemes, breaking through the technical and device bottlenecks of fine mapping of community-scale open spaces, saving manpower and time costs, and has broad application prospects and practical significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 A schematic diagram of the overall process of the community open space mapping method provided in an embodiment of the present invention.

[0026] Figure 2 A schematic diagram of the internal structure of a hierarchical mapping architecture in a community open space mapping method provided in an embodiment of the present invention.

[0027] Figure 3 A schematic diagram of the internal structure of the cascade map module in the community open space mapping method provided in an embodiment of the present invention.

[0028] Figure 4 A schematic diagram of the overall structure of a community open space mapping device provided in an embodiment of the present invention.

[0029] Figure 5 A schematic diagram of the overall structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In one embodiment of the present invention, a community open space mapping method of hierarchical joint mapping is provided. Figure 1 The overall flow diagram of the community open space mapping method provided by the embodiment of the present invention includes:

[0031] S101, based on VHR remote sensing image data, OSM crowd-source geographic data and public annotated data, developed a hierarchical community open space sample automated production process, and produced a spatially unified hierarchical community open space sample library, including 5,245 VHR remote sensing image blocks and their corresponding ground element annotated image blocks and ground element annotated image blocks, each image size is 512x512, distributed in 31 major cities, totaling 46727.56 square kilometers, with a spatial resolution of 1.07 meters.

[0032] The very high resolution (VHR) image of Google Earth used in this embodiment is used to study the high-resolution distribution of community open spaces with a resolution of 1.07 meters, including red, green, and blue spectral bands. Google Earth is a widely used data source in the field of remote sensing, and is often used for land use and land cover classification. It generates high-quality images by integrating data from multiple satellite sources. This work collected several TB of urban images in 2023. However, because Google Earth uses stitched images to ensure low cloud cover and high quality, the specific stitched fragments of the image range from 2022 to 2023, and there may be seasonal changes.

[0033] This embodiment uses a series of auxiliary data to construct training data sets and processing results, including OpenStreetMap (OSM), Urban Green Space data set (UGSet), Global Urban Boundary (GUB) and urban landscape data. OSM provides free geospatial data based on volunteer field surveys, including detailed vector layers and attribute annotations in urban areas. OSM provides a large amount of manually annotated land use data related to ground elements, including various vector data related to water bodies, roads and squares. UGSet provides a publicly available urban green space annotation data set, providing 4544 512x512 pixel images from a certain province, which are derived from the Gaofen-2 satellite with a full-color resolution of 1 meter. The present invention uses UGSet to expand and construct a pre-annotated sample set of green spaces covering multiple (31) cities. The GUB data set is derived from artificial impervious areas in 2020 with a resolution of 30 meters. In the present invention, GUB data is used as the maximum urban boundary of community open spaces.

[0034] The definition of community open space in this embodiment is based on the "Specific Assessment of Public Space" guidelines released by the United Nations Human Settlements Programme in 2020 and the metadata manual on open space for indicator 11.7.1 of the Sustainable Development Goals. The United Nations classifies open spaces into green spaces, waterfront areas, parks and urban forests, playgrounds, squares, community gardens, pocket parks, sidewalks, bike lanes, etc. Since the focus of the present invention is on the community scale, we have refined and summarized the original classification system to include green spaces, water bodies, squares and trails, and excluded large urban parks, forests and driveways that exceed the community scale. Therefore, the hierarchical structure of the community open space involved in this embodiment can be described as: the coupling of ground green spaces dominated by trees with ground water bodies, squares and trails.

[0035] The ground element samples of community open spaces in this embodiment mainly refer to green spaces, which are annotated through transfer learning. In this embodiment, a powerful semantic segmentation model SegFormer is trained using public annotated data and migrated to 31 different cities to generate pre-annotated sample blocks.

[0036] The community open space ground element samples of this embodiment, including blue spaces, squares, and trails, are mainly labeled through the reclassification of OSM data. Blue spaces are derived from categories such as rivers, lakes, and fountains labeled in OSM; squares are classified from annotations such as playgrounds, courts, and parks; trails are filtered from OSM roads to filter community-level low-level roads based on road grades. Additional vector rules are set during the annotation acquisition process. For example, square annotations are filtered from OSM parks larger than 1 square kilometer to exclude large plot-level parks. In addition, the annotation of trails requires additional buffering operations to convert them into surface vector labels, because road vectors in crowdsourced geographic data are usually road centerlines without width. According to the road engineering design specifications, a buffer width of 2 to 4 meters is applied according to the trail category to simulate the actual width as much as possible.

[0037] Based on the above embodiment, this embodiment rasterizes the ground feature annotation vector data and the ground feature annotation vector data respectively to align with the Google Earth image with a resolution of 1.07 meters, and uses the GUB data mask to filter out the community open space elements in non-urban areas, and obtains the label raster data of the ground features and ground features of 31 cities. Due to the memory limitations of deep learning, both images and annotations need to be cropped into blocks for use. An overlapping strategy is implemented during the cropping process to reduce performance issues caused by the lack of contextual information around boundary elements. We set a 20% overlap of image blocks to ensure that each 512x512 block contains a 64-pixel wide edge to obtain a smoother boundary prediction at the central 384x384 pixels.

[0038] Based on the above embodiment, this embodiment uses quality filtering for pre-labeled sample blocks to prevent the model from having a reduced generalization ability due to sparse or non-stratified samples. A decision tree based on indicators such as overlap, number of types, and intra-block annotation ratio is designed for sample selection. The overlap rate of community open spaces in the samples ranges from 0 to 50%; we set thresholds of 10% and 20% to classify samples into high, medium, and low overlaps, and gradually tighten other conditions to balance the number of samples with different overlaps. Specifically, for low overlap samples, the design requires that the proportion of labels in sample tiles is greater than 40%, the proportion of trails and square labels is greater than 10%, and the number of label categories is greater than three; for medium overlap samples, the design requires that the proportion of labels in sample tiles is greater than 30%, the proportion of trails and square labels is greater than 8%, and the number of label categories is greater than two; for high overlap samples, the design requires that the proportion of labels in sample tiles is greater than 20%, and the proportion of trails and square labels is greater than 5%. Subsequently, stratified sampling is used in the three overlapping sets to construct training sets, validation sets, and test sets in a ratio of 7:2:1.

[0039] Based on the above-mentioned method for generating a hierarchical community open space sample library, remote sensing data and ground-ground elements are spatially unified to form hierarchical community open space samples. Using the generated images and the corresponding ground and ground element samples, a deep learning semantic segmentation model can be trained, and the trained model can be used to perform integrated segmentation of unlabeled areas, that is, the model can be used to jointly map the ground and ground elements of community open spaces in multiple cities over a large area.

[0040] S102, input the VHR remote sensing image block into the feature encoder of the hierarchical joint mapping model, which uses the feature encoder of SegFormer to obtain the high-dimensional depth feature vector of the image, and input the feature vector into the decoder of the hierarchical mapping architecture (LMA) for ground-ground dual-branch decoding to achieve joint mapping of ground elements and ground elements.

[0041] This embodiment designs a dual-branch LMA to naturally decode the above-ground open space elements of the upper branch and the ground open space elements of the lower branch. The input of the lower ground open space elements is to combine the decoded features of the upper branch with the original depth features output by the encoder through a residual connection. In addition, LMA adopts a multi-task joint learning strategy for end-to-end training. A joint loss function is designed for the task of the present invention, and the upper and lower layers obtain losses at the same time to form a joint loss, and training is performed simultaneously. The final hierarchical mapping results of the upper ground open space elements and the lower ground open space elements are predicted and generated by the upper branch and the lower branch of the LMA respectively.

[0042] The LMA in this embodiment adopts a double-layer design to achieve simultaneous decoding and mapping of above-ground and ground open space elements, aiming to accurately map the multi-level open space elements and their occlusion relationships in urban communities. The structure of LMA is as follows: Figure 2 As shown. LMA follows the three-dimensional spatial structure of the elements and explicitly models the occluding elements in the upper layer, the occluded elements in the lower layer, and the occlusion process in between. The overall idea of ​​LMA is to explicitly construct two branches to process the above-ground and ground elements respectively based on the three-dimensional hierarchical structure of open space elements in reality. The upper branch focuses on identifying elements above the ground, such as trees; while the lower branch aims to extract ground elements, such as water bodies, squares, and trails. At the same time, the residual connection design of the LMA architecture allows the tree feature results of the upper layer to be passed to the ground element branch of the lower layer, providing more information guidance for the lower branch.

[0043] The LMA designed by the present invention has two main features. The first is that the two-layer architecture naturally decouples the hierarchical occlusion relationship and is suitable for multiple semantic expressions of pixels at the same position. The second is that the lower-level branches receive both the original features and the upper-level features, providing clues that help extract the occluded objects. The design of LMA includes CGM and a fully convolutional network T. CGM is responsible for establishing long-range associations between nodes and transferring occlusion knowledge. The fully convolutional network is responsible for the preliminary decoding of the input feature vector and decoding and restoring the features to the original image spatial resolution. The prediction layer is responsible for converting the branch features into the final prediction category. Specifically, F (in) ∈R C×H×W It is defined as the image depth feature provided by the encoder, T is a fully convolutional network including convolution Conv and nonlinear operation δ, where Conv uses 3×3 convolution, CGM u and CGM l Represents the upper and lower CGM modules respectively:

[0044] Y (l) =T(CGM1(T(F (f) ))),

[0045] F (f) =Y (u) +F (in) ,

[0046] Y (u) =T(CGM u (T(F (in) ))),

[0047] T(F)=δ(Conv(F))

[0048] Among them, Y (u) Represents the occlusion element features in the image output by the upper branch, F (f) By integrating Y (u)With the input deep feature F (in) The residual connection features formed are used as the input of the lower branch, Y (l) It is the result feature used to predict the occluded elements in the underlying layer.

[0049] In order to achieve end-to-end training of the two branches of LMA, this embodiment adopts a multi-task joint learning strategy. The joint loss function L of the open space elements is designed. The joint L (u) and L (l) L denotes the loss of the upper layer and the lower layer respectively, and α and β are weights used to balance the loss. BCE represents the binary cross entropy loss. sigmoid and softmax are probability conversion functions, W u and W l It is the trainable prediction layer weight for the upper and lower layers, using 1×1 convolution, sigmoid (WY) to form the prediction layer of each branch, which is used to convert branch features into the final prediction category. GT (u) and GT (l) are the true annotations of the upper and lower objects, c is the category of the occluded object, and w c Different open space elements in L (l) After calculating the binary cross entropy loss of the upper layer and the multi-class cross entropy loss of the lower layer, the hyperparameter weights α, β, and w c Control loss fusion. In this embodiment, α and β are set to {0.5, 1}, w c Set to {0.2, 0.3, 0.6}. :

[0050] L=αL (u) +βL (l)

[0051] L (u) =L BCE (sigmoid(W u Y (u) ), GT (u) ),

[0052]

[0053] It can be seen that LMA creates different semantic spaces for above-ground open space elements and ground open space elements. Therefore, a single pixel node overlapping the upper and lower layers can correspond to two different states Y in the two-layer graph. (u) and Y (l) , naturally decouples the attention to occluders and occluded objects. In addition, by providing residual connections from upper-level branches to lower-level branches, occluded objects are no longer considered as interference. Instead, the knowledge of occluders helps understand the occlusion process, providing important clues for decoupling occlusion relations.

[0054] S103, the cascade graph module CGM is an important processing unit in LMA, and it realizes important occlusion information transmission. The CGM designed in the embodiment combines the graph convolution network, cross-graph attention and gating mechanism to learn the occlusion relationship between community open space elements and solve the problem of feature information loss after the ground elements are blocked by trees. Figure 3 The structure of CGM is presented, which models occlusion relationships by establishing cross-layer information transfer between graph convolutions in upper and lower layers.

[0055] Graph convolution GCN is the structure inside CGM. In the CGM structure, GCN upper and GCN lower Represents the graph convolution operations of CGM in the upper and lower branches respectively. GCN upper The standard general graph convolutional network (GCN) structure is adopted, and GCN lower The occlusion knowledge guidance branch is added to the general GCN, and the occlusion process from top to bottom is established from GCN. upper To GCN lower knowledge guidance.

[0056] For the upper layer graph convolution GCN upper , given an adjacency graph G = (V, E) containing a node V and an edge E, the output feature of the graph convolution operation is represented as:

[0057] Z=ρ(AVγ)+V,

[0058] Where V∈R C×N is the input feature, N = H × W is the number of pixels in the feature map, C is the feature dimension of each node, A∈R C×N is the adjacency matrix that represents the neighborhood relationship of graph nodes based on feature similarity, γ∈R C×C′ is the learnable weight matrix of the output transformation, where C′=C. Output feature Z∈R N×C′ The node features updated based on the global graph network information propagation are obtained through a nonlinear function ρ(·) including layer normalization and ReLU function.

[0059] A ij =softmax(F(v i , v j ))

[0060] F(v i , v j )=θ(v i ) T φ(v j )

[0061] Among them, θ and φ are two trainable transformation functions, which implement non-local operations through 1×1 convolution to generate high-confidence feature similarity edges.

[0062] Based on the previous embodiment, this embodiment designs a cross-graph attention mechanism to enable CGM to capture occlusion relationships. CGM takes into account the upper The feature space θ upper and from GCN lower The feature space φ 1ower , construct a cross-adjacency matrix A between layers cross :

[0063] A cross =softmax(F cross (v i ,u j ))

[0064] F cross (v i ,u j )=θ upper (v i ) T φ lower (u j )

[0065] where v i and u i They are the upper graph V∈R C×N and the underlying graph U∈R C×N , T is the vector transpose operation.

[0066] Cross-adjacency matrix A cross Represents the relationship between the occluder and the occluded object. By calculating A cross With the lower branch γ lower The non-local weighted result of the space is used to obtain the occlusion attention feature Ω representing the relevant information of the occluded area. cross Finally, CGM designed a gating unit based on Ω cross The signal controls the selection and filtering of lower-level features. In this way, the model can selectively output lower-level features based on occlusion knowledge and more effectively focus on information in overlapping areas.

[0067]

[0068] G=σ(Ω cross )

[0069] Ω cross =ρ(A cross Vγ lower )

[0070] The gating mechanism is implemented by the sigmoid function σ(·). cross The gating signal G is formed by the gating unit. Symbol represents the element-by-element multiplication of the Hadamard product. The gate signal G is obtained by the Hadamard product. Filtering GCN lower The output features of , and the final feature Z is obtained by residual connection with the input feature U lower .

[0071] In summary, this embodiment improves the existing deep learning network, and the designed hierarchical joint mapping model is suitable for extracting multi-layer, mutually occluded open space elements from remote sensing images. The network has a natural two-layer decoding architecture to adapt to the multi-level spatial structure of the elements. The LMA designed in the decoding part makes it possible to synchronously map the three-dimensional structure of the community open space. The CGM module in LMA promotes the learning and understanding of the occlusion relationship when three-dimensional spatial features are imaged in two-dimensional space, and performs well in solving the problem of hierarchical occlusion in remote sensing images. This embodiment realizes the extraction of community-scale hierarchical open space based on deep learning, and achieves the most advanced mapping effect.

[0072] Based on the above examples, all codes of this example are implemented in Python. In order to facilitate sample generation, libraries such as GDAL and Shapely are used. The hierarchical joint mapping model is implemented using mmsegmentation (an open source deep learning toolbox based on PyTorch). The experiment is run on a 5600X CPU and a 24GB NVIDIA GeForce RTX3090 GPU.

[0073] In all experiments, the optimizer, initial learning rate, learning rate strategy, maximum number of training iterations, and training batch size were set to SGD, 0.001, PolyLR, 160000, and 2, respectively. To accelerate the convergence of the hierarchical joint mapping model, the parameters of the SegFormer-b3 model pre-trained on ImageNet were used as the initial parameters of the encoder. In addition, to prevent overfitting in deep learning, an early stopping strategy was adopted; if the F1 score on the validation set did not improve in five consecutive checks every 4000 iterations, the training would terminate.

[0074] In another embodiment of the present invention, a community open space mapping device is provided, which is used to implement the methods in the above embodiments. Therefore, the description and definition in the above embodiments of the community open space element extraction method can be used to understand the various execution modules in the embodiments of the present invention. Figure 4A schematic diagram of the overall structure of a community open space mapping device provided in an embodiment of the present invention, the device includes a sample generation module 401, a hierarchical mapping module 402 and an occlusion modeling module 403; wherein the sample module 401 is used to couple the remote sensing data, crowd-source geographic data and open source annotation data of the target area into a spatially consistent hierarchical structure; the hierarchical mapping module 402 and the occlusion modeling module 403 are important components of the decoding part of the proposed hierarchical joint mapping model, and realize the hierarchical modeling of the above-ground open space elements.

[0075] Among them, the target area is an urban area where community open space mapping is required, and the sample generation module 401 is responsible for integrating the remote sensing data, crowd-sourced geographic data and open source annotation data of the target area into a spatially consistent hierarchical community open space sample library. Specifically, remote sensing data includes high-resolution multispectral and hyperspectral images, which are used to provide detailed community open space information. Crowd-sourced geographic data comes from platforms such as OpenStreetMap, and contains detailed vector layers and attribute annotations contributed by volunteers, such as roads, water bodies, buildings and green spaces. Open source annotation data uses publicly available annotation data such as Urban Green Space datasets, which contain annotated urban green spaces and other open space elements. However, open source annotation data will face the problem of inconsistent spatial coverage, non-overlapping with the target area, and inability to provide annotation information for the target area during use. To this end, the sample generation module trains the SegFormer model based on the open source annotation data for transfer learning to form a pre-annotated sample of urban green space in the target area. By further screening, processing and fusion of these data from different sources, a hierarchical community open space sample library with a unified spatial reference and resolution is generated. The sample library consists of 512x512 pixel remote sensing images and two sets of corresponding ground-ground samples, covering different urban areas to ensure data diversity and representativeness.

[0076] The hierarchical mapping module 402 obtains the deep VHR remote sensing image feature vector encoded by the hierarchical joint mapping model encoder, inputs the deep feature vector into the LMA hierarchical mapping architecture, and performs dual-branch decoding of the ground and ground open space elements. Specifically, LMA adopts a dual-branch design, one branch is used to decode the upper-layer ground open space elements, such as trees, etc., and the other branch is used to decode the lower-layer ground open space elements, such as water bodies, squares and trails. The decoded features of the upper branch are passed to the lower branch through the residual connection, and combined with the original depth features to enhance the recognition ability of occluded elements. LMA adopts a multi-task joint learning strategy to achieve end-to-end training. The final hierarchical mapping results of the open space elements are jointly predicted and generated by the two branches, which improves the accuracy and robustness of the model.

[0077] The occlusion modeling module 403 is located in the LMA hierarchical mapping architecture and is used to identify occlusion relationships and transfer occlusion knowledge between the two branches. The specific implementation method includes using a graph convolutional network to capture long-range associations within a layer in a single branch to model complex spatial relationships. Between the upper and lower layers, a cross-graph attention mechanism and a gating mechanism are designed to explicitly model the top-down occlusion pattern in the VHR image and transfer occlusion knowledge. In this way, the model can more effectively handle the occlusion problem of multi-level open space elements and improve mapping accuracy.

[0078] This embodiment provides an electronic device, Figure 5 The overall structure diagram of the electronic device provided by the embodiment of the present invention includes: at least one processor 501, at least one memory 502 and a bus 503; wherein,

[0079] The processor 501 and the memory 502 communicate with each other via the bus 503;

[0080] The memory 502 stores program instructions that can be executed by the processor 501. The processor calls the program instructions to execute the methods provided by the above-mentioned method embodiments, for example, including: creating a spatially unified hierarchical community open space sample library through transfer learning based on remote sensing image data, multi-source geographic data and open source annotation data; inputting pre-annotated image blocks into the feature encoder, extracting the deep features of the VHR image, and using the feature pyramid network for multi-scale fusion; combining the multi-task joint learning strategy to hierarchically decode the ground-ground element features through the dual-branch LMA; constructing the occlusion relationship of the community open space through CGM, transferring the upper-level features to the lower-level branches, and explicitly modeling the occlusion process of the upper-level elements on the lower-level elements. Finally, the device can generate accurate community open space hierarchical mapping results.

[0081] This embodiment provides a non-transitory computer-readable storage medium, which stores computer instructions. The computer instructions enable the computer to execute the methods provided by the above-mentioned method embodiments, for example, including: integrating the remote sensing image data, multi-source geographic data and development annotation data of the target area to produce a spatially unified hierarchical community open space sample library; using Google Earth's ultra-high resolution images to conduct a high-resolution community open space distribution study to generate a high-resolution hierarchical community open space dataset containing multiple types of annotations; combining transfer learning, rule decision trees, and overlapping cropping and splicing to construct a training dataset and processing results. The annotated image blocks are input into the feature encoder to extract the deep features of the VHR image, and the hierarchical structure of the community open space elements in the real three-dimensional space is predicted through the LMA hierarchical mapping architecture and the CGM cascade map module.

[0082] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, etc., various media that can store program codes.

[0083] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0084] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0085] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A hierarchical joint mapping method for community open spaces in remote sensing images, characterized in that: The steps include: A1. Develop an automated production process for hierarchical community open space samples, and produce a hierarchical community open space sample library with unified horizontal spatial positions based on open source public data. Each remote sensing image block in the sample library is equipped with a set of ground element annotations and above-ground element annotations; A2. Design a hierarchical occlusion-aware model with a dual-branch hierarchical mapping architecture (LMA) to explicitly decouple the overlapping occluded community open spaces in remote sensing images and perform joint mapping of above-ground and ground elements; The hierarchical occlusion perception model includes an encoder and a decoder; the encoder is used to encode the input remote sensing image in the hierarchical community open space sample library into a deep feature vector; The decoder is a hierarchical mapping architecture LMA with a dual-branch structure including an upper branch and a lower branch, which are used to draw the above-ground open space elements and the ground open space elements respectively; the decoder receives the deep feature vector output by the encoder and inputs it into the upper branch for processing to obtain the above-ground open space element features; Then, the ground open space element features and the deep features output by the encoder are fused through residual connection to obtain fused features; the fused features are input into the lower layer branch for processing to obtain the ground open space element features; And construct a joint loss of dual-branch output features to achieve joint mapping of above-ground open space elements and ground open space elements; A3. Design a cascade graph module (CGM) to represent the occlusion process of ground elements by ground elements. Use the upper branch features of LMA to guide the screening of lower branch features and enhance the information transmission between the upper and lower branches of LMA. The cascaded graph module (CGM) consists of two graph convolutional networks, a cross-graph attention mechanism, and a gating mechanism. Two graph convolutions are located in the upper and lower branches of LMA, respectively, to extract the spatial relationship of features within a single branch; The cross-graph attention mechanism is used to select the feature spaces of two graph convolutions, construct the cross-attention between the features of the ground elements and the ground elements, and mine the attention features that represent the occlusion relationship between the two layers of elements; The gating mechanism obtains the attention features and uses the occlusion relationship information contained in the attention features to filter the features of the lower-level branches; After classification through the classification output layers of the upper and lower branches, semantic segmentation maps of the ground and open space elements on the ground are obtained.

2. The hierarchical joint mapping method of community open space in remote sensing images as claimed in claim 1, characterized in that: The data used to construct the training dataset include: OpenStreetMap, Urban Green Space dataset, Global UrbanBoundary data, and urban landscape data.

3. The hierarchical joint mapping method of community open space in remote sensing images as claimed in claim 1, characterized in that: In step A1, the public tree annotation data of local areas and ultra-high-resolution remote sensing images VHR are used to train a SegFormer-based transfer learning model to migrate tree annotation samples covering multiple cities. Based on the road, square and water body annotations of OpenStreetMap crowd-source geographic data, a spatially unified high-resolution hierarchical community open space sample library is created by designing quality screening and overlapping cropping methods, including VHR remote sensing image blocks and their corresponding ground element annotation image blocks and ground element annotation image blocks.

4. The hierarchical joint mapping method of community open space in remote sensing images as claimed in claim 3, characterized in that: Specifically, the semantic segmentation model SegFormer is trained using publicly labeled data and migrated to generate pre-labeled sample blocks. The pre-labeled sample blocks are quality filtered, and a decision tree based on overlap, number of types, and intra-block annotation ratio is designed for sample selection. A threshold is set for the overlap rate of community open spaces in the samples, and the samples are classified into high, medium, and low overlap. A stratified sampling method is used in different overlap sets to construct training sets, validation sets, and test sets in different proportions.

5. The hierarchical joint mapping method of community open space in remote sensing images as claimed in claim 1, characterized in that: LMA includes CGM and a fully convolutional network T, which is used to naturally decouple the hierarchical occlusion relationship for the two-layer architecture, and the lower branch receives both the original features and the upper features, providing clues that help extract the occluded objects; it is expressed as: AND (l) =T(CGM l (T(F (f) ))), F (f) =Y (u) +F (in) , AND (u) =T(CGM u (T(F (in) ))), T(F)=δ(Conv(F)) Among them, Y (l) It is the result feature used to predict the occluded elements in the lower layer; T is the full convolutional network; F (f) By integrating Y (u) With the input deep feature F (in) The residual connection features formed are used as the input of the lower branch; Y (u) Represents the characteristics of occluded elements in the image output by the upper branch; CGM u and CGM l They represent the CGM modules of the upper branch and the lower branch respectively; Conv represents the convolution operation; δ represents the nonlinear operation.

6. The hierarchical joint mapping method of community open space in remote sensing images as claimed in claim 5, characterized in that: In the CGM structure, GCN upper and GCN lower Represents the graph convolution operation of CGM in the upper branch and the lower branch respectively; GCN upper Adopt graph convolutional network GCN structure; GCN lower Add an occlusion knowledge guidance branch on GCN, and build a GCN from the top to the bottom occlusion process upper To GCN lower knowledge guidance; And construct the cross-adjacency matrix A between layers cross , expressed as: A cross =softmax(F cross (v i ,u j )) F cross (v i ,u j )=θ upper (v i ) T φ lower (u j ) where v i and u j They are the upper graph V∈R C×N and the underlying graph U∈R C×N Nodes, T is the vector transpose operation; cross adjacency matrix A cross Represents the relationship between the occluder and the occluded object; θ upper For GCN upper The feature space of φ lower For GCN lower The feature space of CGM also designs a gating unit to implement occlusion-based attention feature Ω cross The signal controls the selection and filtering of the lower-level features, expressed as: G=σ(Ω cross ) Oh cross =ρ(A cross Vγ lower ) Among them, γ lower represents the lower layer branch; the gating mechanism is implemented by the sigmoid function σ(·); the occlusion attention feature Ω cross The gating signal G is formed by the gating unit; symbol represents the element-wise multiplication of the Hadamard product; the gated signal G passes through the Hadamard product; filtering GCN lower The output features of , and the final feature Z is obtained by residual connection with the input feature U lower .

7. The hierarchical joint mapping method of community open space in remote sensing images as claimed in claim 6, characterized in that: In step A2, a joint loss function is designed, and the upper and lower branches simultaneously obtain losses to form a joint loss, and training is performed simultaneously; The joint loss of the dual-branch output features, that is, the joint loss function of the open space elements, is expressed as: L=αL (u) +BL (l) L (u) =L BCE (sigmoid(W u Y (u) ),GT (u) ), Among them, L is the joint loss function; L (u) and L (l) denote the losses of the upper and lower branches respectively; α and β are the hyperparameter weights used to balance the losses; L BCE represents the binary cross entropy loss; sigmoid and softmax are probability conversion functions; sigmoid is used to convert branch features into prediction categories; W u and W l are the trainable prediction layer weights for the upper and lower branches respectively; Y (u) Represents the occlusion element features in the image output by the upper branch; Y (l) It is the result feature used to predict the occluded elements in the lower layer; GT (u) and GT (l) are the true annotations of the upper and lower objects respectively; c is the category of the occluded object; w c Different open space elements in L (l) The weight in .

8. A device using the community open space mapping method with hierarchical occlusion perception according to claim 1, characterized in that it comprises: Sample generation module, layered mapping module and occlusion modeling module; among them, The sample generation module is used to integrate VHR remote sensing data, OSM crowd-sourced geographic data and public annotation data of the target area into a spatially consistent hierarchical community open space sample library; The hierarchical mapping module is used to input the deep features output by the feature extraction module into the hierarchical occlusion perception model with a dual-branch hierarchical mapping architecture, perform dual-branch decoding of above-ground open space elements and ground open space elements, and realize hierarchical mapping and joint mapping of above-ground and ground open space elements; The occlusion modeling module is used to identify the potential occlusion relationship between the two-branch decoders of the hierarchical occlusion perception model and transfer the occluder features between the two branches; the occlusion relationship of the upper-level elements to the lower-level elements is modeled through the graph convolutional network and cross-graph attention mechanism to improve the mapping accuracy.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the community open space mapping method with hierarchical occlusion perception as claimed in claim 1 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the community open space mapping method with hierarchical occlusion perception as claimed in claim 1 is implemented.

Images