Urban surface feature height reconstruction method and device based on remote sensing synthetic data
By constructing a synthetic urban remote sensing dataset and a multi-scale dual encoder network framework, and combining band-guided indexes for self-supervision and transfer learning, the problems of low accuracy in urban feature height reconstruction and difficulty in inter-sensor transfer were solved, achieving efficient and accurate urban feature height reconstruction.
Patent Information
- Application Number
- CN202511624086.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-07
- Publication Date
- 2026-02-17
AI Technical Summary
In existing technologies, the accuracy of urban feature height reconstruction is low, and traditional methods have high data acquisition and manual processing costs. Deep learning models have difficulty in predicting migration between different satellite sensors, and the lack of sufficient pixel-level height image data leads to insufficient model accuracy and robustness.
A synthetic dataset of urban remote sensing images was constructed. A multi-scale dual encoder network framework was used for self-supervised and transfer-supervised learning. A pre-trained model was generated and optimized by combining a band-guided index. Transfer learning was then performed using real remote sensing images to form an urban feature height reconstruction model.
It improves the model's feature representation and generalization capabilities, solves the problem of difficult transfer between different sensors, ensures high reconstruction accuracy of the model in real-world scenarios, reduces data acquisition costs, and improves model accuracy.
Smart Images
Figure CN121544747A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of remote sensing image processing technology, and in particular to a method and apparatus for reconstructing the height of urban features based on remote sensing synthetic data. Background Technology
[0002] As urbanization continues, vertical information about cities is becoming increasingly important. Urban feature height data plays a crucial role in urban planning and design, disaster assessment and emergency response, and the construction of intelligent transportation systems. It is also key to evaluating the level of the urban ecological environment.
[0003] Traditional methods for extracting urban feature height information primarily rely on multi-view photogrammetry or LiDAR point cloud methods. However, these traditional methods incur high costs for data acquisition and manual processing, and their data acquisition efficiency is insufficient to meet the needs of large-scale smart city construction and dynamic monitoring. Newer intelligent methods mainly establish a mapping relationship between remote sensing imagery and feature height information through deep learning networks, extracting heights from single remote sensing images. However, this intelligent approach has several unresolved issues: First, remote sensing images obtained from different satellite sensors exhibit variations, including satellite observation angles, spatial resolution, and spectral resolution. Currently, prediction models using supervised learning methods cannot achieve data transfer prediction between sensors. Second, architectural styles and spatial layouts vary significantly across different cities. Training deep learning models requires sufficient reference data, namely real remote sensing images and their corresponding height image labels. However, pixel-level height imagery data for ground buildings is currently scarce, and its quality is difficult to guarantee. This leads to insufficient training samples, causing prediction models to struggle with accuracy and robustness, and potentially resulting in overfitting, thus significantly impacting the accuracy of prediction results. Summary of the Invention
[0004] This invention provides a method and apparatus for urban feature height reconstruction based on remote sensing synthetic data, which can solve the problem of low accuracy in urban feature height reconstruction in the prior art.
[0005] To address the aforementioned technical problems, this invention provides a method for urban feature height reconstruction based on remote sensing synthetic data, comprising:
[0006] Construct a synthetic remote sensing dataset for cities; wherein, the synthetic remote sensing dataset for cities includes synthetic remote sensing images of several urban scenes and corresponding synthetic height images;
[0007] Based on real remote sensing images collected by satellite sensors, a first real remote sensing training set and a second real remote sensing training set were formed respectively.
[0008] The true height images of each real remote sensing image in the second real remote sensing training set are determined respectively, and a true height image set is formed;
[0009] Based on the first real remote sensing training set and the preset multi-scale dual encoder network framework, the geometric spatial features of remote sensing images are learned through self-supervised training to generate a pre-trained model.
[0010] Based on the aforementioned urban remote sensing synthetic dataset, the pre-trained model is subjected to supervised training to form an optimized model.
[0011] Based on the second real remote sensing training set and the real height image set, the optimized model is subjected to transfer supervised learning to form an urban feature height reconstruction model.
[0012] Real-time acquisition of urban remote sensing images, and the use of the urban feature height reconstruction model to reconstruct the height of the urban remote sensing images to generate real-time height images.
[0013] As a preferred embodiment, the preset multi-scale dual encoder network framework includes: a preprocessing module, a first feature extraction encoder, a second feature extraction encoder, a feature fusion unit, and a decoder;
[0014] The preprocessing module is used to perform spectral band analysis on remote sensing images and generate band guiding indices;
[0015] The first feature extraction encoder is used to extract features from the remote sensing image using a Comba module and a first KAN multilayer perceptron neural network to obtain a first feature;
[0016] The second feature extraction encoder is used to extract features from the remote sensing image using a feature extraction convolution module and a second KAN multilayer perceptron to obtain a second feature;
[0017] The feature fusion unit is used to fuse the first feature and the second feature to generate a fused feature;
[0018] The decoder is used to restore the fused features into a height image based on the band guidance index.
[0019] As a preferred embodiment, the step of performing spectral band analysis on the remote sensing image to generate a band guiding index includes:
[0020] Determine the reflectance values of the red band, blue band, and near-infrared band of the remote sensing image respectively;
[0021] The normalized vegetation index is calculated based on the red light band reflectance value and the near-infrared band reflectance value.
[0022] The enhanced vegetation index is calculated based on the red light band reflectance value, the blue light band reflectance value, and the near-infrared band reflectance value.
[0023] The corrected soil-regulated vegetation index is calculated based on the red light band reflectance value and the near-infrared band reflectance value.
[0024] The OTU method is used to reassign values to the pixels of the remote sensing image to form an adjusted remote sensing image;
[0025] Based on the adjusted remote sensing image, the normalized vegetation index, the enhanced vegetation index, and the modified soil-regulated vegetation index are adjusted, and the adjusted normalized vegetation index, enhanced vegetation index, and modified soil-regulated vegetation index are stitched together in the channel dimension to generate a band guiding index.
[0026] As a preferred embodiment, the step of using a Comba module and a first KAN multilayer perceptron neural network to extract features from the remote sensing image to obtain a first feature includes:
[0027] The Comba module is used to perform feature transformation on the remote sensing image, and calculate the key vector, value vector and query vector respectively;
[0028] Calculate the model memory state based on the key vector and the value vector;
[0029] Based on the key vector, the query vector, and the model memory state, the first intermediate feature output by the Comba module is derived;
[0030] The first intermediate feature is obtained by nonlinear transformation using a first KAN multilayer perceptron neural network.
[0031] As a preferred embodiment, the step of using a feature extraction convolutional module and a second KAN multilayer perceptron to extract features from the remote sensing image to obtain a second feature includes:
[0032] Combine KAN convolution and standard convolution into a feature extraction convolution module;
[0033] The feature extraction convolution module is used to extract features from the remote sensing image to obtain a second intermediate feature.
[0034] The second intermediate feature is added to the remote sensing image by residual to obtain the enhanced feature;
[0035] The enhanced features are nonlinearly transformed using a second KAN multilayer perceptron to obtain the second feature.
[0036] As a preferred embodiment, the step of fusing the first feature and the second feature to generate a fused feature includes:
[0037] For the first and second features at the same layer, the following steps are used to fuse features and generate fused features:
[0038] The first feature and the second feature are selected by using a gated loop unit to obtain the selected first feature and the selected second feature.
[0039] Obtain the previous layer fused features output by the feature fusion builder;
[0040] The selected first feature, the selected second feature, and the previous layer fusion feature are added and fused together to generate a fusion feature.
[0041] As a preferred embodiment, the step of restoring the fused features into a height image based on the band guidance index includes:
[0042] Starting with the deepest fusion feature, feature restoration processing is performed on the fusion features of each layer in turn to obtain the height image;
[0043] Specifically, the fused features undergo feature restoration processing, which includes:
[0044] The fusion features of the current layer are processed layer by layer through a multi-layer ViT and upsampling structure to obtain several layers of corresponding decoding features; wherein, the layer corresponding to each decoding feature is higher than the layer of the current layer;
[0045] Based on the band guidance index, the weights of the fusion features corresponding to each decoding feature level are adjusted to form the filtered fusion features;
[0046] Each decoded feature is concatenated with the corresponding level of filtered and fused features along the channel dimension to restore the decoded features into a single-channel image, thus obtaining a height image.
[0047] As a preferred embodiment, the construction of the urban remote sensing synthetic dataset includes:
[0048] Several urban scenes were modeled in 3D using modeling software; each urban scene has different urban characteristics.
[0049] The graphics engine is used to adjust the light source, shooting angle, and camera wavelength to generate several shooting parameters.
[0050] Based on the aforementioned city scenes and the aforementioned shooting parameters, several remote sensing composite data are generated; wherein, the remote sensing composite data includes composite remote sensing images and corresponding composite height images;
[0051] By integrating several of the aforementioned remote sensing composite data, a city remote sensing composite dataset is generated.
[0052] As a preferred embodiment, the step of determining the true height images of each real remote sensing image in the second real remote sensing training set and forming a true height image set includes:
[0053] For two real remote sensing images taken from different perspectives in the same area, obtain the satellite parameters corresponding to each real remote sensing image;
[0054] Stereo matching is performed on each pixel of the two real remote sensing images to generate a disparity map;
[0055] Based on the satellite parameters of the two real remote sensing images and the disparity map, the reference plane height of each pixel in each real remote sensing image is determined respectively;
[0056] For each real remote sensing image, a true height image is generated by arranging the pixels according to their positions on the real remote sensing image, based on the reference plane height of each pixel.
[0057] Accordingly, the present invention provides an urban feature height reconstruction device based on remote sensing synthetic data, comprising: a synthetic data acquisition module, a real data acquisition module, a real height image generation module, a model pre-training module, a model supervised training module, a model transfer learning module, and a model application module;
[0058] The synthetic data acquisition module is used to construct a city remote sensing synthetic dataset; wherein, the city remote sensing synthetic dataset includes synthetic remote sensing images of several city scenes and corresponding synthetic height images;
[0059] The real data acquisition module is used to collect real remote sensing images based on satellite sensors, and form a first real remote sensing training set and a second real remote sensing training set respectively.
[0060] The true height image generation module is used to determine the true height images of each true remote sensing image in the second true remote sensing training set, and form a true height image set;
[0061] The model pre-training module is used to learn the geometric spatial features of remote sensing images through self-supervised training based on the first real remote sensing training set and a preset multi-scale dual encoder network framework, and generate a pre-trained model.
[0062] The model supervised training module is used to supervise the training of the pre-trained model based on the urban remote sensing synthetic dataset to form an optimized model;
[0063] The model transfer learning module is used to perform supervised transfer learning on the optimized model based on the second real remote sensing training set and the real height image set to form an urban feature height reconstruction model.
[0064] The model application module is used to acquire urban remote sensing images in real time, and to reconstruct the height of the urban remote sensing images using the urban feature height reconstruction model to generate real-time height images.
[0065] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:
[0066] This invention provides a method for urban feature height reconstruction based on remote sensing synthetic data. The method involves constructing an urban remote sensing synthetic dataset; acquiring real remote sensing images using satellite sensors to form a first real remote sensing training set and a second real remote sensing training set; determining the true height images of each real remote sensing image in the second real remote sensing training set to form a true height image set; using the first real remote sensing training set and a pre-set multi-scale dual encoder network framework, self-supervised training is performed to learn the geometric spatial features of the remote sensing images, generating a pre-trained model; supervised training is then performed on the pre-trained model using the urban remote sensing synthetic dataset to form an optimized model; transfer-supervised learning is then performed on the optimized model using the second real remote sensing training set and the true height image set to form an urban feature height reconstruction model; and urban remote sensing images are acquired in real time, and the urban feature height reconstruction model is used to reconstruct the height of the urban remote sensing images, generating real-time height images. This invention utilizes self-supervised learning based on the first real remote sensing training set, which effectively improves the model's feature representation ability and generalization ability in downstream tasks, thereby solving the problem of difficulty in transferring supervised learning across different sensor images. This invention uses urban remote sensing synthetic data rich in image information as model training data, which can effectively solve the problem of insufficient reference data; and by using a second real remote sensing training set to perform transfer learning on the model, the model's height reconstruction capability in real scenes can be guaranteed. Therefore, the urban feature height reconstruction model trained by this invention can effectively improve the accuracy of urban feature height reconstruction. Attached Figure Description
[0067] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0068] Figure 1 This is a flowchart illustrating an embodiment of the urban feature height reconstruction method based on remote sensing synthetic data provided by the present invention.
[0069] Figure 2A flowchart illustrating an embodiment of the band guidance index calculation method provided by the present invention;
[0070] Figure 3 A schematic diagram of the structure of an embodiment of the first feature extraction encoder provided by the present invention;
[0071] Figure 4 This is a schematic diagram of the structure of an embodiment of the second feature extraction encoder provided by the present invention;
[0072] Figure 5 A schematic diagram of the structure of one embodiment of the feature fusion device provided by the present invention;
[0073] Figure 6 This is a schematic diagram of one embodiment of the urban feature height reconstruction device based on remote sensing synthetic data provided by the present invention. Detailed Implementation
[0074] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0075] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.
[0076] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.
[0077] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.
[0078] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.
[0079] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), similarly, "multiple sets" refers to two or more (including two sets), and "multiple pieces" refers to two or more (including two pieces).
[0080] In the description of the embodiments of this application, unless otherwise expressly specified and limited, technical terms such as "installation," "connection," "joining," and "fixing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. For those skilled in the art, the specific meaning of the above terms in the embodiments of this application can be understood according to the specific circumstances.
[0081] See Figure 1 To address the problem of low accuracy in urban feature height reconstruction in existing technologies, an embodiment of the present invention provides an urban feature height reconstruction method based on remote sensing synthetic data. This method includes steps 101 to 107, each step of which is detailed below:
[0082] Step 101: Construct a synthetic urban remote sensing dataset; wherein the synthetic urban remote sensing dataset includes synthetic remote sensing images of several urban scenes and corresponding synthetic height images.
[0083] Step 102: Collect real remote sensing images based on satellite sensors to form the first real remote sensing training set and the second real remote sensing training set.
[0084] Step 103: Determine the true height image of each real remote sensing image in the second real remote sensing training set, and form a true height image set.
[0085] Step 104: Based on the first real remote sensing training set and the preset multi-scale dual encoder network framework, the geometric spatial features of the remote sensing images are learned through self-supervised training to generate a pre-trained model.
[0086] Step 105: Based on the urban remote sensing synthetic dataset, supervise the training of the pre-trained model to form an optimized model.
[0087] Step 106: Based on the second real remote sensing training set and the real height image set, perform transfer supervised learning on the optimized model to form an urban feature height reconstruction model.
[0088] Step 107: Acquire urban remote sensing images in real time, and use the urban feature height reconstruction model to reconstruct the height of the urban remote sensing images to generate real-time height images.
[0089] In this embodiment of the invention, the urban remote sensing synthetic dataset is generated by modeling software and is a data set used to simulate urban spatial scenes, containing typical urban features (such as buildings, roads, green spaces, and water bodies) and related attribute information. Because synthetic datasets are easy to acquire and rich in image information, using them for model training can solve the problem of insufficient reference data; moreover, synthetic datasets are of high quality, with no noise or outliers in the images, which helps ensure training accuracy. Therefore, constructing an urban remote sensing synthetic dataset can compensate for the limitations of real remote sensing images in terms of acquisition cost, annotation quality, and scene diversity.
[0090] In this embodiment of the invention, training the urban feature height reconstruction model not only utilizes a synthetic urban remote sensing dataset but also requires acquiring real remote sensing images. These real remote sensing images are classified into a first real remote sensing training set and a second real remote sensing training set. The first real remote sensing training set is used for self-supervised training, effectively improving the model's feature representation ability and generalization ability in downstream tasks, thus solving the problem of difficulty in transferring supervised learning across different sensor images. The second real remote sensing training set is used for fine-tuning the model to ensure its height reconstruction capability in real-world scenarios. Therefore, it is necessary to acquire the actual height images corresponding to each real remote sensing image in the second real remote sensing training set.
[0091] In this embodiment of the invention, the datasets required for training the urban feature height reconstruction model include a synthetic urban remote sensing dataset, a first real remote sensing training set, and a second real remote sensing training set. Specifically, based on a pre-defined multi-scale dual encoder network framework, self-supervised training is first performed using the first real remote sensing training set to learn the general features of remote sensing images, forming a pre-trained model. Then, based on the pre-trained model, supervised training is performed using the synthetic urban remote sensing dataset, enabling the model to achieve height reconstruction functionality, forming an optimized model. Finally, based on the second real remote sensing training set and its corresponding set of real height images, transfer-supervised learning is performed on the optimized model to ensure its height reconstruction capability in real-world scenarios, forming the urban feature height reconstruction model. The urban feature height reconstruction model formed through model training can reconstruct the height of real-time acquired urban remote sensing images, thereby generating real-time height images.
[0092] As a preferred embodiment, a multi-scale dual encoder network framework is pre-defined, including: a preprocessing module, a first feature extraction encoder, a second feature extraction encoder, a feature fusion unit, and a decoder;
[0093] The preprocessing module is used to perform spectral band analysis on remote sensing images and generate band guiding indices;
[0094] The first feature extraction encoder is used to extract features from the remote sensing image using a Comba module and a first KAN multilayer perceptron neural network to obtain a first feature;
[0095] The second feature extraction encoder is used to extract features from the remote sensing image using a feature extraction convolution module and a second KAN multilayer perceptron to obtain a second feature;
[0096] The feature fusion unit is used to fuse the first feature and the second feature to generate a fused feature;
[0097] The decoder is used to restore the fused features into a height image based on the band guidance index.
[0098] In this embodiment of the invention, the urban feature height reconstruction model is trained based on a preset multi-scale dual encoder network framework. The preset multi-scale dual encoder network framework includes a preprocessing module, a first feature extraction encoder, a second feature extraction encoder, a feature fusion unit, and a decoder. Specifically, the preprocessing module performs spectral band analysis on the input remote sensing image to generate a band guiding index. The first feature extraction encoder extracts features from the input remote sensing image to obtain a first feature. The second feature extraction encoder extracts features from the input remote sensing image to obtain a second feature. The feature fusion unit fuses the first and second features to generate a fused feature. The decoder reconstructs the height image from the fused feature based on the band guiding index.
[0099] As a preferred embodiment, spectral band analysis is performed on the remote sensing image to generate a band guiding index, including:
[0100] Determine the reflectance values of the red band, blue band, and near-infrared band of the remote sensing image respectively;
[0101] The normalized vegetation index is calculated based on the red light band reflectance value and the near-infrared band reflectance value.
[0102] The enhanced vegetation index is calculated based on the red light band reflectance value, the blue light band reflectance value, and the near-infrared band reflectance value.
[0103] The corrected soil-regulated vegetation index is calculated based on the red light band reflectance value and the near-infrared band reflectance value.
[0104] The OTU method is used to reassign values to the pixels of the remote sensing image to form an adjusted remote sensing image;
[0105] Based on the adjusted remote sensing image, the normalized vegetation index, the enhanced vegetation index, and the modified soil-regulated vegetation index are adjusted, and the adjusted normalized vegetation index, enhanced vegetation index, and modified soil-regulated vegetation index are stitched together in the channel dimension to generate a band guiding index.
[0106] Existing methods for urban feature height reconstruction rely on remote sensing imagery for height estimation, often neglecting the intrinsic relationship between feature categories and height, and failing to fully utilize the inherent feature category information within the data itself. Therefore, band-guided indices (SGIs) can be introduced during model training to aid in height reconstruction. These SGIs include: Normalized Difference Vegetation Index (NDVI), Enhanced Vegetation Index (EVI), and Modified Soil-Adjusted Vegetation Index (MSAVI).
[0107] In this embodiment of the invention, the band guiding index is generated by performing spectral band analysis on remote sensing images. See also Figure 2 This is a flowchart illustrating an embodiment of the band guidance index calculation method provided by the present invention. Specifically, firstly, the Normalized Difference Vegetation Index (NDVI), Enhanced Vegetation Index (EVI), and Modified Soil-Regulated Vegetation Index (MSAVI) are calculated using the red band reflectance value (R), blue band reflectance value (B), and near-infrared band reflectance value (NIR) of a remote sensing image, respectively. Then, the calculated NDVI, EVI, and MSAVI are automatically binarized using the OSTU method. The OSTU method finds an optimal threshold by maximizing the inter-class variance. Based on this threshold, the pixels of the image are classified into foreground and background, with foreground pixels assigned a value of 1 and background pixels assigned a value of 0, forming an adjusted remote sensing image. Based on the adjusted remote sensing image, the NDVI, EVI, and MSAVI are redefined, and these three indices are concatenated along the channel dimension to obtain the band guidance index (SGI) composed of three channels.
[0108] In this embodiment of the invention, the calculation formulas for the Normalized Difference Vegetation Index (NDVI), the Enhanced Vegetation Index (EVI), and the Modified Soil-Adjusted Vegetation Index (MSAVI) are as follows:
[0109]
[0110] In the formula, NDVI is the normalized vegetation index; EVI is the enhanced vegetation index; MSAVI is the modified soil-regulated vegetation index; NIR is the near-infrared reflectance value; R is the red light reflectance value; and B is the blue light reflectance value.
[0111] The purpose of the Band Guiding Index (SGI) is to fully utilize the multispectral characteristics of remote sensing imagery, extracting usable land cover category information (such as vegetation, impervious surfaces, etc.) from the remote sensing data itself, and guiding a deep learning network to reconstruct the height, thereby solving the problem of spectral complexity of land cover in remote sensing imagery. Therefore, combining the Band Guiding Index (SGI) with model training takes into account the inherent relationship between land cover category and height, and can fully utilize the inherent land cover category information of the data itself, thereby improving the accuracy of the model.
[0112] Current CNN architectures focus on extracting local features and are weak at capturing global contextual information; ViT relies on a global self-attention mechanism and lacks the ability to apply local prior knowledge. When existing urban feature height reconstruction models integrate CNN and ViT, the resulting architecture fails to combine the advantages of both, leading to less than ideal performance in real-world remote sensing scenarios.
[0113] Therefore, the training model of this invention adopts a multi-scale dual-encoder network framework, which is a deep nonlinear network with multiple scales and two branches. This can specifically solve the problems that existing deep learning networks have insufficient learning and representation capabilities when facing remote sensing images, and it is difficult to effectively integrate the advantages of CNN's local receptive field and ViT's global modeling.
[0114] In this embodiment of the invention, the dual encoders in the multi-scale dual encoder network framework include a first feature extraction encoder and a second feature extraction encoder. The architecture of the first feature extraction encoder is Comba-KAN, and the architecture of the second feature extraction encoder is Res-KAN.
[0115] As a preferred embodiment, the remote sensing image is extracted using a Comba module and a first KAN multilayer perceptron neural network to obtain a first feature, including:
[0116] The Comba module is used to perform feature transformation on the remote sensing image, and calculate the key vector, value vector and query vector respectively;
[0117] Calculate the model memory state based on the key vector and the value vector;
[0118] Based on the key vector, the query vector, and the model memory state, the first intermediate feature output by the Comba module is derived;
[0119] The first intermediate feature is obtained by nonlinear transformation using a first KAN multilayer perceptron neural network.
[0120] In an embodiment of the present invention, see Figure 3 This is a schematic diagram of an embodiment of the first feature extraction encoder provided by the present invention. The first feature extraction encoder consists of a Comba module and a first KAN multilayer perceptron neural network. Both the Comba module and the first KAN multilayer perceptron neural network adopt nonlinear network structures, resulting in a higher degree of nonlinearity in the constructed first feature extraction encoder, thus possessing a deeper learning space and a more powerful expressive ability. In the Comba-KAN branch of the first feature extraction encoder, the model progressively extracts features from the input remote sensing image (Input Feature) at different scales based on a four-layer Comba-KAN Block structure. The feature scales can be set sequentially to 112×112×96, 56×56×192, 28×28×384, and 14×14×768, and outputs the extracted features (Output Feature). The extracted features can be labeled as... Each Comba-KAN Block consists of two Comba-KAN structures connected in series. The Comba-KAN structure includes a Comba module, a normalization module BN, and a first KAN multilayer perceptron neural network.
[0121] The Comba module employs a bilinear recurrent neural network architecture with Comba closed-loop control, introducing a product term between the state S and the key, making the network an overall nonlinear structure. Furthermore, key vector interactions are incorporated into the state updates, thus forming a bilinear system. For the input remote sensing image, the Comba module first performs feature transformation to calculate the key vector k. t Value vector v t and query vector q t Then, state transitions and output corrections are performed.
[0122] Wherein, the key vector k t Value vector v t and query vector q t Calculate using the following formula:
[0123] q t =SiLU(W q ·x)
[0124] k t =SiLU(W k ·x)
[0125] v t =SiLU(Wv ·x)
[0126] In the formula, q t For query vector; k t v is the key vector; t W is a value vector; q To query the weight matrix; W k W is the key weight matrix; v is the weight matrix; x is the input remote sensing image.
[0127] After calculating the key vector k t Value vector v t and query vector q t Then, based on the key vector k t Sum vector v t The model memory state S is calculated using the following formula. t :
[0128]
[0129] In the formula, S t S represents the model's memory state at time t, recording the model's memory and feedback of past inputs at the current time; t-1 The model memory state at time t-1; α t , β t d represents a learnable scalar; R represents the dimension.
[0130] Based on key vector k t Query vector q t and the current model memory state S t The first intermediate feature f output by the Comba module is calculated using the following formula:
[0131] f = O t =S t (q t -dk t ) ∈R dv
[0132] In the formula, f is the first intermediate feature; O t Let f be the output of the model at time t, that is, the output feature of the input remote sensing image x after passing through the Comba module.
[0133] The output feature f of the Comba module is input into the first KAN multilayer perceptron for nonlinear transformation. It is further learned through the learnable activation function of the first KAN multilayer perceptron to derive the first feature. This process can be described as follows:
[0134] F C=KAN(f)
[0135] In the formula, F C The first feature is KAN(·), which is the nonlinear transformation function performed by the first KAN multilayer perceptron neural network.
[0136] As a preferred embodiment, a feature extraction convolutional module and a second KAN multilayer perceptron are used to extract features from the remote sensing image to obtain a second feature, including:
[0137] Combine KAN convolution and standard convolution into a feature extraction convolution module;
[0138] The feature extraction convolution module is used to extract features from the remote sensing image to obtain a second intermediate feature.
[0139] The second intermediate feature is added to the remote sensing image by residual to obtain the enhanced feature;
[0140] The enhanced features are nonlinearly transformed using a second KAN multilayer perceptron to obtain the second feature.
[0141] In an embodiment of the present invention, see Figure 4 This is a schematic diagram of an embodiment of the second feature extraction encoder provided by the present invention. The second feature extraction encoder consists of a feature extraction convolutional module and a second KAN multilayer perceptual neural network. In the Res-KAN branch of the second feature extraction encoder, the model extracts features from the input remote sensing image at different scales based on a four-layer Res-KAN block structure. The feature scales can be set sequentially to 112×112×96, 56×56×192, 28×28×384, and 14×14×768, and outputs the extracted features. The extracted features are labeled as follows. Each Res-KAN Block consists of two Res-KAN structures connected in series. The Res-KAN structure includes a KAN convolution (KanConv), a standard convolution (Conv), and a second KAN multilayer perceptron neural network.
[0142] The second feature extraction encoder employs a hybrid KAN convolution (KanConv) and standard convolution approach for feature extraction. It retains the linear computation characteristic of standard convolution while incorporating the nonlinear structural advantages of KAN, preserving the residual structure to ensure the network's learning ability. Specifically, for the input remote sensing image x, the second feature extraction encoder first extracts features using a 3×3 hybrid KAN convolution (KanConv) layer and a 3×3 standard convolution (Conv) layer, obtaining the second intermediate feature. Then, the second intermediate feature is added to the residual of the input remote sensing image, and the resulting enhanced feature is input into the second KAN multilayer perceptron neural network for nonlinear fitting to obtain the second feature.
[0143] As a preferred embodiment, the first feature and the second feature are fused to generate a fused feature, including:
[0144] For the first and second features at the same layer, the following steps are used to fuse features and generate fused features:
[0145] The first feature and the second feature are selected by using a gated loop unit to obtain the selected first feature and the selected second feature.
[0146] Obtain the previous layer fused features output by the feature fusion builder;
[0147] The selected first feature, the selected second feature, and the previous layer fusion feature are added and fused together to generate a fusion feature.
[0148] In an embodiment of the present invention, see Figure 5 This is a schematic diagram of an embodiment of the feature fusion processor provided by the present invention. The feature fusion processor introduces a gated recurrent unit (GRU) to assist in feature fusion. Feature 1 is the first feature output by the first feature extraction encoder. Feature 2 is the second feature output by the second feature extraction encoder. Feature 3 is the previous layer fused feature output by the feature fusion unit. (When in the first layer of the network, there are no fusion features from the previous layer). The Fusion Feature is the final fusion feature obtained.
[0149] To match the scale of Feature 1 and Feature 2, the previous layer fused features The feature fusion unit requires scale adjustment via convolutional layers, and the adjusted feature labels are as follows: And the first feature Second feature The information needs to be selected by a gated recurrent unit (GRU) to obtain the first feature after selection. and the second feature after selection The process of information selection can be represented as:
[0150]
[0151] The information selection process is as follows: The features of the input gated recurrent unit (GRU) are calculated with the learnable weight matrix and bias matrix. The calculation results are adjusted by the sigmoid activation function to obtain the update gate and reset gate results. Then, the input features, reset gate results, weight matrix, and bias matrix are calculated, and the output results are adjusted by the hyperbolic tangent activation function tanh to obtain the candidate hidden state results. Finally, the update gate results and candidate hidden states are used to calculate the selected features, thereby filtering the original information.
[0152] After the gated recurrent unit (GRU) completes its processing... and The resulting fusion feature is labeled as follows: for example, and Fusion And so on, until finally... The role of the Gated Recurrent Unit (GRU) is to control the feature information from the input, alleviate the vanishing gradient problem, and improve the performance and stability of the model. This invention introduces a feature fusion processor based on the GRU that can dynamically adjust the feature weights during the fusion process. By controlling the information flow, it avoids the vanishing gradient problem and achieves the best feature fusion effect.
[0153] As a preferred embodiment, restoring the fused features into a height image based on the band guiding index includes:
[0154] Starting with the deepest fusion feature, feature restoration processing is performed on the fusion features of each layer in turn to obtain the height image;
[0155] Specifically, the fused features undergo feature restoration processing, which includes:
[0156] The fusion features of the current layer are processed layer by layer through a multi-layer ViT and upsampling structure to obtain several layers of corresponding decoding features; wherein, the layer corresponding to each decoding feature is higher than the layer of the current layer;
[0157] Based on the band guidance index, the weights of the fusion features corresponding to each decoding feature level are adjusted to form the filtered fusion features;
[0158] Each decoded feature is concatenated with the corresponding level of filtered and fused features along the channel dimension to restore the decoded features into a single-channel image, thus obtaining a height image.
[0159] In this embodiment of the invention, the number of decoder layers corresponds to the number of generated fused features. For example, if four fused features are generated, the decoder is a four-layer ViT decoder. Each ViT module upsamples the features after decoding them, gradually restoring the image scale. The bottom-level ViT receives the fused features from the current layer. Decoding features are obtained by upsampling layer by layer. Meanwhile, the encoder's partial skip connection features After band-guided index processing, and the corresponding layer's decoding features Channel-dimensional concatenation yields feature scales that can be set sequentially to 28×28×768, 56×56×384, and 112×112×192. The feature images are labeled as follows. And so on, features After the final ViT decoding and upsampling, a height image (nDSM) with a scale of 224×224×1 is obtained.
[0160] In this embodiment of the invention, the band steering index is used to enhance regional features during network hop connections, thereby increasing effective information while suppressing noise. The band steering index is used to adjust the weights of the fusion features corresponding to each decoded feature at different levels, forming filtered fusion features. Specifically, the band steering index is used to weight and integrate feature region information through a self-attention module, and then the global values are adjusted using the Softmax function to activate them as feature filtering weights. The hop connection features output by the encoder are multiplied channel-by-channel with the feature filtering weights to achieve the regional feature enhancement process. During this process, the features… Change to filtered features This can be expressed as the following formula:
[0161]
[0162] Weight=Softmax(Attention(SGI))
[0163]
[0164] In the formula, d k Q represents the dimension of key information; K represents query information; V represents key information; and Q represents value information. Q, K, and V are all calculated by the band guidance index through the attention module. Indicates skip connection characteristics; The text represents the filtered features; Weight represents the weight; and ⊙ indicates that the feature is multiplied by the Weight channel by channel. Before calculating the weights, a band-guided index is used to perform scale matching correction for each layer of features, ensuring the consistency of the weight vector scale.
[0165] As a preferred embodiment, a synthetic urban remote sensing dataset is constructed, including:
[0166] Several urban scenes were modeled in 3D using modeling software; each urban scene has different urban characteristics.
[0167] The graphics engine is used to adjust the light source, shooting angle, and camera wavelength to generate several shooting parameters.
[0168] Based on the aforementioned city scenes and the aforementioned shooting parameters, several remote sensing composite data are generated; wherein, the remote sensing composite data includes composite remote sensing images and corresponding composite height images;
[0169] By integrating several of the aforementioned remote sensing composite data, a city remote sensing composite dataset is generated.
[0170] In this embodiment of the invention, the urban remote sensing composite dataset can be generated using modeling software. Specifically, firstly, 3D models of various urban scenes are created using modeling software such as Blender to simulate the distribution of real ground objects in the city, as the urban characteristics of each scene are different; then, by adjusting the light source, shooting angle, and camera band through a graphics engine, multi-band composite remote sensing images and corresponding composite height images are generated to simulate the way remote sensing satellites photograph the Earth, thereby constructing the urban remote sensing composite dataset.
[0171] As a preferred embodiment, the true height images of each real remote sensing image in the second real remote sensing training set are determined respectively, and a true height image set is formed, including:
[0172] For two real remote sensing images taken from different perspectives in the same area, obtain the satellite parameters corresponding to each real remote sensing image;
[0173] Stereo matching is performed on each pixel of the two real remote sensing images to generate a disparity map;
[0174] Based on the satellite parameters of the two real remote sensing images and the disparity map, the reference plane height of each pixel in each real remote sensing image is determined respectively;
[0175] For each real remote sensing image, a true height image is generated by arranging the pixels according to their positions on the real remote sensing image, based on the reference plane height of each pixel.
[0176] In this embodiment of the invention, the real remote sensing images include, but are not limited to, Gaofen series satellite imagery, Google satellite imagery, ESRI imagery, etc., which can be acquired through various satellite sensors. The spatial resolution of the collected images is between 1m and 2.5m to ensure the effectiveness of feature learning and height reconstruction; the image bands are red, green, blue, and near-infrared. The image coverage includes the styles and types of urban buildings in multiple regions, with the vast majority of images showing densely built-up areas, and also including various land cover categories such as vegetation, water surfaces, and roads. The images have undergone atmospheric and radiometric correction and have no significant cloud cover, making ground objects clearly visible. Therefore, the real remote sensing images acquired by this invention focus on multimodal, high-resolution, and feature-rich characteristics, which can effectively improve the generalization ability of deep learning networks.
[0177] In this embodiment of the invention, the acquired real remote sensing images are classified into a first real remote sensing training set and a second real remote sensing training set. The second real remote sensing training set is used to fine-tune the model to ensure its height reconstruction capability in real-world scenes. Therefore, it is necessary to obtain the true height images corresponding to each real remote sensing image in the second real remote sensing training set. The true height images of the real remote sensing images can be obtained through the following process:
[0178] For two real remote sensing images taken from different perspectives in the same area, first determine the satellite parameters corresponding to each real remote sensing image; perform stereo matching on each pixel of the two real remote sensing images to generate a disparity map; based on the satellite parameters and disparity map of the two real remote sensing images, determine the reference plane height of each pixel on each real remote sensing image; for each real remote sensing image, based on the reference plane height of each pixel, arrange the pixels according to their positions on the real remote sensing image to generate a true height image.
[0179] In this embodiment of the invention, training the urban feature height reconstruction model includes three main steps: First, a pre-trained model is generated by self-supervised training of a preset multi-scale dual encoder network framework using a first real remote sensing training set; then, the pre-trained model is trained under supervision using an urban remote sensing synthetic dataset to form an optimized model; finally, the optimized model is trained under transfer supervision using a second real remote sensing training set and its corresponding real height image set to form an urban feature height reconstruction model.
[0180] Specifically, a pre-trained model is generated by self-supervising the pre-defined multi-scale dual encoder network framework using the first real remote sensing training set, as follows:
[0181] The real remote sensing images in the first training set are segmented into smaller patches. These patches are then randomly masked with a masking ratio of 25%, meaning 25% of the segmented patches are assigned a pixel value of 0. The image patches are then numbered and sequentially fed into the encoder structure of a pre-defined multi-scale dual encoder network framework. The decoder consists of a lightweight transformer module and a linear layer, and is solely used to reconstruct image information. Mean Squared Error (MSE) is used as the reconstruction loss function, and the original information is reconstructed by predicting the pixel values of each masked patch.
[0182] The pre-trained model was trained under supervised training using a synthetic urban remote sensing dataset to form an optimized model, specifically as follows:
[0183] First, the synthetic remote sensing images in the urban remote sensing composite dataset are sequentially numbered and input into the preprocessing module at a size of 224×224 pixels. The preprocessing module calculates the band guiding index for each synthetic remote sensing image. The band guiding index is then used in the supervised training of the pre-trained model and automatically applied to the feature selection training process. In supervised training, the results of each training round are compared with real data to calculate the loss and backpropagation is performed to optimize and update the network parameters, obtaining better training results. The loss function in supervised training mainly includes global height loss, local building loss, local vegetation loss, and local surface loss. The global height loss uses the MSE loss function to calculate the loss over a global scale. The local building loss function is based on the band guiding index. A second MSE calculation is performed on the regions in the image that are divided according to the optimal threshold and reassigned a value of 1 to obtain the local building loss, thereby achieving local area reinforcement learning. The implementation methods of the local vegetation loss function and the local surface loss function are consistent with the local building loss function, and the data are based on NDVI, EVI, and MSAVI, respectively. The complete loss function calculation process can be represented as follows:
[0184] Loss = 0.5 × L Global +α×L B +(β1×L NDVI +β2×L EVI )+γ×L MSAVI
[0185] In the formula, Loss is the total loss; L Global For global height loss; L B For localized building damage; L NDVI Normalized vegetation loss; L EVI For enhanced vegetation loss; L MSAVIThe local surface loss is represented by α, β1, β2, and γ, which are the weights of the local loss, respectively. The weighted sum of normalized vegetation loss and enhanced vegetation loss represents the local vegetation loss.
[0186] By setting hyperparameters such as the number of training iterations, learning rate, batch size, and optimizer, multiple iterations of training can be performed on a synthetic urban remote sensing dataset. In each iteration, the gradient descent algorithm is used to reduce the model loss value, while simultaneously optimizing and updating the model parameters, ultimately yielding an optimized model.
[0187] Using a second set of real remote sensing training data and its corresponding set of real height images, the optimized model is subjected to transfer-supervised learning to form an urban feature height reconstruction model, specifically:
[0188] Based on the optimized model, lightweight supervised learning is performed using a second real remote sensing training set and its corresponding real height image set. The supervised learning process is consistent with the process of supervising the training of the pre-trained model to form the optimized model using the urban remote sensing synthetic dataset, thereby transferring the model's height reconstruction capability to the real remote sensing dataset and finally forming an urban feature height reconstruction model.
[0189] Implementing the above embodiments has the following effects:
[0190] This invention provides a method for urban feature height reconstruction based on remote sensing synthetic data. The method involves constructing an urban remote sensing synthetic dataset; acquiring real remote sensing images using satellite sensors to form a first real remote sensing training set and a second real remote sensing training set; determining the true height images of each real remote sensing image in the second real remote sensing training set to form a true height image set; using the first real remote sensing training set and a pre-set multi-scale dual encoder network framework, self-supervised training is performed to learn the geometric spatial features of the remote sensing images, generating a pre-trained model; supervised training is then performed on the pre-trained model using the urban remote sensing synthetic dataset to form an optimized model; transfer-supervised learning is then performed on the optimized model using the second real remote sensing training set and the true height image set to form an urban feature height reconstruction model; and urban remote sensing images are acquired in real time, and the urban feature height reconstruction model is used to reconstruct the height of the urban remote sensing images, generating real-time height images. This invention utilizes self-supervised learning based on the first real remote sensing training set, which effectively improves the model's feature representation ability and generalization ability in downstream tasks, thereby solving the problem of difficulty in transferring supervised learning across different sensor images. This invention uses urban remote sensing synthetic data rich in image information as model training data, which can effectively solve the problem of insufficient reference data. Furthermore, it utilizes a second real remote sensing training set to perform transfer learning on the model, ensuring the model's height reconstruction capability in real-world scenarios. Therefore, the urban feature height reconstruction model trained by this invention can effectively improve the accuracy of urban feature height reconstruction.
[0191] like Figure 6 As shown, based on the above method embodiments, corresponding apparatus embodiments are provided;
[0192] An embodiment of the present invention provides an urban feature height reconstruction device based on remote sensing synthetic data, comprising: a synthetic data acquisition module, a real data acquisition module, a real height image generation module, a model pre-training module, a model supervised training module, a model transfer learning module, and a model application module;
[0193] The synthetic data acquisition module is used to construct a city remote sensing synthetic dataset; wherein, the city remote sensing synthetic dataset includes synthetic remote sensing images of several city scenes and corresponding synthetic height images;
[0194] The real data acquisition module is used to collect real remote sensing images based on satellite sensors, and form a first real remote sensing training set and a second real remote sensing training set respectively.
[0195] The true height image generation module is used to determine the true height images of each true remote sensing image in the second true remote sensing training set, and form a true height image set;
[0196] The model pre-training module is used to learn the geometric spatial features of remote sensing images through self-supervised training based on the first real remote sensing training set and a preset multi-scale dual encoder network framework, and generate a pre-trained model.
[0197] The model supervised training module is used to supervise the training of the pre-trained model based on the urban remote sensing synthetic dataset to form an optimized model;
[0198] The model transfer learning module is used to perform supervised transfer learning on the optimized model based on the second real remote sensing training set and the real height image set to form an urban feature height reconstruction model.
[0199] The model application module is used to acquire urban remote sensing images in real time, and to reconstruct the height of the urban remote sensing images using the urban feature height reconstruction model to generate real-time height images.
[0200] It is understood that the above-described device embodiments correspond to the method embodiments of the present invention, and can realize the urban feature height reconstruction method based on remote sensing synthetic data provided by any of the above-described method embodiments of the present invention.
[0201] It should be noted that the device embodiments described above are merely illustrative, and some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can specifically be implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.
[0202] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.
Claims
1. A method for reconstructing the height of urban ground objects based on remote sensing synthetic data, characterized in that, include: Construct a synthetic remote sensing dataset for cities; wherein, the synthetic remote sensing dataset for cities includes synthetic remote sensing images of several urban scenes and corresponding synthetic height images; Based on real remote sensing images collected by satellite sensors, a first real remote sensing training set and a second real remote sensing training set were formed respectively. The true height images of each real remote sensing image in the second real remote sensing training set are determined respectively, and a true height image set is formed; Based on the first real remote sensing training set and the preset multi-scale dual encoder network framework, the geometric spatial features of remote sensing images are learned through self-supervised training to generate a pre-trained model. Based on the aforementioned urban remote sensing synthetic dataset, the pre-trained model is subjected to supervised training to form an optimized model. Based on the second real remote sensing training set and the real height image set, the optimized model is subjected to transfer supervised learning to form an urban feature height reconstruction model. Real-time acquisition of urban remote sensing images, and the use of the urban feature height reconstruction model to reconstruct the height of the urban remote sensing images to generate real-time height images. 2.The method of claim 1, wherein, The preset multi-scale dual encoder network framework includes: a preprocessing module, a first feature extraction encoder, a second feature extraction encoder, a feature fusion unit, and a decoder; The preprocessing module is used to perform spectral band analysis on remote sensing images and generate band guiding indices; The first feature extraction encoder is used to extract features from the remote sensing image using a Comba module and a first KAN multilayer perceptron neural network to obtain a first feature; The second feature extraction encoder is used to extract features from the remote sensing image using a feature extraction convolution module and a second KAN multilayer perceptron to obtain a second feature; The feature fusion unit is used to fuse the first feature and the second feature to generate a fused feature; The decoder is used to restore the fused features into a height image based on the band guidance index. 3.The method of claim 2, wherein, The step of performing spectral band analysis on remote sensing images to generate band guiding indices includes: Determine the reflectance values of the red band, blue band, and near-infrared band of the remote sensing image respectively; The normalized vegetation index is calculated based on the red light band reflectance value and the near-infrared band reflectance value. The enhanced vegetation index is calculated based on the red light band reflectance value, the blue light band reflectance value, and the near-infrared band reflectance value. The corrected soil-regulated vegetation index is calculated based on the red light band reflectance value and the near-infrared band reflectance value. The OTU method is used to reassign values to the pixels of the remote sensing image to form an adjusted remote sensing image; Based on the adjusted remote sensing image, the normalized vegetation index, the enhanced vegetation index, and the modified soil-regulated vegetation index are adjusted, and the adjusted normalized vegetation index, enhanced vegetation index, and modified soil-regulated vegetation index are stitched together in the channel dimension to generate a band guiding index.
4. The method for urban feature height reconstruction based on remote sensing synthetic data according to claim 2, characterized in that, The first feature is obtained by using the Comba module and the first KAN multilayer perceptron neural network to extract features from the remote sensing image, including: The Comba module is used to perform feature transformation on the remote sensing image, and calculate the key vector, value vector and query vector respectively; Calculate the model memory state based on the key vector and the value vector; Based on the key vector, the query vector, and the model memory state, the first intermediate feature output by the Comba module is derived; The first intermediate feature is obtained by nonlinear transformation using a first KAN multilayer perceptron neural network.
5. The method for urban feature height reconstruction based on remote sensing synthetic data according to claim 2, characterized in that, The feature extraction convolutional module and the second KAN multilayer perceptron neural network are used to extract features from the remote sensing image to obtain the second feature, including: Combine KAN convolution and standard convolution into a feature extraction convolution module; The feature extraction convolution module is used to extract features from the remote sensing image to obtain a second intermediate feature. The second intermediate feature is added to the remote sensing image by residual to obtain the enhanced feature; The enhanced features are nonlinearly transformed using a second KAN multilayer perceptron to obtain the second feature.
6. The method for urban feature height reconstruction based on remote sensing synthetic data according to claim 2, characterized in that, The process of fusing the first feature and the second feature to generate a fused feature includes: For the first and second features at the same layer, the following steps are used to fuse features and generate fused features: The first feature and the second feature are selected by using a gated loop unit to obtain the selected first feature and the selected second feature. Obtain the previous layer fused features output by the feature fusion builder; The selected first feature, the selected second feature, and the previous layer fusion feature are added and fused together to generate a fusion feature.
7. The method for urban feature height reconstruction based on remote sensing synthetic data according to claim 2, characterized in that, The step of restoring the fused features into a height image based on the band guidance index includes: Starting with the deepest fusion feature, feature restoration processing is performed on the fusion features of each layer in turn to obtain the height image; Specifically, the fused features undergo feature restoration processing, which includes: The fusion features of the current layer are processed layer by layer through a multi-layer ViT and upsampling structure to obtain several layers of corresponding decoding features; wherein, the layer corresponding to each decoding feature is higher than the layer of the current layer; Based on the band guidance index, the weights of the fusion features corresponding to each decoding feature level are adjusted to form the filtered fusion features; Each decoded feature is concatenated with the corresponding level of filtered and fused features along the channel dimension to restore the decoded features into a single-channel image, thus obtaining a height image.
8. The method for urban feature height reconstruction based on remote sensing synthetic data according to claim 1, characterized in that, The construction of the urban remote sensing synthetic dataset includes: Several urban scenes were modeled in 3D using modeling software; each urban scene has different urban characteristics. The graphics engine is used to adjust the light source, shooting angle, and camera wavelength to generate several shooting parameters. Based on the aforementioned city scenes and the aforementioned shooting parameters, several remote sensing composite data are generated; wherein, the remote sensing composite data includes composite remote sensing images and corresponding composite height images; By integrating several of the aforementioned remote sensing composite data, a city remote sensing composite dataset is generated.
9. The method for urban feature height reconstruction based on remote sensing synthetic data according to claim 1, characterized in that, The step of determining the true height images of each real remote sensing image in the second real remote sensing training set and forming a true height image set includes: For two real remote sensing images taken from different perspectives in the same area, obtain the satellite parameters corresponding to each real remote sensing image; Stereo matching is performed on each pixel of the two real remote sensing images to generate a disparity map; Based on the satellite parameters of the two real remote sensing images and the disparity map, the reference plane height of each pixel in each real remote sensing image is determined respectively; For each real remote sensing image, a true height image is generated by arranging the pixels according to their positions on the real remote sensing image, based on the reference plane height of each pixel.
10. A device for reconstructing the height of urban features based on remote sensing synthetic data, characterized in that, include: The system includes a synthetic data acquisition module, a real data acquisition module, a real height image generation module, a model pre-training module, a model supervised training module, a model transfer learning module, and a model application module. The synthetic data acquisition module is used to construct a city remote sensing synthetic dataset; wherein, the city remote sensing synthetic dataset includes synthetic remote sensing images of several city scenes and corresponding synthetic height images; The real data acquisition module is used to collect real remote sensing images based on satellite sensors, and form a first real remote sensing training set and a second real remote sensing training set respectively. The true height image generation module is used to determine the true height images of each true remote sensing image in the second true remote sensing training set, and form a true height image set; The model pre-training module is used to learn the geometric spatial features of remote sensing images through self-supervised training based on the first real remote sensing training set and a preset multi-scale dual encoder network framework, and generate a pre-trained model. The model supervised training module is used to supervise the training of the pre-trained model based on the urban remote sensing synthetic dataset to form an optimized model; The model transfer learning module is used to perform supervised transfer learning on the optimized model based on the second real remote sensing training set and the real height image set to form an urban feature height reconstruction model. The model application module is used to acquire urban remote sensing images in real time, and to reconstruct the height of the urban remote sensing images using the urban feature height reconstruction model to generate real-time height images.