A city electromagnetic voxel parameter inversion method, system and electronic equipment fusing multi-source remote sensing and street view

Abstract

The application discloses a kind of fusion multi-source remote sensing and city electromagnetic voxel parameter inversion method, system and electronic equipment of street view, comprising the following steps: obtaining multi-source heterogeneous data and being unified to same geographic coordinate system, establish the association of same building instance in multi-source heterogeneous data, output after alignment multi-source heterogeneous data and extracted each building facade feature;Based on the multi-source heterogeneous data and building facade feature after alignment, the building surface material is classified by multi-modal semantic segmentation network, and the material category probability distribution is output;According to material category probability distribution, mapping to preset frequency-dependent electromagnetic parameter library, the target electromagnetic parameter and its uncertainty measure are obtained by inversion;The target electromagnetic parameter and its uncertainty measure are labeled to the outer surface attribute of the corresponding building three-dimensional electromagnetic voxel, and the application multi-source data complement each other, and the accuracy of city building material identification is greatly improved, and large-scale city automatic processing is supported.

Description

Technical Field

[0001] This invention belongs to the field of urban electromagnetic environment modeling technology, specifically involving a method, system, and electronic equipment for inverting urban electromagnetic element parameters by integrating multi-source remote sensing and street view. Background Technology

[0002] The accuracy of urban electromagnetic environment simulation (such as ray tracing and finite-difference time-domain methods) is highly dependent on the fidelity of the scene model. Among them, the material parameters of building surfaces directly affect the calculation accuracy of reflection coefficient, transmission coefficient and diffraction coefficient. Existing urban building scene data mainly come from three types: Open Street View (OSM), LiDAR point cloud and Building Information Model (BIM). OSM has defects such as material label coverage of less than 5%, limited geometric accuracy and lag in updates; LiDAR point cloud cannot directly identify material type and has a global coverage of less than 10%; BIM data is difficult to obtain and the coordinates are inconsistent, so it cannot be directly used for electromagnetic simulation. Existing remote sensing segmentation technology only distinguishes between buildings and non-buildings, does not subdivide the material level, and does not integrate multi-source complementary data such as synthetic aperture radar and street view.

[0003] In summary, the existing technology has the following drawbacks:

[0004] 1. The difficulty in obtaining information on urban building materials, and the lack of material information, is the biggest bottleneck to improving the accuracy of electromagnetic simulation;

[0005] 2. The segmentation results need to be manually mapped to electromagnetic parameters by looking up tables, lacking automated inversion function;

[0006] 3. The uncertainty of material identification is not considered, making it impossible to provide probabilistic input for downstream Monte Carlo simulation;

[0007] To this end, we propose a method, system, and electronic equipment for inverting urban electromagnetic element parameters by integrating multi-source remote sensing and street view. Summary of the Invention

[0008] The purpose of this invention is to provide a method, system, and electronic device for inverting urban electromagnetic element parameters by integrating multi-source remote sensing and street view, so as to solve the problems mentioned in the background art.

[0009] To achieve the above objectives, the present invention provides the following technical solution: a method for inverting urban electromagnetic voxel parameters by integrating multi-source remote sensing and street view, comprising the following steps:

[0010] Data acquisition and alignment steps: Acquire multi-source heterogeneous data and unify them into the same geographic coordinate system, establish the association relationship of the same building instance in the multi-source heterogeneous data, obtain the aligned multi-source heterogeneous data, extract building facade features, and output the aligned multi-source heterogeneous data and the extracted building facade features.

[0011] Facade fusion and segmentation steps: Based on the aligned multi-source heterogeneous data and building facade features, the building surface materials are classified through a multimodal semantic segmentation network, and the probability distribution of material categories is output.

[0012] Parameter inversion steps: Based on the probability distribution of material type, map to the preset frequency-dependent electromagnetic parameter library, and invert to obtain the target electromagnetic parameters and their uncertainty measure;

[0013] Voxel annotation steps: Annotate the target electromagnetic parameters and their uncertainty measures to the outer surface properties of the corresponding three-dimensional electromagnetic voxels of the building.

[0014] Preferably, in the data acquisition and alignment step, the multi-source heterogeneous data includes at least optical satellite imagery, synthetic aperture radar imagery, and near-ground street view imagery;

[0015] The data acquisition and alignment steps specifically include:

[0016] After fine correction using a rational polynomial coefficient model, the optical satellite images are registered to the universal transverse Mercator projection coordinate system.

[0017] Synthetic aperture radar images are registered to the universal transverse Mercator projection coordinate system by feature point matching between optical images and synthetic aperture radar images.

[0018] The camera pose is determined based on the GPS tags and motion recovery structure 3D reconstruction of the street view image, and the 2D street view pixels are back-projected to the 3D universal transverse Mercator projection coordinates.

[0019] Using the vector building outlines of the open street view map as the index, the same building instances in multi-source heterogeneous data are associated by judging the spatial overlap ratio, and each building is assigned a globally unique identifier.

[0020] Preferably, in the data acquisition and alignment step, building facade feature extraction is performed on the street view image, specifically including:

[0021] Motion recovery and structural pose restoration are performed on the street view image sequence collected along the street to determine the camera position and orientation of each street view image;

[0022] Project the pixel regions belonging to the same building facade in each street view image onto the local three-dimensional coordinate system of that building;

[0023] Depth values ​​are assigned to building facade pixels in street view images by multi-view stereo matching or deep learning depth estimation methods.

[0024] Multi-view facade images are stitched together to create a panoramic view of the building's facade, which is then incorporated as part of the building's facade features.

[0025] Preferably, the multimodal semantic segmentation network used in the facade fusion and segmentation steps specifically includes:

[0026] A satellite-overlooking encoder is used to extract visible light and near-infrared features of the building's overhead area from optical satellite imagery.

[0027] Synthetic Aperture Radar (SAR) encoders are used to directly process in-phase and orthogonal complex data from SAR images and extract scattering feature maps.

[0028] Street view facade encoder for extracting sequential features from a wraparound panoramic facade image;

[0029] The cross-modal feature fusion module is used to generate fused features by using the features output by the street view facade encoder as the main query and the features output by the satellite overhead encoder and synthetic aperture radar encoder as the key values ​​through a cross-modal attention mechanism.

[0030] The decoder is used to decode the fused features and output the probability distribution of each surface of the building belonging to each material category.

[0031] Preferably, the material categories output by the multimodal semantic segmentation network include the following fifteen categories: ordinary concrete, smooth concrete, red brick, stone, ordinary glass, metal-coated glass, glazed ceramic tile, metal plate, aluminum curtain wall, painted wall, wood structure, gypsum board, composite material, vegetation cover, and unknown mixed materials.

[0032] Furthermore, the multimodal semantic segmentation network simultaneously outputs the following three uncertainty measures for each building surface:

[0033] Classification entropy is used to represent the overall degree of uncertainty in classifying a surface as a material.

[0034] The difference between the probability of the most likely material and the probability of the second most likely material is used to represent the difficulty of distinguishing between the top two candidate materials;

[0035] Cross-modal consistency measure is used to represent the degree of mutual consistency between prediction results using only satellite imagery, only synthetic aperture radar imagery, and only street view imagery.

[0036] Preferably, in the parameter inversion step, the specific method for obtaining the target electromagnetic parameters and their uncertainty measure based on the probability distribution of material type is as follows:

[0037] Using the probability of each material category as a weight, the dielectric constant of each material category at the target frequency is weighted and summed to obtain the expected dielectric constant; the conductivity of each material category at the target frequency is weighted and summed to obtain the expected conductivity.

[0038] Using the probability of each material category as a weight, the variances of the desired dielectric constant and desired conductivity are calculated respectively, and the variances are used as a measure of the uncertainty of the target electromagnetic parameters.

[0039] When the frequency-dependent electromagnetic parameter library only provides parameter values ​​at discrete frequency points, cubic spline interpolation is performed on the known discrete points in the logarithmic frequency domain to obtain the dielectric constant and conductivity values ​​at any target frequency.

[0040] Preferably, in the parameter inversion step, after obtaining the desired dielectric constant and desired conductivity, a pre-calculation step for the reflection coefficient and transmission coefficient is also included:

[0041] Calculate the intrinsic impedance of the material based on the desired dielectric constant and desired conductivity;

[0042] Based on the intrinsic impedance and the Fresnel formula, calculate the vertical polarization reflection coefficient and the horizontal polarization reflection coefficient of electromagnetic waves on the voxel surface of the building.

[0043] The vertical polarization reflection coefficient, the horizontal polarization reflection coefficient, and the transmission coefficient derived from the reflection coefficient are stored as additional attribute fields for direct use by downstream ray tracing.

[0044] Preferably, the voxel annotation steps specifically include:

[0045] For each building voxel, determine whether each of its six faces is adjacent to an air voxel, and identify the face adjacent to the air voxel as the outer surface.

[0046] For voxel patches identified as outer surfaces, the corresponding building facade area output by the multimodal semantic segmentation network is matched according to the position coordinates and orientation of the patch in three-dimensional space.

[0047] The matched material category probability distribution, expected dielectric constant, expected conductivity, and uncertainty measure are all labeled into the material properties of the voxel surface.

[0048] The confidence score of the annotation result is calculated based on the information entropy of the probability distribution of the material category, and the confidence score is written into the voxel attribute.

[0049] In downstream electromagnetic simulation, two operating modes are supported: in deterministic mode, the desired dielectric constant and desired conductivity are used directly for a single simulation; in Monte Carlo mode, multiple samples are taken from the probability distribution of material categories, and a complete simulation is performed each time using the material parameters obtained from the sampling, and finally the statistical distribution of multiple simulation results is output.

[0050] A system for inverting urban electromagnetic voxel parameters by integrating multi-source remote sensing and street view data, applied to the method for inverting urban electromagnetic voxel parameters by integrating multi-source remote sensing and street view data, includes:

[0051] The data alignment module is used to acquire multi-source heterogeneous data and unify it into the same geographic coordinate system, establish the association relationship of the same building instance in the multi-source heterogeneous data, obtain the aligned multi-source heterogeneous data, extract building facade features, and output the aligned multi-source heterogeneous data and the extracted building facade features.

[0052] The segmentation processing module is used to classify building surface materials based on aligned multi-source heterogeneous data and building facade features through a multimodal semantic segmentation network, and output the probability distribution of material categories.

[0053] The parameter inversion module is used to map the probability distribution of material type to a preset frequency-dependent electromagnetic parameter library and invert the target electromagnetic parameters and their uncertainty measure.

[0054] The voxel annotation module is used to annotate the target electromagnetic parameters and their uncertainty measures to the outer surface properties of the corresponding three-dimensional electromagnetic voxels of the building.

[0055] An electronic device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement a method for inverting urban electromagnetic pixel parameters by fusing multi-source remote sensing and street view.

[0056] Compared with the prior art, the beneficial effects of the present invention are:

[0057] 1. This invention is the first to deeply fuse three complementary data sources: optical satellite imagery, synthetic aperture radar imagery, and near-ground street view imagery. Optical imagery provides color and texture information, synthetic aperture radar imagery is unaffected by lighting and fog and is sensitive to metal surfaces, and street view imagery provides fine details of facades. The three data sources complement each other, comprehensively improving the accuracy of material recognition. This invention achieves a recognition accuracy of 92% for glass curtain walls, 94% for concrete, 91% for red bricks, 88% for stone, and 93% for metal plates, with an overall recognition accuracy of 91.3%. In contrast, existing open street view maps have a material label coverage of less than 5%, and the segmentation accuracy of pure optical imagery is only about 60% to 70%. This invention significantly improves the accuracy of material recognition.

[0058] 2. This invention establishes a fine classification system for 15 categories of building surface materials. This classification system directly corresponds to the standard material parameter library of the International Telecommunication Union Radiocommunication Sector, which enables the material identification results to be seamlessly mapped to the physical parameters required for electromagnetic simulation, avoiding the problem of parameter value deviation caused by overly coarse material classification.

[0059] 3. This invention establishes an automated mapping pipeline from semantic segmentation results to frequency-dependent electromagnetic parameters: the material probability distribution output by the segmentation network is automatically calculated by weighted summation of the expected dielectric constant and expected conductivity, and then the parameter values ​​at any target frequency are supported by logarithmic frequency domain cubic spline interpolation. Finally, the reflection coefficient and transmission coefficient are automatically pre-calculated by Fresnel formula, and finally automatically labeled into the electromagnetic voxel attributes. The whole process does not require manual intervention, which greatly improves efficiency compared to manual labeling of each building.

[0060] 4. This invention uses the vector building outlines of open street view maps as the index reference, assigns a globally unique identifier to each building, and associates the same building instance in satellite imagery, synthetic aperture radar imagery and street view imagery through spatial overlap judgment, realizing automatic mapping from two-dimensional pixels to three-dimensional voxel patches. It judges whether each building voxel's six faces are exposed to air one by one, and only annotates the material parameters of the outer surface, avoiding the calculation of internal voxels. It is suitable for automated processing of cities of any size. Detailed Implementation

[0061] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0062] The urban electromagnetic voxel parameter inversion method integrating multi-source remote sensing and street view provided by this invention includes the following steps:

[0063] Data acquisition and alignment steps: Acquire multi-source heterogeneous data and unify it to the same geographic coordinate system. The multi-source heterogeneous data should include at least optical satellite imagery, synthetic aperture radar imagery, and near-ground street view images. Establish the association relationship of the same building instance in the multi-source heterogeneous data to obtain aligned multi-source heterogeneous data. Extract building facade features and output the aligned multi-source heterogeneous data and the extracted building facade features. Specific steps include:

[0064] After fine correction using a rational polynomial coefficient model, the optical satellite images are registered to the universal transverse Mercator projection coordinate system.

[0065] Synthetic aperture radar images are registered to the universal transverse Mercator projection coordinate system by feature point matching between optical images and synthetic aperture radar images.

[0066] The camera pose is determined based on the GPS tags and motion recovery structure 3D reconstruction of the street view image, and the 2D street view pixels are back-projected to the 3D universal transverse Mercator projection coordinates.

[0067] Using the vector building outlines of the open street view map as the index, the same building instances in multi-source heterogeneous data are associated by judging the spatial overlap ratio, and each building is assigned a globally unique identifier.

[0068] This invention first aligns all data sources to the same geographic coordinate system (Universal Transverse Mercator Projection, EPSG:326xx series). The alignment strategy is as follows: Satellite imagery typically contains rational polynomial coefficients or Universal Transverse Mercator Projection geocoding, serving as a reference; Synthetic Aperture Radar (SAR) imagery is converted to the Universal Transverse Mercator Projection coordinate system through SAR geometric correction (range-Doppler positioning) and registration with optical imagery (based on feature point matching); Street view images are determined by using GPS tags (or acquisition device trajectories) and motion-reconstructed structures (MRCs) to determine the camera pose (position and orientation) of each street view image, and the 2D street view pixels are back-projected to the 3D Universal Transverse Mercator Projection coordinate system; the vector building outlines of open street view maps are naturally in WGS84 coordinates, which are then converted to the Universal Transverse Mercator Projection; LiDAR point clouds typically come with Universal Transverse Mercator Projection coordinates or can be registered to the Universal Transverse Mercator Projection using control points.

[0069] Associating the same building from different data sources: using the vector building outline of the Open Street View map as the index reference, assigning a globally unique identifier to each building; and associating building pixel blocks in satellite imagery, building scattering areas in synthetic aperture radar imagery, building facades in street view images, and vector building outlines in the Open Street View map through spatial overlap judgment (intersection over union ratio greater than 0.5).

[0070] Extracting building facade features from street view images, specifically including:

[0071] Motion recovery and structural pose restoration are performed on the street view image sequence collected along the street to determine the camera position and orientation of each street view image;

[0072] Project the pixel regions belonging to the same building facade in each street view image onto the local three-dimensional coordinate system of that building;

[0073] Depth values ​​are assigned to building facade pixels in street view images by multi-view stereo matching or deep learning depth estimation methods.

[0074] Multi-view facade images are stitched together to form a panoramic view of the building's facade, and this panoramic view is used as part of the building's facade features for subsequent facade material segmentation.

[0075] Facade fusion and segmentation steps: Based on the aligned multi-source heterogeneous data and building facade features, the building surface materials are classified through a multimodal semantic segmentation network, and the probability distribution of material categories is output.

[0076] The multimodal semantic segmentation network used in the facade fusion and segmentation steps simultaneously processes satellite overhead views, synthetic aperture radar scattering features, and street view facade perspectives, outputting the material categories of each building surface, specifically including:

[0077] For each building, the multimodal Transformer segmentation network receives the following input:

[0078] A satellite-overlooking encoder is used to extract visible light and near-infrared features of the building's overhead area from optical satellite imagery.

[0079] Synthetic Aperture Radar (SAR) encoders are used to directly process in-phase and orthogonal complex data from SAR images and extract scattering feature maps. SAR encoders directly process complex data (in-phase / orthogonal channels), retain phase information, and have the ability to distinguish between metal surfaces (strong scattering) and dielectric surfaces (weak scattering).

[0080] The street view facade encoder is used to extract sequential features from the panoramic view of the surrounding facade. It extracts the visible light image block sequence of each facade of the building from the stitched panoramic view of the surrounding facade and encodes it into sequential features using a visual Transformer.

[0081] The cross-modal feature fusion module is used to generate fused features by using the features output by the street view facade encoder as the main query and the features output by the satellite overhead encoder and synthetic aperture radar encoder as the key values ​​through a cross-modal attention mechanism. After the three encoders output features respectively, they are processed by their respective transformers and then fused features are generated through a cross-modal attention mechanism.

[0082] The decoder is used to decode the fused features and output the probability distribution of each surface of the building belonging to each material category;

[0083] The multimodal semantic segmentation network outputs material categories including the following fifteen types: ordinary concrete, smooth concrete, red brick, stone, ordinary glass, metal-coated glass, glazed ceramic tiles, metal plates, aluminum curtain walls, painted walls, wood structures, gypsum board, composite materials, vegetation cover, and unknown mixed materials. For each building surface voxel patch, it outputs a probability vector. Where material represents the material category number, and P represents the probability value of the patch belonging to each material category, satisfying the condition that the sum of the probabilities of all categories equals 1. ;

[0084] Furthermore, the multimodal semantic segmentation network simultaneously outputs the following three uncertainty measures for each building surface:

[0085] Classification entropy, used to represent the overall uncertainty of the material classification of the surface, is calculated as shown in formula (1):

[0086] (1);

[0087] in, This represents the predicted probability of the i-th material type;

[0088] H represents classification entropy; a high entropy value indicates that the model is uncertain about the material of the patch.

[0089] The difference between the probability of the most likely material and the probability of the second most likely material is used to represent the difficulty of distinguishing between the top two candidate materials. The calculation formula is shown in formula (2):

[0090] ΔP = P1st - P2nd (2;

[0091] Where P1st represents the maximum class probability;

[0092] P2nd represents the probability of the second largest category;

[0093] ΔP represents the difference between the two; a small difference indicates that the top two candidate materials are difficult to distinguish.

[0094] Cross-modal consistency measure is used to represent the degree of mutual consistency between prediction results using only satellite imagery, only synthetic aperture radar imagery, and only street view imagery. Low consistency indicates that the material features of the patch are blurred.

[0095] The parameter inversion steps are as follows: Based on the probability distribution of the material type, the parameters are mapped to a preset frequency-dependent electromagnetic parameter library. The specific method for inverting the target electromagnetic parameters and their uncertainty measures is as follows:

[0096] Using the probability of each material category as a weight, the dielectric constant of each material category at the target frequency is weighted and summed to obtain the expected dielectric constant. The calculation formula is shown in formula (3):

[0097] (3);

[0098] in, Indicates the desired dielectric constant;

[0099] I indicates the material category index;

[0100] This represents the predicted probability of the i-th material type;

[0101] This represents the relative permittivity of the i-th type of material at frequency f;

[0102] The conductivity of each material type at the target frequency is weighted and summed with equal weights to obtain the desired conductivity. The calculation formula is shown in formula (4):

[0103] (4);

[0104] in, Indicates the desired conductivity;

[0105] This represents the electrical conductivity of the i-th type of material at frequency f;

[0106] Using the probability of each material category as a weight, the variances of the desired dielectric constant and desired conductivity are calculated respectively. The variance is used as a measure of the uncertainty of the target electromagnetic parameters. The calculation formula is shown in formula (5):

[0107] (5);

[0108] in, This represents the variance of the desired dielectric constant at frequency f;

[0109] When the frequency-dependent electromagnetic parameter library only provides parameter values ​​at discrete frequency points, cubic spline interpolation is performed on the known discrete points in the logarithmic frequency domain to obtain the dielectric constant and conductivity values ​​at any target frequency. The calculation formula is shown in formula (6):

[0110] (6);

[0111] in, Represents the natural logarithm;

[0112] This represents the cubic spline interpolation function;

[0113] This represents the k-th known discrete frequency point;

[0114] This represents the relative permittivity value at a discrete frequency point fk;

[0115] This interpolation ensures that the result is physically reasonable (dielectric constant greater than 1, conductivity greater than 0), and the conductivity is interpolated in the same way as the dielectric constant.

[0116] In the parameter inversion step, after obtaining the desired dielectric constant and desired conductivity, a pre-calculation step for the reflection coefficient and transmission coefficient is also included:

[0117] The intrinsic impedance of the material is calculated based on the desired dielectric constant and desired conductivity, as shown in formula (7):

[0118] (7);

[0119] in, Indicates the permeability of free space;

[0120] Represents the vacuum permittivity;

[0121] Represents the relative permittivity;

[0122] Represents the imaginary unit;

[0123] Indicates the intrinsic impedance of the dielectric;

[0124] Based on the intrinsic impedance and Fresnel's formula, the vertical polarization reflection coefficient and horizontal polarization reflection coefficient of electromagnetic waves on the voxel surface of the building are calculated respectively. The calculation formulas are shown in formulas (8) and (9) respectively:

[0125] (8);

[0126] (9);

[0127] in, Indicates the vertical polarization reflection coefficient;

[0128] Indicates the horizontal polarization reflection coefficient;

[0129] The intrinsic impedance of the medium in which the incident wave is located.

[0130] This represents the intrinsic impedance of the medium in which the transmitted wave is located;

[0131] Indicates the angle of incidence;

[0132] Indicates the transmission angle;

[0133] The vertical polarization reflection coefficient, the horizontal polarization reflection coefficient, and the transmission coefficient derived from the reflection coefficient are stored as additional attribute fields for direct use by downstream ray tracing.

[0134] The formula for calculating the transmission coefficient is shown in formula (10):

[0135] (10);

[0136] in, Indicates the transmission coefficient;

[0137] Indicates the reflection coefficient;

[0138] Voxel annotation steps: The target electromagnetic parameters and their uncertainty measures are annotated to the outer surface properties of the corresponding 3D electromagnetic voxels of the building. Specific steps include:

[0139] Voxel patch mapping: For each of the six faces of a building voxel, determine whether the face is adjacent to an air voxel (i.e., whether it is located on the building's outer surface). Faces adjacent to air voxels are identified as outer surfaces. For voxel patches identified as outer surfaces, match them to the corresponding building facade area output by the multimodal semantic segmentation network based on the patch's position coordinates and orientation in 3D space. The matched material category probability distribution, expected dielectric constant, expected conductivity, and uncertainty metric are also labeled into the material properties of the voxel patch.

[0140] The confidence score of the annotation result is calculated based on the information entropy of the probability distribution of the material category, and the confidence score is written into the voxel attribute.

[0141] The voxel type is uniformly set to "Building Entity" with a fill rate of 1.0; the material identifier is the material category with the highest probability; the confidence score is calculated based on the classification entropy; and the source identifier is "Neural Network".

[0142] Two operating modes are supported in downstream electromagnetic simulation:

[0143] In deterministic mode, the desired dielectric constant and desired conductivity are used directly for a single simulation, which is fast and suitable for preliminary evaluation.

[0144] In Monte Carlo mode, multiple samples are taken from the probability distribution of material categories. Each time, the material parameters obtained from the sampling are used to perform a complete simulation. Finally, the statistical distribution of the multiple simulation results is output, which is suitable for high-precision scenarios.

[0145] The urban electromagnetic voxel parameter inversion system integrating multi-source remote sensing and street view provided by this invention is applied to the urban electromagnetic voxel parameter inversion method integrating multi-source remote sensing and street view, including:

[0146] The data alignment module is used to acquire multi-source heterogeneous data and unify it into the same geographic coordinate system, establish the association relationship of the same building instance in the multi-source heterogeneous data, obtain the aligned multi-source heterogeneous data, extract building facade features, and output the aligned multi-source heterogeneous data and the extracted building facade features.

[0147] The segmentation processing module is used to classify building surface materials based on aligned multi-source heterogeneous data and building facade features through a multimodal semantic segmentation network, and output the probability distribution of material categories.

[0148] The parameter inversion module is used to map the probability distribution of material type to a preset frequency-dependent electromagnetic parameter library and invert the target electromagnetic parameters and their uncertainty measure.

[0149] The voxel annotation module is used to annotate the target electromagnetic parameters and their uncertainty measures to the outer surface properties of the corresponding three-dimensional electromagnetic voxels of the building.

[0150] The electronic device provided by the present invention includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements a method for inverting urban electromagnetic element parameters by fusing multi-source remote sensing and street view.

[0151] Example 1: High-rise building complex in urban commercial district:

[0152] Scenario: A city's CBD area, comprising 30 high-rise buildings, of which 15 are glass curtain wall office buildings, 10 are concrete residential buildings, and 5 are mixed-material commercial complexes;

[0153] Step 1: Data Collection

[0154] Gaofen-2 satellite imagery: 1m resolution, RGB+NIR four-band, covering 2km×2km;

[0155] TerraSAR-X imagery: X-band, 1m resolution, dual-view with both up-orbit and down-orbit orbits;

[0156] Baidu Street View API: A sequence of approximately 200 street view images collected along the main roads of the area;

[0157] OSM Building Outline: Vector data of this area, consisting of 30 building polygons; LiDAR DSM: 2m resolution digital surface model;

[0158] Step 2: Multi-source data alignment:

[0159] Based on the OSM outline, satellite imagery is registered after fine correction using the RPC model;

[0160] SAR images are registered to UTM coordinates based on optical-SAR feature point matching (SIFT plus mutual information);

[0161] Street view images are used to recover the pose of buildings using camera GPS and SfM, and the building facade pixels are projected onto UTM coordinates with an average projection error of less than 0.5m.

[0162] Step 3: MBFST segmentation:

[0163] For each building, extract satellite ROI (256×256 pixels), SAR ROI (256×256 pixels), and street view facade sequence (6 images per building on average, 512×512 pixels each); and output the material probability distribution of each facade of each building through MBFST network inference.

[0164] result:

[0165] Building A (glass curtain wall office building): P(glass)=0.92, P(metal_panel)=0.06, P(concrete)=0.02;

[0166] Building B (concrete dwelling): P(concrete) = 0.88, P(brick) = 0.10, P(unknown) = 0.02;

[0167] Building C (commercial complex, ground floor stone with upper glass): Ground floor P(stone) = 0.85, upper floor P(glass) = 0.90;

[0168] Step 4: Electromagnetic parameter inversion:

[0169] Regarding Building A: (3.5GHz) = 0.92×6.27 + 0.06×∞ + 0.02×5.31 ≈6.1 (equivalent dielectric constant);

[0170] Regarding Building B: (3.5GHz) = 0.88×5.31 + 0.10×3.91 ≈ 5.14;

[0171] Parameter uncertainty: Var[ = 0.88×(5.31-5.14)² + 0.10×(3.91-5.14)² ≈0.18;

[0172] Step 5: Voxel annotation:

[0173] The inversion parameters were labeled onto approximately 5 × 10⁻⁶ units within a 2 km × 2 km × 200 m space in this region. 6 On each electromagnetic voxel; the outer surface of the voxel of the glass curtain wall building is marked with a high reflectivity. ≈0.45, the reflectance of concrete buildings is rated as moderate. ≈0.42.

[0174] Example 2: Brick and Stone Building Complex in the Old Town

[0175] Scene: An old town, where the buildings are mainly 19th-century brick and stone structures with red tile roofs and some buildings have wooden decorations on their facades;

[0176] The old town is densely built, and satellite imagery suffers from severe obstruction (narrow streets only allow rooftops to be seen from above); street view images are limited by street width and shooting angles, making it difficult to capture the full view of buildings.

[0177] Satellite view: Focuses on identifying roof materials (red tiles, metal roofs), with an accuracy rate of 85% in roof material identification;

[0178] SAR perspective: Masonry walls scatter weakly in SAR, while wooden decorations appear bright due to angular reflection, which helps to distinguish the mixed masonry and wood facades;

[0179] Street view perspective: The street view along the main street captures the front facade of the building, and the back street facade is inferred by the "consistency of architectural style in the same block" assumption (that is, adjacent buildings in the same block are likely to use the same materials and architectural style).

[0180] Results: The accuracy rate for roof material identification was 88%; the accuracy rate for front facade material identification was 92%; the accuracy rate for back street facade inference was 75% (based on the assumption of style consistency); and the overall volume material labeling accuracy rate (weighted average) was approximately 85%.

[0181] Example 3: Mixed-material building (commercial complex)

[0182] Scene: A commercial complex with a stone base on the ground floor (floors 1 to 3), a glass curtain wall on the middle floor (floors 4 to 15), and a metal roof equipment room on the top floor (floors 16 to 18).

[0183] Hybrid material modeling:

[0184] For a single building facade containing multiple materials, this invention adopts a layered voxel annotation strategy: the building is divided into layers along the height direction: bottom layer (z < 15m), middle layer (15m < z < 60m), top layer (z > 60m); material segmentation and parameter inversion are performed independently for each layer; during voxel annotation, the corresponding layer is matched according to the voxel z coordinate, and the material parameters of that layer are annotated.

[0185] For material transition areas (such as the boundary between stone and glass): on the voxel patch at the boundary, a dual-material hybrid annotation is used. The patch is divided into two sub-regions, with stone parameters and glass parameters annotated respectively. During ray tracing, the effective material parameters of the point are obtained by interpolation based on the intersection of the ray and the patch.

[0186] Example 4: Large-scale city batch processing

[0187] Scenario: The main urban area of ​​a provincial capital city, with approximately 100,000 buildings, covering an area of ​​200 km²;

[0188] Data preparation: 20 high-resolution satellite images (covered in sections), total data size approximately 50GB; 10 SAR images (ascending and descending orbits), total data size approximately 30GB; street view images collected along main and secondary roads, approximately 50,000 images, total data size approximately 100GB; OSM outlines of 100,000 building polygons; LiDAR DSM 200km² full coverage;

[0189] Parallel processing: The city is divided into 100 1km×2km tiles; each tile independently performs multi-source alignment to segmentation to inversion to annotation; parallel processing is performed using an 8×NVIDIA A100 GPU cluster, with approximately 10 minutes per tile; the total processing time is approximately 17 hours.

[0190] 1,000 buildings were randomly selected for manual verification; the overall material recognition accuracy was 91.3%; buildings with an accuracy of less than 80% were marked to trigger supplementary street view data collection.

[0191] Table 1 below shows a comparison of the material identification accuracy between the present invention and existing technologies:

[0192] Table 1:

[0193] ;

[0194] This invention is the first to deeply fuse three complementary data sources: optical satellite imagery (overhead view), synthetic aperture radar imagery (scattering features), and near-ground street view imagery (facade details). Optical imagery provides color and texture information, synthetic aperture radar imagery is unaffected by lighting and fog and is sensitive to metal surfaces, and street view imagery provides fine facade details. The three data sources complement each other, comprehensively improving material recognition accuracy. This invention achieves a 92% accuracy rate for glass curtain walls, 94% for concrete, 91% for red brick, 88% for stone, and 93% for metal panels, with an overall accuracy rate of 91.3%. In contrast, existing open street view maps have a material label coverage of less than 5%, and pure optical image segmentation accuracy is only about 60% to 70%. This invention significantly improves material recognition accuracy.

[0195] This invention establishes a fine classification system for 15 categories of building surface materials. This classification system directly corresponds to the standard material parameter library of the International Telecommunication Union Radiocommunication Sector, enabling the material identification results to be seamlessly mapped to the physical parameters required for electromagnetic simulation, thus avoiding the problem of parameter value deviation caused by overly coarse material classification.

[0196] This invention establishes an automated mapping pipeline from semantic segmentation results to frequency-dependent electromagnetic parameters: the material probability distribution output by the segmentation network is automatically calculated by weighted summation of the expected dielectric constant and expected conductivity, and then the parameter values ​​at any target frequency are supported by logarithmic frequency domain cubic spline interpolation. Finally, the reflection coefficient and transmission coefficient are automatically pre-calculated by Fresnel formula, and finally automatically labeled into the electromagnetic voxel attributes. The entire process requires no manual intervention, the processing time for a single building is less than 1 minute, and the full processing time for a medium-sized city (100,000 buildings) is less than 1 week (GPU cluster parallel processing), which is significantly more efficient than manual building-by-building labeling.

[0197] This invention uses the vector building outlines of open street view maps as the index reference, assigns a globally unique identifier to each building, and associates the same building instance in satellite imagery, synthetic aperture radar imagery, and street view imagery through spatial overlap judgment, realizing automatic mapping from two-dimensional pixels to three-dimensional voxel patches. It judges whether each building voxel's six faces are exposed to air one by one, and only annotates the material parameters of the outer surface, avoiding the calculation of internal voxels. It is suitable for automated processing of cities of any size.

[0198] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for inverting urban electromagnetic pixel parameters by integrating multi-source remote sensing and street view, characterized in that, Includes the following steps: Data acquisition and alignment steps: Acquire multi-source heterogeneous data and unify them into the same geographic coordinate system, establish the association relationship of the same building instance in the multi-source heterogeneous data, obtain the aligned multi-source heterogeneous data, extract building facade features, and output the aligned multi-source heterogeneous data and the extracted building facade features. Facade fusion and segmentation steps: Based on the aligned multi-source heterogeneous data and building facade features, the building surface materials are classified through a multimodal semantic segmentation network, and the probability distribution of material categories is output. Parameter inversion steps: Based on the probability distribution of material type, map to the preset frequency-dependent electromagnetic parameter library, and invert to obtain the target electromagnetic parameters and their uncertainty measure; Voxel annotation steps: Annotate the target electromagnetic parameters and their uncertainty measures to the outer surface properties of the corresponding three-dimensional electromagnetic voxels of the building.

2. The method for inverting urban electromagnetic voxel parameters by integrating multi-source remote sensing and street view as described in claim 1, characterized in that: In the data acquisition and alignment steps, multi-source heterogeneous data include at least optical satellite imagery, synthetic aperture radar imagery, and near-ground street view images; The data acquisition and alignment steps specifically include: After fine correction using a rational polynomial coefficient model, the optical satellite images are registered to the universal transverse Mercator projection coordinate system. Synthetic aperture radar images are registered to the universal transverse Mercator projection coordinate system by feature point matching between optical images and synthetic aperture radar images. The camera pose is determined based on the GPS tags and motion recovery structure 3D reconstruction of the street view image, and the 2D street view pixels are back-projected to the 3D universal transverse Mercator projection coordinates. Using the vector building outlines of the open street view map as the index, the same building instances in multi-source heterogeneous data are associated by judging the spatial overlap ratio, and each building is assigned a globally unique identifier.

3. The method for inverting urban electromagnetic voxel parameters by integrating multi-source remote sensing and street view as described in claim 2, characterized in that: In the data acquisition and alignment steps, building facade features are extracted from the street view images, specifically including: Motion recovery and structural pose restoration are performed on the street view image sequence collected along the street to determine the camera position and orientation of each street view image; Project the pixel regions belonging to the same building facade in each street view image onto the local three-dimensional coordinate system of that building; Depth values ​​are assigned to building facade pixels in street view images by multi-view stereo matching or deep learning depth estimation methods. Multi-view facade images are stitched together to create a panoramic view of the building's facade, which is then incorporated as part of the building's facade features.

4. The method for inverting urban electromagnetic voxel parameters by integrating multi-source remote sensing and street view as described in claim 3, characterized in that: The multimodal semantic segmentation network used in the facade fusion and segmentation steps specifically includes: A satellite-overlooking encoder is used to extract visible light and near-infrared features of the building's overhead area from optical satellite imagery. Synthetic Aperture Radar (SAR) encoders are used to directly process in-phase and orthogonal complex data from SAR images and extract scattering feature maps. Street view facade encoder for extracting sequential features from a wraparound panoramic facade image; The cross-modal feature fusion module is used to generate fused features by using the features output by the street view facade encoder as the main query and the features output by the satellite overhead encoder and synthetic aperture radar encoder as the key values ​​through a cross-modal attention mechanism. The decoder is used to decode the fused features and output the probability distribution of each surface of the building belonging to each material category.

5. The method for inverting urban electromagnetic voxel parameters by integrating multi-source remote sensing and street view as described in claim 4, characterized in that: The material categories output by the multimodal semantic segmentation network include the following fifteen categories: ordinary concrete, smooth concrete, red brick, stone, ordinary glass, metal-coated glass, glazed ceramic tile, metal plate, aluminum curtain wall, painted wall, wood structure, gypsum board, composite material, vegetation cover, and unknown mixed material. Furthermore, the multimodal semantic segmentation network simultaneously outputs the following three uncertainty measures for each building surface: Classification entropy is used to represent the overall degree of uncertainty in classifying a surface as a material. The difference between the probability of the most likely material and the probability of the second most likely material is used to represent the difficulty of distinguishing between the top two candidate materials; Cross-modal consistency measure is used to represent the degree of mutual consistency between prediction results using only satellite imagery, only synthetic aperture radar imagery, and only street view imagery.

6. The method for inverting urban electromagnetic voxel parameters by integrating multi-source remote sensing and street view as described in claim 5, characterized in that: In the parameter inversion step, the specific method for obtaining the target electromagnetic parameters and their uncertainty measure based on the probability distribution of material type is as follows: Using the probability of each material category as a weight, the dielectric constant of each material category at the target frequency is weighted and summed to obtain the expected dielectric constant; the conductivity of each material category at the target frequency is weighted and summed to obtain the expected conductivity. Using the probability of each material category as a weight, the variances of the desired dielectric constant and desired conductivity are calculated respectively, and the variances are used as a measure of the uncertainty of the target electromagnetic parameters. When the frequency-dependent electromagnetic parameter library only provides parameter values ​​at discrete frequency points, cubic spline interpolation is performed on the known discrete points in the logarithmic frequency domain to obtain the dielectric constant and conductivity values ​​at any target frequency.

7. The method for inverting urban electromagnetic voxel parameters by integrating multi-source remote sensing and street view as described in claim 6, characterized in that: In the parameter inversion step, after obtaining the desired dielectric constant and desired conductivity, a pre-calculation step for the reflection coefficient and transmission coefficient is also included: Calculate the intrinsic impedance of the material based on the desired dielectric constant and desired conductivity; Based on the intrinsic impedance and the Fresnel formula, calculate the vertical polarization reflection coefficient and the horizontal polarization reflection coefficient of electromagnetic waves on the voxel surface of the building. The vertical polarization reflection coefficient, the horizontal polarization reflection coefficient, and the transmission coefficient derived from the reflection coefficient are stored as additional attribute fields for direct use by downstream ray tracing.

8. The method for inverting urban electromagnetic voxel parameters by integrating multi-source remote sensing and street view as described in claim 7, characterized in that: The voxel annotation process specifically includes: For each building voxel, determine whether each of its six faces is adjacent to an air voxel, and identify the face adjacent to the air voxel as the outer surface. For voxel patches identified as outer surfaces, the corresponding building facade area output by the multimodal semantic segmentation network is matched according to the position coordinates and orientation of the patch in three-dimensional space. The matched material category probability distribution, expected dielectric constant, expected conductivity, and uncertainty measure are all labeled into the material properties of the voxel surface. The confidence score of the annotation result is calculated based on the information entropy of the probability distribution of the material category, and the confidence score is written into the voxel attribute. In downstream electromagnetic simulation, two operating modes are supported: in deterministic mode, the desired dielectric constant and desired conductivity are used directly for a single simulation; in Monte Carlo mode, multiple samples are taken from the probability distribution of material categories, and a complete simulation is performed each time using the material parameters obtained from the sampling, and finally the statistical distribution of multiple simulation results is output.

9. A system for inverting urban electromagnetic voxel parameters by integrating multi-source remote sensing and street view, applied to the urban electromagnetic voxel parameter inversion method by integrating multi-source remote sensing and street view as described in any one of claims 1-8, characterized in that, include: The data alignment module is used to acquire multi-source heterogeneous data and unify it into the same geographic coordinate system, establish the association relationship of the same building instance in the multi-source heterogeneous data, obtain the aligned multi-source heterogeneous data, extract building facade features, and output the aligned multi-source heterogeneous data and the extracted building facade features. The segmentation processing module is used to classify building surface materials based on aligned multi-source heterogeneous data and building facade features through a multimodal semantic segmentation network, and output the probability distribution of material categories. The parameter inversion module is used to map the probability distribution of material type to a preset frequency-dependent electromagnetic parameter library and invert the target electromagnetic parameters and their uncertainty measure. The voxel annotation module is used to annotate the target electromagnetic parameters and their uncertainty measures to the outer surface properties of the corresponding three-dimensional electromagnetic voxels of the building.

10. An electronic device, characterized in that, It includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the urban electromagnetic element parameter inversion method that integrates multi-source remote sensing and street view as described in any one of claims 1-8.