A method for estimating the height of urban buildings using street view images under severe occlusion

Abstract

The present invention provides a method for estimating the height of urban buildings using street view images under severe occlusion conditions: first, obtain building footprint data and street view panoramic images; second, use a spherical panoramic camera as an intermediary to establish a projection conversion model of two-dimensional street view targets and three-dimensional geographic entities to achieve the correspondence between street view panoramic images and buildings in geographic space; then, in view of the widespread occlusion of buildings in street view images, a method for eliminating pseudo-height feature points of buildings based on slope mutation test is proposed to identify the real building height feature points under severe occlusion conditions; finally, project the two-dimensional image coordinates of the true height feature points into three-dimensional geographic space to estimate the height of the building. On the one hand, the present invention solves the perspective deformation problem of street view images in the traditional pinhole camera model, and on the other hand, eliminates the influence of severe occlusion in street view images on the extraction of building height feature points, significantly improving the accuracy and stability of building height estimation.

Description

Technical Field

[0001] The invention belongs to the field of urban three-dimensional modeling, and in particular to a method for estimating the height of urban buildings by using street view images under severe occlusion conditions. Background Art

[0002] By 2050, cities will carry more than 60% of the world's population with only 1.13% of the world's land area. The vertical expansion of cities has become an inevitable trend to accommodate population growth in a limited space. Building height is a key parameter of 3D urban morphology and is essential for understanding the impact of vertical characteristics on urban development and evolution. Building height has been widely used in many fields, such as 3D reconstruction, disaster assessment, population estimation, and smart cities. With the continuous development of cities, the density of buildings is increasing, and municipal facilities such as trees, billboards, and telephone poles are closely distributed around buildings. Especially in crowded old urban areas, dense tree canopies often severely block buildings, which poses a severe challenge to the observation of building height.

[0003] Remote sensing technology plays a key role in estimating the height of urban buildings. The estimation method of the height of a single building mainly relies on a variety of remote sensing data sources, including optical remote sensing images, synthetic aperture radar (SAR) images, and light detection and ranging (LiDAR) data. Using the shadow of the building in the optical remote sensing image to invert the building height is a classic and effective method. It extracts the shadow of the building and combines the azimuth and altitude of the sun and satellite to establish a spatial model to invert the building height. Another commonly used method is stereo image pair matching, which uses two or more optical images with a certain parallax to infer the height of the building through stereo matching and triangulation. This method is also applicable to SAR data. By measuring the target phase difference between multiple radar images, the elevation information of the target is obtained. Finally, LiDAR technology can directly obtain the 3D coordinate point cloud of the target and use it to directly estimate the height of the building. It is a direct and effective method. Although the above methods have played an important role in estimating building heights, they have some limitations in terms of observation perspective and usage cost: first, the top-down observation perspective is limited in application in areas with low and dense buildings; second, high-resolution remote sensing images, SAR data, and LiDAR data are not widely available, especially in underdeveloped areas. The acquisition of these data usually relies on expensive equipment, resulting in high data collection costs, which restricts their promotion and application on a large scale.

[0004] With the rapid development of the Internet and electronic map technology, many street view platforms such as Google Street View, Baidu Street View, OpenStreetMap, etc. have emerged, covering 114 countries and regions. Street view data, with its wide coverage, free access, and rich urban 3D information, provides a new alternative data for bottom-up observation of building objects. The basic format of street view images is street view panorama, which is composed of multiple images pointing in different directions through a stitching algorithm to form a 360° surround view to reflect the 3D information of the real world. However, since the specific implementation details of the stitching algorithm are not disclosed, the order and direction of the stitched images are unclear, which makes the street view panorama lack directional information and difficult to automatically align with buildings in the real world, thus limiting the estimation of building height. Relatively speaking, street view images are more commonly used to estimate building height. They project a rectangular area in the street view panorama by given the observation direction and image size. Its corresponding parameters, such as position, direction angle and elevation angle, can be obtained after the street view platform is projected, which is crucial for estimating the height of the building.

[0005] At present, accurate estimation of building height based on street view data mainly relies on two methods. One is the monocular method and the other is the pinhole camera method. The monocular method requires a single street view image, a reference object, and parameters such as the field of view, and applies geometric and optical principles to measure the size of the target object. The pinhole camera method extracts building height features from street view images, such as roof lines, corners, or the highest point of the building. Using the pinhole camera model, triangles are constructed using the distance between the street view image and the building and the location of the building height feature points to estimate the building height. On the one hand, both methods are affected by camera parameters, requiring that the building photographed by the street view image must be parallel to the phase plane, that is, the building must be photographed facing the building. This is because when the building is not parallel to the phase plane of the street view image, the shape and size of the building in the image will be perspectively deformed. The degree of perspective deformation depends on the angle between the building and the phase plane of the street view image. The larger the angle, the more severe the perspective deformation. On the other hand, street view data in cities will inevitably produce buildings that are blocked. For example, the building may be blocked by objects such as trees, vehicles, and telephone poles in front. The boundaries of these objects can easily be confused with the real roof line of the building, resulting in pseudo-height feature points. This situation will lead to errors in the building height estimation. This problem becomes particularly prominent in the case of severe occlusion, greatly increasing the uncertainty of building height estimation. Therefore, the treatment method for severe occlusion needs to be paid more attention and studied in depth to improve the accuracy and reliability of building height estimation. Summary of the invention

[0006] The technical problem to be solved by the present invention is: accurately estimating the height of buildings under severe occlusion of street view images, proposing a method for estimating the height of urban buildings using street view images under severe occlusion, establishing a projection conversion model of two-dimensional street view targets-three-dimensional geographic entities to solve the perspective deformation problem of street view images in traditional pinhole camera models, eliminating its potential impact on building height estimation, and using a method for eliminating pseudo-height feature points of buildings based on slope mutation test to identify real building height feature points under severe occlusion, and further obtaining image coordinates of the real height feature points of the buildings and projecting them into 3D geographic space to estimate the building height.

[0007] The present invention solves the technical problem by adopting the following technical solutions:

[0008] The method for estimating the height of urban buildings using street view images under severe occlusion provided by the present invention is specifically:

[0009] (1) Obtain building footprint data and street view panoramic images;

[0010] (2) Using the spherical panoramic camera as an intermediary, a projection transformation model of two-dimensional street view objects to three-dimensional geographic entities is established to achieve the correspondence between buildings in geographic space and street view panoramic images.

[0011] (3) For the widespread situation of building occlusion in street view images, a method for eliminating pseudo-height feature points of buildings based on slope mutation test is proposed to identify the real building height feature points under severe occlusion.

[0012] (4) Project its two-dimensional image coordinates into three-dimensional geographic space to estimate the height of the building.

[0013] In the above method, the following methods are used to obtain building footprint data and street view panoramic images: first, the building footprint data and road data in the building height estimation area are downloaded from the OpenStreetMap website; second, the point-along-line tool in ArcMap is used to generate street view collection points at intervals of 10m; finally, a Baidu Street View download program is written based on Python, and the street view panoramic images at the street view collection points and their camera height and yaw angle are obtained based on the Baidu Map Static Panorama API.

[0014] In the above method, a spherical panoramic camera is used as an intermediary to align the vehicle's forward direction in the street view panoramic image with the yaw angle of the street view panoramic image, thereby establishing a projection conversion model of a two-dimensional street view target-three-dimensional geographic entity, and realizing the correspondence between the geographic space and the buildings in the street view panoramic image, which specifically includes:

[0015] (1) Establishing two-dimensional street view panoramic image coordinates, spherical panoramic camera coordinates and three-dimensional geographic space coordinates;

[0016] (2) Establish the correspondence between two-dimensional pixel coordinates and spherical panoramic camera coordinates, establish the correspondence between panoramic camera coordinates and three-dimensional geospatial coordinates, construct a projection transformation model of two-dimensional street view targets and three-dimensional geographic entities, and realize the correspondence between buildings in geographic space and street view panoramic images.

[0017] In the above method, the establishment of two-dimensional street view panoramic image coordinates, spherical panoramic camera coordinates and three-dimensional geographic space coordinates specifically includes:

[0018] (1) Taking the upper left corner of the street view panoramic image as the origin, and using its length and height as the horizontal and vertical coordinates, respectively, to establish the image coordinates of the two-dimensional street view panoramic image, and uniquely representing each pixel by its column number and row number;

[0019] (2) Using a spherical panoramic camera model, with due north as 0°, defining a 360° horizontal azimuth, with the horizontal direction as 0°, elevation as positive, depression as negative, defining a 180° vertical azimuth, establishing spherical panoramic camera coordinates, and uniquely representing each pixel in the image with a horizontal azimuth and a vertical azimuth;

[0020] (3) A three-dimensional geographic space coordinate system is established with the camera position as the origin, the east direction as the positive x-axis, the north direction as the positive y-axis, and the top as the positive z-axis. The coordinates of the three coordinate axes uniquely represent the position in space.

[0021] In the above method, the construction of a projection conversion model of a two-dimensional street view target-three-dimensional geographic entity to achieve the correspondence between the geographic space and the buildings in the street view panoramic image specifically includes:

[0022] (1) Align the vehicle's forward direction in the street view panoramic image with the yaw angle of the street view panoramic image, and establish a corresponding relationship between the two-dimensional pixel coordinates and the spherical panoramic camera coordinates. The process is as follows: Align the vehicle's forward direction in the street view panoramic image with the yaw angle of the street view panoramic image, and establish a corresponding relationship between the two-dimensional pixel coordinates and the spherical panoramic camera coordinates. The specific relationship is expressed by the following formula:

[0023]

[0024] h=(90-ρ)H / 180#

[0025] Where W is the length of the street view panoramic image, H is the height of the street view panoramic image, w and h are the horizontal and vertical coordinates of the pixels in the street view panoramic image, and C yaw is the column number of the vehicle's forward direction in the street view panoramic image. and ρ represent the horizontal and vertical azimuth angles of the pixel in the panoramic camera coordinates. θ represents the yaw angle of the street view panoramic image.

[0026] (2) Taking the street view panoramic image location as the observation point and the building as the observed point, the Build Line of Sight tool in ArcMap software is used to determine the horizontal distance from the panoramic image location to the building at different azimuths and estimate the building height. The specific calculation formulas for horizontal distance and height are as follows:

[0027]

[0028]

[0029] z=dtanρ+h camera #,

[0030] Where d represents the azimuth from the panoramic image position to Horizontal distance of the building in the direction, h camera is the camera height, and x, y, and z are the coordinates of the street view panoramic image pixel coordinates projected into 3D geographic space.

[0031] In the above method, for the situation where buildings are widely blocked in street view images, the method for eliminating pseudo-height feature points of buildings based on slope mutation test is used to identify the real height feature points of buildings under severe blockage, which specifically includes:

[0032] (1) locating buildings in the street view panoramic image based on the projection conversion model of the two-dimensional street view target-three-dimensional geographic entity;

[0033] (2) For the situation where buildings are widely occluded in street view images, a method for eliminating pseudo-height feature points of buildings based on slope mutation test is used to identify the real height feature points of buildings under severe occlusion.

[0034] In the above method, the step of locating a building in a street view panoramic image specifically includes:

[0035] (1) The street view panoramic image is processed through a preset semantic segmentation model to extract pixels with building types, thereby obtaining the accurate outline of the building;

[0036] (2) extracting building breakpoints from building footprint data;

[0037] (3) Taking the location of the street view panoramic image as the observation point and the building inflection point as the observed point, an observation sightline is constructed to obtain the horizontal azimuth and horizontal distance of the building inflection point observed from the street view panoramic image;

[0038] (4) Based on the projection transformation relationship from the street view panoramic image to the 3D geographic space, the horizontal azimuth angle of each building turning point is converted into a pixel column number, and the pixel column range of each building is located in the street view panoramic image.

[0039] In the above method, the method for eliminating pseudo-height feature points of buildings based on slope mutation test identifies the real height feature points of buildings under severe occlusion, specifically including:

[0040] (1) Scanning row by row within the building to identify the minimum vertical coordinate value of the building pixel to form the roof line of the building;

[0041] (2) Calculate the slope of the roof line between the two inflection points of the building and generate a slope change curve. The slope change formula is as follows:

[0042]

[0043] Where k is the slope of the roof line, h2-h1 is the image ordinate difference between two adjacent pixels on the roof line, and w2-w1 is the image abscissa difference between two adjacent pixels on the roof line.

[0044] (3) When the difference in the image ordinate at the slope change point exceeds 10% of the height of the street view panoramic image, the roof line of the building is divided into two parts. The roof line with a larger image ordinate value of the pixel is considered to be occluded, and the roof corner point of the building on this roof line is identified as a pseudo-height feature point. On the contrary, the roof corner point of the building on the unoccluded roof line is identified as a true height feature point.

[0045] In the above method, the step of obtaining the image coordinates of the building height feature points and projecting them into the 3D geographic space to estimate the building height specifically includes:

[0046] (1) Extract the image coordinates of all true height feature points of buildings in the street view panoramic image;

[0047] (2) Calculating the 3D geographic space coordinates of all true height feature points of the building based on the projection transformation relationship from the above street view panoramic image to the 3D geographic space;

[0048] (3) Sort the true building height feature points from high to low, remove the first 10% and the last 10% of the true height feature points, and average the heights of the remaining true height feature points to estimate the building height. If a true building height feature point has less than three, use the average height of all true height feature points to estimate the building height.

[0049] Compared with the prior art, the present invention has the following main technical effects:

[0050] (1) When estimating building heights based on street view images, the traditional pinhole camera model has uncorrectable perspective distortion when the building is not parallel to the street view image phase plane. This perspective distortion increases the uncertainty of building height estimation. The present invention proposes a projection transformation model from a two-dimensional street view target to a three-dimensional geographic entity, which uses a spherical panoramic camera model to map the street view panoramic image onto a spherical surface. By establishing a corresponding relationship between the street view panoramic image and the building in the geographic space, the model eliminates the influence of perspective distortion on the building height estimation, thereby improving the estimation accuracy. In addition, the model exhibits stronger adaptability to buildings with complex shapes.

[0051] (2) There are occluded scenes in the street view panorama, such as objects in front of buildings, such as trees, electric poles, etc., which are mixed with the real roof lines, resulting in incorrect identification as the roof lines of buildings, resulting in pseudo-height feature points. The present invention proposes a method for eliminating pseudo-height feature points of buildings based on slope mutation test. This method extracts the roof lines of buildings in street view images, and can identify the real building height feature points under severe occlusion according to the slope changes of the roof lines, thereby reducing the impact of common occlusion scenes in street view images on building height estimation and improving the accuracy and stability of building height estimation.

[0052] In summary, the present invention solves the perspective deformation problem of street view images in the traditional pinhole camera model on the one hand, and eliminates the influence of severe occlusion in street view images on the extraction of building height feature points on the other hand, thereby significantly improving the accuracy and stability of building height estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a schematic flow chart of the method of the present invention.

[0054] Figure 2 It is a schematic diagram of building footprints and street view data.

[0055] Figure 3 It is a schematic diagram of the projection transformation model from a two-dimensional street view target to a three-dimensional geographic entity.

[0056] Figure 4 It is a schematic diagram of the scope of buildings located in the street view panoramic image.

[0057] Figure 5 It is a schematic diagram of the PSPNet semantic segmentation model extracting building outlines.

[0058] Figure 6 This is a schematic diagram of identifying unobstructed building height feature points based on slope mutation test.

[0059] Figure 7 This is a diagram of the height measurement results of the science and technology park buildings provided in this embodiment. DETAILED DESCRIPTION

[0060] The method for estimating the height of urban buildings using street view images under severe occlusion proposed in the present invention uses a spherical panoramic camera model to map the street view image onto a spherical surface, and constructs a projection conversion model of a two-dimensional street view target-three-dimensional geographic entity, thereby eliminating the potential impact of perspective deformation in street view images on building height estimation; further, a method for eliminating pseudo-height feature points of buildings based on slope mutation test is proposed to identify the real building height feature points under severe occlusion conditions, and reduce the adverse effects of common occlusion scenes in street view images on building height estimation. The projection conversion model of a two-dimensional street view target-three-dimensional geographic entity constructed by the present invention can achieve the correspondence between street view panoramic images and buildings in geographic space, and has stronger adaptability to buildings of complex shapes; at the same time, the method for eliminating pseudo-height feature points of buildings based on slope mutation test proposed in the present invention can effectively reduce the adverse effects of common occlusion scenes in street view images on building height estimation, improve the accuracy and stability of building height estimation, and provide a more reliable and effective way to solve the problem of building height estimation in urban environments.

[0061] In order to have a clearer understanding of the technical features, purposes and effects of the present invention, a science and technology park in Wuhan is taken as an example to obtain the OpenStreetMap linear road network data and OpenStreetMap building footprint data in the area, and the technical solutions in the embodiments of the present invention are clearly and completely described in combination with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments.

[0062] The method for estimating the height of urban buildings using street view images under severe occlusion conditions provided by the present invention comprises the following steps: Figure 1 As shown:

[0063] Step 1: Obtain building footprint data and street view panoramic images.

[0064] First, download the building footprint data and road data within the building height estimation area from the OpenStreetMap website;

[0065] Secondly, use the point-along-line tool in ArcMap to generate street view collection points at intervals of 10m; finally, write a Baidu Street View download program based on Python, and obtain the street view panoramic image at the street view collection point and its camera height and yaw angle based on the Baidu Map static panorama API. Figure 2 shown.

[0066] Step 2: Using the spherical panoramic camera as an intermediary, a projection transformation model of 2D street view target-3D geographic entity is established to achieve the correspondence between geographic space and buildings in street view panoramic images, such as Figure 3 As shown;

[0067] (1) Establishing the image coordinates of the two-dimensional street view panoramic image, the coordinates of the spherical panoramic camera and the three-dimensional geographic space coordinates. In step 2, the image coordinates of the two-dimensional street view panoramic image, the coordinates of the spherical panoramic camera and the three-dimensional geographic space coordinates are established. The process is as follows:

[0068] 1) Taking the upper left corner of the street view panoramic image as the origin, and using its length 1024 and height 512 as the horizontal and vertical coordinates respectively, a 1024*512 two-dimensional street view panoramic image coordinate is established, and each pixel is uniquely represented by the column number and row number of the pixel;

[0069] 2) Using a spherical panoramic camera model, with due north as 0°, define a 360° horizontal azimuth, with the horizontal direction as 0°, elevation as positive, depression as negative, define a 180° vertical azimuth, establish spherical panoramic camera coordinates, and uniquely represent each pixel in the image with a horizontal azimuth and a vertical azimuth;

[0070] 3) A three-dimensional geographic space coordinate system is established with the camera position as the origin, the east direction as the positive x-axis, the north direction as the positive y-axis, and the top as the positive z-axis. The coordinates of the three coordinate axes uniquely represent the position in space.

[0071] (2) Establish the correspondence between the 2D pixel coordinates and the spherical panoramic camera coordinates, and establish the correspondence between the panoramic camera coordinates and the 3D geospatial coordinates. Construct a projection transformation model of 2D street view objects and 3D geographic entities to achieve the correspondence between geographic space and buildings in street view panoramic images. The specific process is as follows:

[0072] 1) Align the vehicle's forward direction in the street view panoramic image with the yaw angle of the street view panoramic image, and establish the corresponding relationship between the two-dimensional pixel coordinates and the spherical panoramic camera coordinates. The process is: align the vehicle's forward direction in the street view panoramic image with the yaw angle of the street view panoramic image, and establish the corresponding relationship between the two-dimensional pixel coordinates and the spherical panoramic camera coordinates. The specific relationship is expressed by the following formula:

[0073]

[0074] h=(90-ρ)H / 180#,

[0075] Where W is the length of the street view panoramic image, H is the height of the street view panoramic image, w and h are the horizontal and vertical coordinates of the pixels in the street view panoramic image, and C yaw is the column number of the vehicle's forward direction in the street view panoramic image. and ρ represent the horizontal and vertical azimuth angles of the pixel in the panoramic camera coordinates. θ represents the yaw angle of the street view panoramic image.

[0076] 2) Use the Construct Line of Sight tool in ArcMap, with the street view panoramic image location as the observation point and the building as the observed point, to determine the horizontal distance from the panoramic image location to the building at different azimuths and estimate the building height. The specific horizontal distance and height calculation formula is:

[0077]

[0078] h=(90-ρ)H / 180#,

[0079] Where d represents the azimuth from the panoramic image position to Horizontal distance of the building in the direction, h camera is the camera height, and x, y, and z are the coordinates of the street view panoramic image pixel coordinates projected into 3D geographic space.

[0080] Step 3: For the widespread situation of building occlusion in street view images, a method for eliminating pseudo-height feature points of buildings based on slope mutation test is proposed to identify the real building height feature points under severe occlusion, including:

[0081] (1) Based on the projection conversion model of the two-dimensional street view target-three-dimensional geographic entity, locate the building in the street view panoramic image, such as Figure 5 As shown, the specific process is as follows:

[0082] 1) The street view panoramic image is processed by a preset semantic segmentation model to extract pixels with building types, thereby obtaining the accurate outline of the building. The preset semantic segmentation model is a PSPNet model trained with the ADE20K dataset, such as Figure 4 As shown;

[0083] 2) Use the Feature Vertex to Point tool in ArcMap to extract the building vertex points in the building footprint data;

[0084] 3) Use the Construct Line of Sight tool in ArcMap to construct the observation line with the street view panoramic image location as the observation point and the building vertex as the observed point to obtain the horizontal azimuth and horizontal distance of the building vertex observed from the street view panoramic image, such as Figure 5 (a)

[0085] 4) Based on the projection transformation relationship from the street view panoramic image to the 3D geographic space, the horizontal azimuth angle of each building turning point is converted into a pixel column number, and the pixel column range of each building is located in the street view panoramic image, such as Figure 5(b) as shown.

[0086] (2) For the situation where buildings are widely occluded in street view images, a method for eliminating pseudo-height feature points of buildings based on slope mutation test is used to identify the real building height feature points under severe occlusion, such as Figure 6 As shown, the specific process is as follows:

[0087] 1) Scan the building column by column to identify the minimum vertical coordinate of the building pixel and form the roof line of the building, such as Figure 6 (b)

[0088] 2) Calculate the slope of the roof line between the two inflection points of the building and generate a slope change curve, such as Figure 6 As shown in (c), the slope change formula is as follows:

[0089]

[0090] Where k is the slope of the roof line, h2-h1 is the image ordinate difference between two adjacent pixels on the roof line, and w2-w1 is the image abscissa difference between two adjacent pixels on the roof line.

[0091] 3) When the image ordinate difference at the slope change location exceeds 10% of the street view panoramic image height, the roof line of the building is divided into two parts. The roof line with a larger image ordinate value of the pixel is considered to be blocked, and the building roof corner point on the roof line is identified as a pseudo-height feature point. On the contrary, the building roof corner point on the unblocked roof line is identified as a true height feature point, such as Figure 6 (d) as shown.

[0092] Step 4: Project the 2D image coordinates of the obscured height feature points of the building into the 3D geographic space to estimate the height of the building. The specific process is as follows:

[0093] (1) Extract the image coordinates of all true height feature points of buildings in the street view panoramic image;

[0094] (2) Calculating the 3D geographic space coordinates of all true height feature points of the building based on the projection transformation relationship from the above street view panoramic image to the 3D geographic space;

[0095] (3) Sort the true building height feature points from high to low, remove the first 10% and the last 10% of the true height feature points, and average the heights of the remaining true height feature points to estimate the building height. If a true building height feature point has less than three, use the average height of all true height feature points to estimate the building height.

[0096] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

[0097] Application examples:

[0098] This case takes a certain science and technology park in Wuhan as the research area, uses the proposed method of estimating the height of urban buildings using street view images under severe occlusion conditions, and further illustrates the invention in conjunction with the accompanying drawings.

[0099] Specific processing steps: Figure 1 )as follows:

[0100] Step 1: Obtain building footprint data and street view panoramic images. The specific process is as follows:

[0101] (1) Download building footprint data and road data within the building height estimation area from the OpenStreetMap website;

[0102] (2) Use the Generate Points Along Line tool in ArcMap to generate street view collection points at intervals of 10 m;

[0103] (3) Finally, a Baidu Street View download program is written based on Python, and the street view panoramic image at the street view collection point and its camera height and yaw angle are obtained based on the Baidu Map static panoramic image API; among them, the building footprint data, street view collection point data and street view panoramic image examples in this embodiment are as follows Figure 2 shown.

[0104] Step 2: Establish the 2D street view panoramic image coordinates, spherical panoramic camera coordinates and 3D geospatial coordinates. The process is as follows:

[0105] (1) Taking the upper left corner of the street view panoramic image as the origin, and using its length 1024 and height 512 as the horizontal and vertical coordinates, respectively, a 1024*512 two-dimensional street view panoramic image coordinate system is established, and each pixel is uniquely represented by its column number and row number;

[0106] (2) Using a spherical panoramic camera model, with due north as 0°, defining a 360° horizontal azimuth, with the horizontal direction as 0°, elevation as positive, depression as negative, defining a 180° vertical azimuth, establishing spherical panoramic camera coordinates, and uniquely representing each pixel in the image with a horizontal azimuth and a vertical azimuth;

[0107] (3) A three-dimensional geographic space coordinate system is established with the camera position as the origin, the east direction as the positive x-axis, the north direction as the positive y-axis, and the top as the positive z-axis. The coordinates of the three coordinate axes uniquely represent the position in space.

[0108] (4) Construct a projection conversion model of 2D street view target-3D geographic entity, align the vehicle forward direction in the street view panoramic image with the yaw angle of the street view panoramic image, and establish the corresponding relationship between the 2D pixel coordinates and the spherical panoramic camera coordinates. The process is: align the vehicle forward direction in the street view panoramic image with the yaw angle of the street view panoramic image, and establish the corresponding relationship between the 2D pixel coordinates and the spherical panoramic camera coordinates. The specific relationship is expressed by the following formula:

[0109]

[0110] h=(90-ρ)H / 180#,

[0111] Where W is the length of the street view panoramic image, H is the height of the street view panoramic image, w and h are the horizontal and vertical coordinates of the pixels in the street view panoramic image, and C yaw It is the column number of the vehicle's forward direction in the street view panoramic image. and ρ represent the horizontal and vertical azimuth angles of the pixel in the panoramic camera coordinates. θ represents the yaw angle of the street view panoramic image.

[0112] (5) Use the Construct Line of Sight tool in ArcMap, with the street view panoramic image location as the observation point and the building as the observed point, to determine the horizontal distance from the panoramic image location to the building at different azimuths and estimate the building height. The specific calculation formula for horizontal distance and height is:

[0113]

[0114]

[0115] z=dtanρ+h camera #,

[0116] Where d represents the azimuth from the panoramic image position to Horizontal distance of the building in the direction, h camera is the camera height, and x, y, and z are the coordinates of the street view panoramic image pixel coordinates projected into 3D geographic space.

[0117] Step 3: For the situation where buildings are widely blocked in street view images, a method for eliminating pseudo-height feature points of buildings based on slope mutation test is used to identify the real height feature points of buildings under severe blockage;

[0118] (1) The street view panoramic image is processed by a preset semantic segmentation model to extract pixels with building types, thereby obtaining the accurate outline of the building. The preset semantic segmentation model is a PSPNet model trained with the ADE20K dataset, such as Figure 4 As shown;

[0119] (2) Use the feature vertex to point tool in ArcMap to extract the building vertex in the building footprint data;

[0120] (3) Use the Construct Line of Sight tool in ArcMap to construct the observation line with the street view panoramic image location as the observation point and the building vertex as the observed point to obtain the horizontal azimuth and horizontal distance of the building vertex observed from the street view panoramic image, as shown in the figure below: Figure 5 (a)

[0121] (4) Based on the projection transformation relationship from the street view panoramic image to the 3D geographic space, the horizontal azimuth angle of each building vertex is converted into a pixel column number, and the pixel column range of each building is located in the street view panoramic image, such as Figure 5 (b) as shown.

[0122] (5) Scanning the building column by column to identify the minimum vertical coordinate of the building pixel, forming the roof line of the building, such as Figure 6 (b)

[0123] (6) Calculate the slope of the roof line between the two inflection points of the building and generate a slope change curve, such as Figure 6 As shown in (c), the slope change formula is as follows:

[0124]

[0125] Where k is the slope of the roof line, h2-h1 is the image ordinate difference between two adjacent pixels on the roof line, and w2-w1 is the image abscissa difference between two adjacent pixels on the roof line.

[0126] (7) When the image ordinate difference at the slope change location exceeds 10% of the height of the street view panoramic image, the roof line of the building is divided into two parts. The roof line with a larger image ordinate value of the pixel is considered to be blocked, and the building roof corner point on the roof line is identified as a pseudo-height feature point. On the contrary, the building roof corner point on the unblocked roof line is identified as a true height feature point, such as Figure 6 (d) as shown.

[0127] Step 4: Project the 2D image coordinates of the obscured height feature points of the building into the 3D geographic space to estimate the height of the building. The specific process is as follows:

[0128] (1) Extract the image coordinates of all true height feature points of buildings in the street view panoramic image;

[0129] (2) Calculating the 3D geographic space coordinates of all true height feature points of the building based on the projection transformation relationship from the above street view panoramic image to the 3D geographic space;

[0130] (3) Sort the real building height feature points from high to low, remove the first 10% and the last 10% of the real height feature points, and average the heights of the remaining real height feature points to estimate the building height. If a real building height feature point is less than three, the average height of all the real height feature points is used to estimate the building height. The building height estimation results in this application case are as follows: Figure 7 As shown, the error of the estimated building height is 1.85m compared with the actual height, and the relative error is 5.89%. The buildings with errors less than 2m account for 77.77% and the buildings with errors less than 4m account for 95.56%.

[0131] The method for estimating the height of urban buildings using street view images under severe occlusion provided by the present invention has the following characteristics:

[0132] (1) When estimating building heights using street view images, when the buildings are not parallel to the street view image phase plane, the buildings in the street view images will produce uncorrectable perspective deformation. By using street view panoramic images and mapping them onto a spherical surface based on a spherical panoramic camera model, the influence of perspective deformation on building height estimation is eliminated, thereby improving the accuracy of building height estimation.

[0133] (2) Considering the existence of occluded scenes in street view panoramas, a method for eliminating pseudo-building height feature points based on slope mutation test is adopted to identify and eliminate pseudo-building height feature points in street view panorama images, thereby improving the accuracy and stability of building height estimation.

[0134] (3) In response to the problem that traditional building height estimation requires expensive measurement equipment and a large amount of human resources, a standardized ArcMap processing flow was designed to achieve low-cost and fast building height estimation to meet the needs of urban building height estimation over a wide range and provide data support for urban research.

[0135] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for estimating the height of urban buildings using street view images under severe occlusion conditions, characterized by: (1) Obtain building footprint data and street view panoramic images; (2) Using the spherical panoramic camera as an intermediary, a projection transformation model of two-dimensional street view objects to three-dimensional geographic entities is established to achieve the correspondence between geographic space and buildings in street view panoramic images, using the following methods: First, establish the image coordinates of a two-dimensional street view panoramic image, the coordinates of a spherical panoramic camera and the coordinates of a three-dimensional geographic space, and the steps include: 1) taking the upper left corner of the street view panoramic image as the origin, and using its length and height as the horizontal and vertical coordinates respectively, to establish the image coordinates of a two-dimensional street view panoramic image, and uniquely representing each pixel by the column number and row number of the pixel; 2) using a spherical panoramic camera model, taking the due north direction as 0°, defining a 360° horizontal azimuth, taking the horizontal direction as 0°, the elevation angle as positive, and the depression angle as negative, and defining a 180° vertical azimuth, to establish the coordinates of a spherical panoramic camera, and uniquely representing each pixel in the image by the horizontal azimuth and the vertical azimuth; 3) taking the camera position as the origin, the east direction as the positive semi-axis of the x-axis, the north direction as the positive semi-axis of the y-axis, and the top as the positive semi-axis of the z-axis to establish a three-dimensional geographic space coordinate system, and uniquely representing the position in space by the coordinates of the three coordinate axes, Secondly, establish the correspondence between the two-dimensional pixel coordinates and the spherical panoramic camera coordinates, establish the correspondence between the panoramic camera coordinates and the three-dimensional geospatial coordinates, construct a projection transformation model of the two-dimensional street view target-three-dimensional geographic entity, and realize the correspondence between the geographic space and the buildings in the street view panoramic image. The steps include: 1) Align the vehicle's forward direction in the street view panoramic image with the yaw angle of the street view panoramic image, and establish a corresponding relationship between the two-dimensional pixel coordinates and the spherical panoramic camera coordinates. The process is: align the vehicle's forward direction in the street view panoramic image with the yaw angle of the street view panoramic image, and establish a corresponding relationship between the two-dimensional pixel coordinates and the spherical panoramic camera coordinates. The specific relationship is expressed by the following formula: h=(90-ρ)H / 180 # , Wherein, W is the length of the street view panoramic image, H is the height of the street view panoramic image, w and h are the horizontal and vertical coordinates of the pixels in the street view panoramic image, and Cyaw is the column number of the vehicle's forward direction in the street view panoramic image. and ρ represent the horizontal and vertical azimuth angles of the pixel in the panoramic camera coordinates, θ represents the yaw angle of the street view panoramic image, 2) Taking the street view panoramic image location as the observation point and the building as the observed point, use the Build Line of Sight tool in ArcMap software to determine the horizontal distance from the panoramic image location to the building at different azimuths and estimate the building height. The specific horizontal distance and height calculation formula is: z=dtanρ+h camera # , Where d represents the azimuth from the panoramic image position to Horizontal distance of the building in the direction, h camera is the camera height, x, y and z are the coordinates of the street view panoramic image pixel coordinates projected into 3D geographic space; (3) For the widespread situation of building occlusion in street view images, a method for eliminating pseudo-height feature points of buildings based on slope mutation test is proposed to identify the real building height feature points in the case of severe occlusion. The following method is used to remove the false height feature points of buildings: 1) Based on the projection conversion model of the two-dimensional street view target-three-dimensional geographic entity, the building is located in the street view panoramic image, and the process is: the street view panoramic image is processed by a preset semantic segmentation model to extract pixels with building types, so as to obtain the accurate outline of the building; the building inflection points in the building footprint data are extracted; the street view panoramic image position is used as the observation point, the building inflection point is used as the observed point, the observation line of sight is constructed, and the horizontal azimuth and horizontal distance of the building inflection point observed from the street view panoramic image are obtained; based on the projection conversion model of the two-dimensional street view target-three-dimensional geographic entity, the horizontal azimuth of each building inflection point is converted into a pixel column number, and the pixel column range of each building is located in the street view panoramic image, 2) For the widespread situation of building occlusion in street view images, a method for eliminating pseudo-height feature points of buildings based on slope mutation test is used to identify the real building height feature points in severe occlusion. The process is as follows: scan column by column within the range of the building to identify the minimum vertical coordinate of the building pixel to form the roof line of the building; calculate the slope of the roof line between the two inflection points of the building and generate a slope change curve. The slope change formula is as follows: Where k is the slope of the roof line, h2-h1 is the image ordinate difference of two adjacent pixels on the roof line, and w2-w1 is the image abscissa difference of two adjacent pixels on the roof line. When the image ordinate difference at the slope change location exceeds 10% of the street view panoramic image height, the roof line of the building is divided into two parts. The roof line with a larger image ordinate value of the pixel is considered to be blocked, and the building roof corner point on the roof line is identified as a pseudo-height feature point. On the contrary, the building roof corner point on the unblocked roof line is identified as a true height feature point. (4) Project its two-dimensional image coordinates into three-dimensional geographic space to estimate the height of the building.

2. The method for estimating the height of urban buildings using street view images under severe occlusion conditions according to claim 1, characterized in that: Step (1) adopts the following method: Firstly, the building footprint data and road data in the building height estimation area were downloaded from the OpenStreetMap website. Secondly, the Generate Points Along Line tool in ArcMap was used to generate street view collection points at intervals of 10m. Finally, a Baidu Street View downloader was written based on Python, and the street view panoramic images at the street view collection points and their camera height and yaw angle were obtained based on the Baidu Map Static Panorama API.

3. The method for estimating the height of urban buildings using street view images under severe occlusion conditions according to claim 1, characterized in that: The building height is estimated using the following method: (1) Extracting the image coordinates of the true height feature points of buildings in street view panoramic images; (2) calculating the 3D geographic space coordinates of all true height feature points of the building based on the projection transformation model of the two-dimensional street view target-three-dimensional geographic entity; (3) Sort the true height feature points of the building from high to low, remove the first 10% and the last 10% of the height feature points, and average the heights of the remaining height feature points to estimate the building height. If a building has less than three height feature points, the average height of all height feature points is used to estimate the building height.

4. The method for estimating the height of urban buildings using street view images under severe occlusion conditions as described in any one of claims 1 to 3, characterized in that: Used to estimate the height of urban buildings using street view images under severe occlusion conditions.

Images