Orthophoto generation method based on panoramic images

Abstract

The present invention discloses a method for generating orthophotos based on panoramic images, which superimposes and fuses the panoramic images and the panoramic depth of field data when the panoramic images are taken, so that each pixel of the panoramic image has a corresponding depth of field value; the pitch angle and azimuth angle of the reference pixel relative to the shooting device are obtained, and the azimuth angle and pitch angle of the object represented by each pixel relative to the camera can be calculated in combination with the 360-degree panoramic characteristics; the geographical location corresponding to the object represented by each pixel is deduced in combination with the aforementioned calculation results and the camera geographic coordinates determined by the positioning device when the panoramic image is taken, to form point cloud data. The point cloud data generated by adjacent panoramas are eliminated and fused according to certain rules, and the results are projected onto a plane to obtain an orthophoto. The orthophotos obtained based on this method have the advantages of high precision and high resolution, and compared with the orthophotos obtained by traditional satellites and aerial photography, the orthophotos obtained based on this method are not affected by obstacles such as tree canopies.

Description

Technical Field

[0001] The invention belongs to the field of computers, and in particular relates to a method for generating orthophoto images based on panoramic images. Background Art

[0002] Panoramic images are a form of image expression that uses professional cameras to capture the image information of the entire scene, and uses software to stitch the images together to show the surrounding environment in a wide-angle form. It is widely used in painting, photography and other fields. Panoramic images have the characteristics of complete information and real scenes. The 360-degree real scene obtained after software processing can give people an immersive three-dimensional sense of space and has strong interactivity. Compared with the three-dimensional animation of the scene, it has the advantages of short production cycle and high cost performance.

[0003] Digital orthophoto map (DOM) is a digital aerial photograph scanned and processed using a digital elevation model. After correcting the projection difference pixel by pixel, the image is mosaicked and clipped according to the map range to generate image data. The current image sources are mainly satellite sensors or high-altitude airborne cameras, which are often expensive to produce and often cannot reflect the ground conditions under the tree canopy in densely treed areas. Summary of the invention

[0004] The purpose of the present invention is to solve the limitation of traditional digital orthophotos that they cannot reflect the details of objects blocked by trees (or other complex high-density obstructions), and to provide an orthophoto generation method based on panoramic images. The panoramic image containing rich scene information and displayed in a wide-angle form is combined with the shooting position of the panoramic image, the reference pixel orientation (the azimuth and pitch angle of the object represented by the central pixel of the panoramic image relative to the camera), and the corresponding depth of field data to calculate the real position of the object represented by each pixel in the panorama in space to form a point cloud, and the point cloud of the ground area is screened to generate an orthophoto at a certain resolution.

[0005] In order to achieve the above technical objectives, the present invention adopts the following technical solutions:

[0006] The orthophoto generation method based on panoramic images specifically includes the following steps:

[0007] Step 1: Generate point cloud data within the panoramic image range based on the panoramic image, depth of field data and camera parameters (longitude and latitude coordinates of the shooting device, pitch angle and azimuth angle of the shooting device during shooting).

[0008] Step 2: Generate point cloud data corresponding to all panoramic images in the target area according to the method in step 1.

[0009] Step 3: Project the point cloud data obtained in step 2 onto a plane, and generate an orthophoto containing projection information and geographic coordinate information of a specified resolution as required.

[0010] The latitude and longitude coordinates of the shooting device can be recorded in the WGS84 coordinate system commonly used by GPS, and its position accuracy will directly affect the accuracy of the final generated result. When shooting panoramic images, the depth of field data corresponding to the panorama can be saved. The more detailed the depth of field data is recorded, the closer the generated point cloud effect is to reality.

[0011] As a further improvement of the present invention, in step 1, the step of generating point cloud data includes:

[0012] Superimpose the pixel value corresponding to the pixel of each row and column number in the panoramic image with the depth data corresponding to the pixel of the corresponding row and column number in the depth data to obtain a panoramic image containing the depth data;

[0013] The real geographic location data of the object represented by each pixel in the panoramic image containing the depth of field data is calculated by combining the coordinates of the shooting device and the azimuth and pitch angle of the shooting device during shooting, and point cloud data including the pixel value of each pixel and the real geographic location data of the object represented by the pixel is obtained. Furthermore, the real geographic location data includes the latitude and longitude position of the object represented by the pixel and the height relative to the position of the shooting device.

[0014] As a further improvement of the present invention, the method of calculating the real geographical location data of objects represented by each pixel in the panoramic image containing depth of field data by combining the coordinates of the shooting device during shooting and the azimuth and pitch angles of the shooting device, includes: selecting a pixel of the panoramic image as a reference pixel, based on the azimuth and pitch angles of the object represented by the reference pixel relative to the camera, through the characteristics of the panoramic 360 degrees, according to the row number deviation and column number deviation of each pixel data and the reference pixel, respectively obtaining the real azimuth and pitch angle of the pixel relative to the shooting position.

[0015] As a further improvement of the present invention, in step 2, the generated point cloud data is eliminated; the eliminated objects are point clouds that are higher than the shooting center and too far from the shooting center. Further, a value within the range of 30 to 100 meters from the shooting center is selected as a threshold, and the point cloud data whose distance from the shooting center exceeds the threshold is eliminated. Further, the effective area of ​​each independent panoramic image is established according to the adjacency relationship between each panoramic image, and the point cloud data outside the effective area of ​​each panoramic image is eliminated.

[0016] Furthermore, point clouds are eliminated based on the principle of nearest retention, and point clouds close to the shooting center of the panoramic image are preferentially retained; when the point clouds of the current panoramic image overlap with those of the adjacent panoramic image, point clouds close to the shooting center of the current panoramic image are preferentially retained.

[0017] As a further improvement of the present invention, in step 3, the point cloud data set obtained in step 2 is selected with a suitable image resolution according to its point cloud density and application requirements to generate an orthophoto containing projection information and geographic coordinate information.

[0018] As a further improvement of the present invention, in step 3, the range of the point cloud set obtained in step 2 is counted, and the number of rows and columns of the final orthophoto is calculated based on the distance represented by the set pixels, raster data is created, and the pixel values ​​of the point cloud pixels falling within the divided grids are assigned to the grids where the point cloud is located to obtain the orthophoto.

[0019] The method of the present invention can generate orthophotos with the help of panoramic data provided by current Internet platforms or collected by professional panoramic data collection equipment. It has low cost and can supplement the image content of areas with problems such as blurred road data and tree occlusion in the orthophotos provided by current platforms, thus assisting users to obtain better image information. For example, when applied to the digitization of road information, it can solve the problem of unknown and incapable road information due to blurred orthophotos or occlusion by trees, and provide users with clear and high-precision road section images. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a flow chart of Embodiment 1 of the present invention;

[0021] Figure 2 This is an example of effective area division for point cloud generation example 1. DETAILED DESCRIPTION

[0022] The technical solution of the present invention is further described in detail below with reference to the accompanying drawings and embodiments:

[0023] Example 1

[0024] This example uses Windows 10 as the operating system environment, Visual Studio 2013 as the development environment, C# as the development language, and the color map panoramic data provided by the Google Maps API as the data source to specifically illustrate the process of the orthophoto generation method based on panoramic images of the present invention.

[0025] like Figure 1 As shown in the figure, the orthophoto generation method based on panoramic images is to establish the orthophoto generation program HDMap.exe based on panoramic images according to the following steps:

[0026] Step 1: Generate color point cloud data within the panoramic range based on the panoramic image, depth of field data and camera parameters (camera coordinates, camera pitch angle, azimuth angle during shooting).

[0027] Step 2: Generate color point cloud data corresponding to all panoramic images in the target area according to the method in step 1, and eliminate and merge the point cloud data according to certain rules to form a complete color point cloud data set covering the target area.

[0028] Step 3: Project the point cloud data obtained in step 2 onto a plane and generate an orthophoto with geographic coordinates of a specified resolution as required.

[0029] After completing and starting HDMap.exe, enter the user configuration interface, set the task path of this task (which should contain the ordered street panorama ID), cache path (used to cache point cloud text, etc.), output image resolution, and the final output path of the image, and click the Generate button to start the orthophoto generation task. When the generation task is completed, the task completion prompt will pop up, and the corresponding orthophoto image can be found in the output directory.

[0030] The specific implementation steps are as follows:

[0031] Step 1 generates point cloud data of the panoramic capture range according to the panoramic image and the geographical location, reference orientation, and corresponding depth of field data when shooting.

[0032] (1) Data acquisition

[0033] The program will obtain the specified panoramic image and the geographical location, reference orientation, and corresponding depth of field data at the time of shooting based on the given panoramic ID for subsequent orthophoto generation.

[0034] (2) Generation of panoramic data with depth of field

[0035] The panoramic image pixels and depth data are traversed row by row and column by column, and the row and column numbers, color values, and depth data of the panoramic image are recorded in the text one by one.

[0036]

[0037] (3) Generation of point cloud data

[0038] Based on the reference orientation of the panoramic image (here, the orientation angle of the center pixel of the panoramic image), the true orientation angle (Heading) of the pixel relative to the shooting position can be calculated based on the row number deviation of each pixel data and the middle pixel through the 360-degree characteristics of the panoramic view. Similarly, the true pitch angle (Pitch) of the pixel can be calculated based on the column number deviation of each pixel and the middle pixel.

[0039] double heading=2.0*(curColRowRGBD.col-curZoomWidth / 2) /

[0040] curZoomWidth*180.0+curPanoInfo.OriginalHeading;

[0041]

[0042] double pitch=1.0*(curZoomHeight / 2-curColRowRGBD.row) / (curZoomHeight)*Math.PI;

[0043] The longitude and latitude positions of each pixel can be calculated based on the pixel azimuth calculated by the above method, the depth of field value fused in the panoramic data (i.e., the distance between the pixel and the photography center), and the longitude and latitude information of the photography center.

[0044] private double EARTH_RADIUS_L=6378137;

[0045] private double EARTH_RADIUS_S=6356752.3142;

[0046] private double EARTH_FLATTENING=1 / 298.2572236;

[0047]

[0048]

[0049]

[0050] Based on the pixel pitch angle calculated by the above method and the depth of field value fused in the panoramic data (i.e., the distance between the pixel and the photography center), the height of each pixel relative to the photography device can be calculated.

[0051] double z=Math.Sin(pitch)*curColRowRGBD.depthValue;

[0052] Through the above method, the latitude and longitude, altitude, and color value of each pixel, that is, point cloud data, can be obtained.

[0053] Step 2: Generate ordered panoramic point cloud data of a road section according to the method of step 1, and remove (crop) the point cloud data according to certain rules, retaining only valid point clouds generated by multiple ordered panoramas, thereby forming a complete point cloud set (stitching) of a road section and storing it in text.

[0054] The rules for point cloud removal include: too far from the center of the photography, higher than the photography equipment, out of the effective range (such as Figure 2 , the effective range is divided according to the perpendicular bisector of the current panorama and the adjacent panorama, and the point cloud generated by the current panoramic image that crosses the effective range is eliminated). In the embodiment, 100m is used as the threshold, and the point cloud outside the range of 100m from the shooting center is eliminated.

[0055]

[0056]

[0057] Step 3: Project the point cloud data obtained in step 2 onto a plane to generate a TIF format image containing geographic information of appropriate resolution, namely, an orthophoto.

[0058] According to the point cloud bounding rectangle obtained in step 2, determine the actual geographic range represented by the final image. According to the actual distance represented by each pixel initially set, calculate the number of rows and columns that the orthophoto should have. Create a TIF image through the C# GDAL class library, and assign the point cloud color value that falls within the divided grid to the grid.

[0059]

[0060]

[0061]

Claims

1. A method for generating orthophoto images based on panoramic images, characterized in that: The steps include: Step 1: Generate point cloud data within the panoramic image range according to the panoramic image, depth of field data and camera parameters; The camera parameters include the latitude and longitude coordinates of the shooting device, the pitch angle and azimuth angle of the shooting device during shooting; The steps to generate point cloud data include: Superimpose the pixel value corresponding to the pixel of each row and column number in the panoramic image with the depth data corresponding to the pixel of the corresponding row and column number in the depth data to obtain a panoramic image containing the depth data; Calculate the real geographical location data of the object represented by each pixel in the panoramic image containing depth data by combining the coordinates of the shooting device and the azimuth and pitch angle of the shooting device during shooting, and obtain point cloud data including the pixel value of each pixel and the real geographical location data of the object represented by the pixel; Calculate the real geographical location data of the object represented by each pixel in the panoramic image containing depth data by combining the coordinates of the shooting device and the azimuth and pitch angle of the shooting device during shooting includes selecting a certain pixel of the panoramic image as a reference pixel, based on the azimuth and pitch angle of the object represented by the reference pixel relative to the camera, and through the characteristics of the 360-degree panorama, respectively obtaining the real azimuth and pitch angle of the pixel relative to the shooting position according to the row number deviation and column number deviation of each pixel data and the reference pixel; Step 2: Generate point cloud data corresponding to all panoramic images in the target area according to the method in step 1; Step 3: Project the point cloud data obtained in step 2 onto a plane to generate an orthophoto with geographic coordinates at a specified resolution.

2. The method for generating orthophotos based on panoramic images according to claim 1, characterized in that: The real geographic location data includes the latitude and longitude positions of the objects represented by the pixels and the height relative to the position of the shooting device.

3. The method for generating orthophotos based on panoramic images according to claim 1, characterized in that: In the step 2, the generated point cloud data is eliminated; the eliminated objects are point clouds that are higher than the shooting center or too far away from the shooting center.

4. The method for generating orthophoto images based on panoramic images according to claim 3, characterized in that: The value within the range of 30~100m from the shooting center is selected as the threshold, and the point cloud data whose distance from the shooting center exceeds the threshold is eliminated.

5. The method for generating orthophotos based on panoramic images according to claim 3 or 4, characterized in that: The effective area of ​​each independent panoramic image is established according to the adjacency relationship between the panoramic images, and the point cloud data outside the effective area of ​​each panoramic image is eliminated.

6. The method for generating orthophoto images based on panoramic images according to claim 5, characterized in that: Point cloud culling is performed based on the principle of nearest retention, with priority given to retaining point clouds close to the shooting center of the panoramic image; when the point clouds of the current panoramic image overlap with those of the adjacent panoramic image, priority is given to retaining the point clouds close to the shooting center of the current panoramic image.

7. The method for generating orthophoto images based on panoramic images according to claim 1, characterized in that: In step 3, the image resolution of the point cloud data set obtained in step 2 is determined according to the point cloud density and application requirements, and an orthophoto containing projection information and geographic coordinate information is generated.

8. The method for generating orthophoto images based on panoramic images according to claim 1, characterized in that: In step 3, the range of the point cloud set obtained in step 2 is counted, and the number of rows and columns of the final orthophoto is calculated according to the distance represented by the set pixels, raster data is created, and the pixel values ​​of the point cloud pixels falling within the divided grid are assigned to the grid where the point cloud is located to obtain the orthophoto.

Images