A method for urban scene reconstruction by integrating ground laser point cloud and panoramic image

Abstract

The present invention discloses a method for urban scene reconstruction by integrating ground laser point cloud and panoramic image. It comprises the following steps: step 1, aerial triangulation processing of ground panoramic image; step 2, registration of panoramic image and ground laser point cloud; step 3, regional network adjustment optimization of panoramic image based on ground laser point cloud constraint; step 4, geometric modeling of combined ground laser point cloud and panoramic image; step 5, texture modeling of three-dimensional mesh. The present invention solves the problem that single-source panoramic image is difficult to achieve high-precision, fine and complete modeling in complex urban scenes in the prior art; it has the advantages of being able to achieve higher-quality three-dimensional reconstruction of complex urban scenes, and completely and finely restore the fine structure of complex areas in large-scale cities.

Description

Technical Field

[0001] The present invention relates to the field of photogrammetry and remote sensing, and in particular to a method for reconstructing an urban scene by integrating ground laser point cloud and panoramic image. Background Art

[0002] With the acceleration of global urbanization, people's demand for detailed three-dimensional information of cities and geographical environments is growing. Detailed three-dimensional models not only play a key role in urban planning and architectural design, but also provide indispensable support in many fields such as traffic flow management, disaster prevention and response, and public safety; especially in response to emergencies such as earthquakes, floods or urban safety threats, high-precision three-dimensional city models can help to make quick and accurate decisions. However, the application of currently widely used drones and close-range photogrammetry technologies in complex building complexes and compact indoor environments is obviously limited or unusable, and it is difficult to cover the target scene completely and without blind spots. Therefore, researching new technologies to improve the completeness and accuracy of three-dimensional reconstruction of urban scenes has become a new demand in the field of complex urban management and planning.

[0003] In modern urban construction, buildings are diverse in form and densely laid out, which makes it difficult for traditional 3D reconstruction technology to be effectively applied in these complex urban scenes. At present, the main data sources for 3D reconstruction of urban ground scenes are ground images and laser point clouds. Panoramic images have the ability to observe in all directions at 360 degrees, which is particularly suitable for capturing complex urban environments and indoor spaces. They effectively provide comprehensive visual information and texture details, greatly enhance the fidelity and visual richness of 3D models, and become an important data source for building fine 3D digital urban models. However, methods that rely solely on panoramic images are difficult to conveniently obtain accurate geometric control information and cannot guarantee the geometric positioning accuracy of panoramic images (i.e., although panoramic images can provide a wide range of viewing angles, due to the lack of effective geometric control information, 3D reconstruction based solely on panoramic images often results in insufficient geometric accuracy and poor detail performance), which greatly limits the application potential of panoramic images in the field of large-scale fine 3D modeling. Although ground laser point cloud data has high-precision geometric information, its coverage is limited and the data density may be insufficient in some cases, especially in urban environments with more occlusions. Data loss is prone to occur. Although the 3D reconstruction method that relies solely on laser point clouds can accurately express the geometric shape of the scene, it cannot effectively obtain the texture information of the scene. At the same time, LiDAR (Light Detection And Ranging) technology has become an ideal control data for large-scale photogrammetry due to its excellent geometric accuracy, and "cloud-controlled photogrammetry" technology effectively uses aerial LiDAR for geometric positioning of oblique aerial images. Despite this, the potential of ground laser point clouds as geometric control information has been ignored, limiting their widespread application in urban fine three-dimensional geographic information perception and modeling.

[0004] The existing publication number is CN110006408A, and the patent name is "LiDAR Data "Cloud Control" Aerial Image Photogrammetry Method", which is mainly used in the process of aerial image photogrammetry to obtain large-scale and large-scale geographic information in areas with complex terrain or difficult to access (such as mountainous areas), using LiDAR (laser radar) point cloud as a geometric reference to replace field control points to achieve high-precision geometric orientation of aerial images and improve the efficiency and automation of aerial image measurement; solve the high cost and low efficiency problems caused by traditional photogrammetry technology relying on a large number of high-precision field control points for geometric orientation of large-scale images; but because it is mainly used to improve the geometric orientation accuracy and measurement efficiency of aerial images and is suitable for wide-area geographic information systems, it cannot be used for the fine restoration of geometric structures and high-quality three-dimensional reconstruction in complex urban scenes (including dense buildings, narrow blocks and highly occluded areas), and cannot achieve detailed restoration of three-dimensional reconstruction of complex urban scenes, and cannot improve the integrity of three-dimensional reconstruction models.

[0005] Therefore, it is necessary to develop a method that can achieve accurate geometric positioning and fine three-dimensional reconstruction of complex urban scenes, restore the details of three-dimensional reconstruction of complex urban scenes, and improve the integrity of the three-dimensional reconstruction model. Summary of the invention

[0006] The purpose of the present invention is to provide a method for reconstructing urban scenes by integrating ground laser point clouds and panoramic images. Through the geometric positioning of panoramic images constrained by ground laser point clouds, the ground laser point clouds and panoramic images are geometrically aligned and jointly reconstructed three-dimensionally, fully integrating the complementarity of laser point clouds and images, achieving higher-quality three-dimensional reconstruction of complex urban scenes, and completely and finely restoring the fine structures of large-scale complex urban scenes.

[0007] In order to achieve the above object, the technical solution of the present invention is: a method for reconstructing urban scenes by integrating ground laser point cloud and panoramic image, characterized in that it comprises the following steps:

[0008] Step 1: aerial triangulation processing of ground panoramic images;

[0009] Step 2: Registration of panoramic image and ground laser point cloud;

[0010] Step 3: Optimize the regional network adjustment of panoramic images based on ground laser point cloud constraints;

[0011] Under the constraint of ground laser point control information, regional network adjustment is performed to optimize the positioning parameters of the panoramic image and obtain the corrected panoramic image.

[0012] Step 4: Combine the ground laser point cloud and the panoramic image to perform geometric modeling;

[0013] Combine the ground laser point cloud with the corrected panoramic image to perform fine geometric modeling and generate a high-precision 3D mesh.

[0014] Step 5: Texture modeling of 3D Mesh.

[0015] In the above technical solution, in step 1, the ground panoramic image aerial triangulation processing method includes:

[0016] The panoramic images collected by the panoramic camera are subjected to POS (Position and Orientation System)-assisted aerial triangulation to calculate the initial orientation parameters (including internal and external orientation elements) and camera distortion parameters of the panoramic image, and generate a sparse feature point cloud.

[0017] In the above technical solution, in step 1, the strict imaging model of the panoramic camera can be expressed by formula (1):

[0018]

[0019] Right now:

[0020]

[0021] In formulas (1) to (2), (x, y, z) are the image point coordinates in the panoramic spherical coordinate system, (X S ,Y S ,Z S ) is the coordinate of the object point in the panoramic spherical coordinate system, (X W ,Y W ,Z W ) are the geodetic coordinates of the object point, a1, a2, a3, b1, b2, b3, c1, c2, c3 and ΔX, ΔY, ΔZ are the coefficients of the spherical coordinate system in the geodetic coordinate system. x ,T y ,T z ] T is the translation vector of the exterior orientation of a lens in the spherical coordinate system;

[0022] Taking the image point coordinates as the observed values ​​and the object coordinates as the unknown parameters, linearize equation (2) to obtain the error equation:

[0023]

[0024] In the formula, x 0 and 0 is the coordinate of the image point calculated by substituting the image exterior orientation elements and the approximate value of the object coordinates into equation (2); and for the posture information provided by the POS system, the positioning constraint v can be obtainedD With attitude angle constraint v angle , as shown in formula (4):

[0025]

[0026] In the formula, (X G ,Y G ,Z G ) is the geodetic coordinate of the object point observed by the POS system; is the shooting attitude angle of the panoramic camera; It is the camera shooting attitude angle observed by the POS system.

[0027] In the above technical solution, in step 2, the registration method of the panoramic image and the ground laser point cloud includes:

[0028] The ground laser point cloud is registered with the sparse feature point cloud generated in step 1 through the iterative nearest point (ICP) algorithm. The panoramic image sparse feature point cloud is rigidly transformed into the ground laser point cloud coordinate system through the ICP algorithm to eliminate the systematic deviation between the panoramic image and the ground laser point cloud, and the exterior orientation elements of the panoramic image and the object coordinates of the sparse feature points are updated, thereby promoting the geometric consistency between the panoramic image sparse feature point cloud and the laser point cloud at the overall level.

[0029] In the above technical solution, step 3, based on step 2, performs regional network adjustment solution under the constraint of ground laser point control information, optimizes the orientation parameters of the panoramic image, and updates the object coordinates of the sparse feature points of the panoramic image until the error in the panoramic image orientation meets the standard.

[0030] In the above technical solution, in step 4, dense matching is used to generate an image geometric model and the laser point cloud data is finely registered using the ICP algorithm to obtain a point cloud geometric model; for the finely reconstructed geometric model M, a model as shown in formula (5) is defined:

[0031] M(x,y,z)=w(x,y,z)·P(x,y,z)+(1-w(x,y,z))·G(x,y,z) (5)

[0032] Wherein, M(x,y,z) is the data point of the final geometric model at the spatial coordinates (x,y,z); P(x,y,z) is the geometric model function of the laser point cloud at the same spatial coordinates (x,y,z), which provides the main geometric structure; G(x,y,z) is the geometric model function of the panoramic image at the same spatial coordinates (x,y,z), which is responsible for providing the geometric structure in the area where the point cloud data is missing; w(x,y,z) is the weight function, which determines the proportion of point cloud reconstruction data P or panoramic image reconstruction data G at a given position (x,y,z);

[0033] The w(x,y,z) function is based on the coverage density calculation of the point cloud. In areas with dense point clouds, w(x,y,z) is close to 1, which means that it relies more on point cloud data. In areas with insufficient point cloud coverage, w(x,y,z) is close to 0, making the model rely more on data reconstructed from panoramic images. The specific calculation formula is:

[0034]

[0035] Among them, ρ max is the maximum value of the point cloud density in all patches, which is used for normalization to ensure that the weight w(x, y, z) varies between 0 and 1; ρ(x, y, z) is the point cloud density, which is calculated as the number of points N on a given patch divided by the area A of the patch:

[0036]

[0037] In the above technical solution, in step 5, the texture modeling method of the three-dimensional Mesh includes:

[0038] Based on panoramic images, texture information is added to the 3D Mesh to improve the visual realism and detail of the model and output the final 3D model.

[0039] In the above technical solution, in step 5, in order to ensure the effectiveness of texture modeling, three aspects of optimization measures are comprehensively considered, including the following parts:

[0040] (1) Priority selection of high-resolution areas;

[0041] When performing texture modeling, priority is given to areas with larger pixel areas on the image; assuming that the resolution function of the image is R(x,y), priority is given to the area with the largest resolution max(R(x,y)) when selecting the texture mapping area;

[0042] (2) Selection of optimal angle patches;

[0043] Based on the principle of photogrammetry, by ensuring that the angle θ between the normal vector n of the facet and the photographic optical axis d is minimized (as shown in formula (8)), the accuracy and naturalness of the texture can be retained to the maximum extent, avoiding visual distortion caused by angle problems.

[0044]

[0045] (3) Prioritize the central area of ​​the image;

[0046] Assume that the center point of the image is (x c ,y c ), the area with the smallest distance from the center point (x, y) is preferentially selected for texture mapping, as shown in formula (9):

[0047]

[0048] The present invention has the following advantages:

[0049] (1) Based on the ground laser point cloud, the high-precision geometric positioning processing of the panoramic image regional network is achieved by registering the sparsely reconstructed point cloud of the panoramic image with the ground laser point cloud, solving the problem of lack of control information in the panoramic image and improving the quality of the panoramic image 3D reconstruction (such as Figure 4 As shown in the left and right figures in the figure, at the corner of the wall, the method of the present invention obviously retains the sharpness of the wall itself and significantly improves the quality of panoramic image 3D reconstruction; while when only relying on panoramic image modeling, the corner of the wall is smoothed to a certain extent, indicating that the quality of panoramic image 3D reconstruction is low);

[0050] (2) It integrates high-precision ground laser point cloud and corrected panoramic image, and performs fine and complete 3D reconstruction by densely matching ground laser point cloud and panoramic image point cloud, repairing spatial gaps caused by incomplete coverage of single-source data. It combines the advantages of ground laser point cloud and panoramic image in 3D reconstruction, and significantly improves the completeness of 3D reconstruction of complex urban scenes. It effectively solves the application limitations of existing methods in large scale (greater than or equal to 1:2000) and complex urban environments (e.g., data collection is difficult (drones are not applicable to such fields); the direct 3D reconstruction effect of data collected by panoramic cameras is poor).

[0051] (3) Based on the ground laser point cloud acquired by the backpack acquisition system, the present invention performs precise geometric positioning processing on the panoramic images acquired at the same time, and combines the ground laser point cloud and the corrected panoramic image to perform high-precision reconstruction of complex scenes; the method of the present invention fully utilizes the excellent plane and elevation accuracy of the laser point cloud to correct the geometric errors of the regional network panoramic image; in the three-dimensional reconstruction process, the complementary characteristics of the ground laser point cloud and the panoramic image are fully considered to achieve higher-quality three-dimensional reconstruction, and completely and finely restore the fine structure of large-scale complex urban areas; solves the problem that single-source panoramic images are difficult to achieve high-precision, fine and complete modeling in complex urban scenes;

[0052] (4) The present invention has wide applicability. The present invention is not only suitable for the three-dimensional reconstruction of complex urban ground scenes, but also suitable for the reconstruction of scenes such as the interior of cultural relics and ruins, the interior of urban buildings, etc., that is, the present invention is suitable for the reconstruction of interior and exterior scenes of complex buildings or natural landscapes that cannot be used by any drone and the three-dimensional reconstruction effect of the image-collected data is poor. For example, the reconstruction can be performed by combining pure handheld laser radar and images (not limited to the panoramic images of the present application), thereby achieving refined reconstruction of the three-dimensional model and ensuring the integrity of the reconstructed three-dimensional model. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is the overall technical flow chart of the present invention;

[0054] Figure 2 A sparse feature point cloud image of a panoramic image of a building in a city in an embodiment of the present invention;

[0055] Figure 3 It is an overall three-dimensional model diagram of a building in a certain city in an embodiment of the present invention;

[0056] Figure 4 A comparison diagram of the three-dimensional model in an embodiment of the present invention and the wall modeling effect based only on panoramic images;

[0057] Figure 5 A comparison diagram of the three-dimensional model in the embodiment of the present invention and the modeling effect of the stone tablet based only on the panoramic image;

[0058] Figure 6 This is a comparison diagram of the three-dimensional model in the embodiment of the present invention and the courtyard modeling effect based only on panoramic images. DETAILED DESCRIPTION

[0059] The following is a detailed description of the implementation of the present invention in conjunction with the accompanying drawings, but they do not constitute a limitation of the present invention and are only given as examples. At the same time, the advantages of the present invention are made clearer and easier to understand through the description.

[0060] The technical solution provided by the present invention is: using the panoramic image and laser point cloud collected by the ground backpack as the basic data source, using the laser point cloud as the geometric reference, correcting the geometric error of the panoramic image, and jointly registering the laser point cloud and the panoramic image to perform high-precision 3D reconstruction. This method is suitable for 3D reconstruction of urban ground scenes where drones cannot be used, and at the same time improves the quality of 3D reconstruction of panoramic images; it overcomes the problem that the existing technology relies on drone data collection for 3D reconstruction and cannot be applied to 3D reconstruction of complex urban ground scenes (areas where drones cannot effectively collect or even cannot collect data, and drones cannot be used in densely built urban areas), and at the same time overcomes the problem of low quality of 3D reconstruction of panoramic images.

[0061] The present invention provides a 3D reconstruction method for complex urban scenes by integrating ground laser point cloud and panoramic image. The method integrates ground laser point cloud and panoramic image, and combines ground laser point cloud with optimized panoramic image through the complementarity of geometric information and texture information of ground laser point cloud and panoramic image, so as to perform geometric modeling with higher accuracy and completeness, and generate high-precision 3D mesh; on this basis, texture modeling of 3D mesh is performed to add rich texture details to the model, thereby improving the complete visual effect and detail expression of the final reconstructed model, ensuring the high fidelity and authenticity of the 3D reconstruction result, which can help to make quick and accurate decisions, and has important significance for urban planning, architectural design and cultural relics protection and other fields. Significance: The method of the present invention can improve the positioning accuracy of panoramic images in scenes such as complex buildings and streets through the precise geometric constraints of ground laser point clouds, and ultimately generate a more detailed and complete three-dimensional model, especially in urban scene areas with complex geometric structures, and can effectively restore building details and environmental features; it solves the problem that the existing technology is mainly used to improve the geometric orientation accuracy and measurement efficiency of aerial images and is suitable for wide-area geographic information systems, but cannot be used for the fine restoration of geometric structures and high-quality three-dimensional reconstruction in complex urban scenes (including dense buildings, narrow blocks and highly occluded areas), cannot achieve detailed restoration of three-dimensional reconstruction of complex urban scenes, and cannot improve the integrity of the three-dimensional reconstruction model;

[0062] At the same time, the present invention realizes precise geometric processing of the strict imaging model of the panoramic camera to improve the geometric positioning accuracy of the panoramic image; the present invention optimizes the positioning parameters of the panoramic image by accurately calculating the geometric relationship between the coordinates of the image points in the panoramic image and the position of the object in space. This processing method significantly improves the geometric positioning accuracy of the panoramic image in complex urban environments (including densely populated buildings, narrow blocks and highly obstructed areas), ensuring a high degree of consistency between the image and the ground laser point cloud in subsequent alignment; it solves the problem of lack of effective geometric control information in panoramic images, and the problem of insufficient geometric accuracy and poor detail performance when using panoramic images for three-dimensional reconstruction.

[0063] The present invention is mainly realized by the following technical solutions: a method for urban scene reconstruction by integrating ground laser point cloud and panoramic image, which takes ground laser point cloud and panoramic image data as input data, uses laser point cloud as geometric reference, corrects the geometric error of panoramic image, and combines the registered laser point cloud and panoramic image to perform high-precision 3D reconstruction. Its core process includes the following steps:

[0064] Step 1: Aerial triangulation of ground panoramic images. Perform POS (Position and Orientation System)-assisted aerial triangulation on the collected panoramic images, calculate the initial orientation parameters (including internal and external orientation elements) and camera distortion parameters of the panoramic images, and generate sparse feature point clouds;

[0065] Step 2: Registration of panoramic image and ground laser point cloud. The ground laser point cloud is registered with the sparse feature point cloud generated in step 1 through the iterative closest point (ICP) algorithm.

[0066] Step 3, based on the ground laser point cloud constraint panoramic image regional network adjustment optimization, update the panoramic image geometric positioning parameters, and obtain the panoramic image after geometric registration. Under the constraint of the ground laser point control information, the regional network adjustment solution is performed to optimize the positioning parameters of the panoramic image; the present invention uses the control information of the ground laser point cloud to perform regional network adjustment optimization on the panoramic image. This process makes the geometric positioning of the panoramic image more accurate, especially in complex urban scenes, and can significantly improve the overall accuracy and detail performance of the three-dimensional model; that is, the method uses the ground laser point cloud as the geometric reference to correct the panoramic image, and integrates the two to achieve higher-quality three-dimensional reconstruction, and completely and finely restores the fine structure of the complex urban environment area of ​​the large-scale city (greater than or equal to 1:2000 scale); the above-mentioned large-scale and complex urban environment, typical scenes such as commercial streets, are affected by the interlaced overpasses or surrounding high-rise buildings. Such areas are not suitable for and generally do not allow drones to fly. Generally, it refers to the lack of drones for image acquisition, and the camera shooting effect is often inefficient. Generally, panoramic cameras are used for street scene shooting;

[0067] Step 4: Combine the ground laser point cloud and the panoramic image for geometric modeling and output a high-quality 3D model. Combine the ground laser point cloud with the corrected panoramic image to perform fine geometric modeling and generate a high-precision 3D mesh.

[0068] Step 5: Texture modeling of the 3D Mesh. Based on the corrected panoramic image, texture information is added to the 3D Mesh to improve the visual realism and detail expression of the model, and the final 3D model is output.

[0069] In step 1, the panoramic image with POS information is processed using POS-assisted aerial triangulation technology. By inputting the panoramic image and its POS information, aerial triangulation is performed to calculate the initial orientation parameters of the image (including internal and external orientation elements) and camera distortion parameters, and generate a sparse feature point cloud. It should be noted that the aerial triangulation results need to be consistent with the ground laser point cloud coordinate system, and the automatic connection points need to be embedded in it for subsequent registration with the ground laser point cloud.

[0070] The strict imaging model of the panoramic camera can be expressed by equation (1):

[0071]

[0072] Right now:

[0073]

[0074] In formulas (1) to (2), (x, y, z) are the image point coordinates in the panoramic spherical coordinate system, (X S ,Y S ,Z S ) is the coordinate of the object point in the panoramic spherical coordinate system, (X W ,Y W ,Z W ) are the geodetic coordinates of the object point, a1, a2, a3, b1, b2, b3, c1, c2, c3 and ΔX, ΔY, ΔZ are the coefficients of the exterior orientation elements of the spherical coordinate system in the geodetic coordinate system. [T x ,T y ,T z ] T is the translation vector of the exterior orientation of a lens in the spherical coordinate system.

[0075] Taking the image point coordinates as the observed values ​​and the object coordinates as the unknown parameters, linearize equation (2) to obtain the error equation:

[0076]

[0077] In the formula, x 0 and 0 is the coordinate of the image point calculated by substituting the image exterior orientation elements and the object coordinate approximation into equation (2). For the position information provided by the POS system, the positioning constraint v can be obtained. D With attitude angle constraint v angle , as shown in formula (4).

[0078]

[0079] In the formula, (X G ,Y G ,ZG ) is the geodetic coordinate of the object point observed by the POS system; is the shooting attitude angle of the panoramic camera; is the shooting attitude angle of the camera observed by the POS system.

[0080] In step 2, through the ICP algorithm, the sparse feature point cloud of the panoramic image is globally rigidly transformed to the ground lidar point cloud coordinate system, eliminating the systematic deviation between the image and the lidar point cloud, and updating the exterior orientation elements of the image and the object space coordinates of the sparse feature points, thereby promoting the geometric consistency between the sparse feature point cloud of the image and the ground lidar point cloud at the global level. The ICP algorithm adopts the principle of the closest point-to-plane distance. For each sparse feature point of the panoramic image, by finding the nearest point set in the lidar point cloud and fitting to obtain the spatial plane, the corresponding "control point" of this point in the lidar point cloud is the foot of the perpendicular from this point to the fitted plane.

[0081] Step 3, under the constraint of the ground lidar control information, perform block adjustment calculation, optimize the orientation parameters of the panoramic image, and update the object space coordinates of the sparse feature points of the image until the medium error of image orientation meets the standard. On the basis of the registration in step 2, obtain the control points from the ground lidar point cloud according to the nearest neighbor principle, and incorporate them as constraint conditions into the self-calibration block adjustment to optimize the camera distortion parameters, the interior orientation elements and the exterior orientation elements of the image. This step can suppress the local deformation and distortion caused by factors such as camera distortion and inconsistent image connection strength at the local level. The change amount Δσ of the medium error of image orientation should satisfy Δσ < T (the threshold T is generally 0.001 pixel).

[0082] In step 4, perform dense matching on the panoramic image optimized by block adjustment to obtain a dense point cloud. Specifically, first use the SIFT (Scale-Invariant Feature Transform) algorithm to extract the feature points in the panoramic image, and perform dense matching on these feature points through a matching algorithm to generate an image geometric model. Subsequently, perform fine registration on the lidar point cloud data using the ICP algorithm to obtain a point cloud geometric model.

[0083] For the geometric model M of the fine reconstruction, a model as shown in Equation (5) can be defined:

[0084] M(x, y, z) = w(x, y, z) · P(x, y, z) + (1 - w(x, y, z)) · G(x, y, z) (5)

[0085] Among them, M(x,y,z) is the data point of the final geometric model at the spatial coordinates (x,y,z); P(x,y,z) is the geometric model function of the laser point cloud at the same spatial coordinates (x,y,z), which provides the main geometric structure; G(x,y,z) is the geometric model function of the panoramic image at the same spatial coordinates (x,y,z), which is responsible for providing the geometric structure in the area where the point cloud data is missing; w(x,y,z) is the weight function, which determines the proportion of point cloud reconstruction data P or image reconstruction data G used at a given position (x,y,z).

[0086] w(x,y,z) is calculated based on the coverage density of the point cloud. In areas with dense point clouds, w(x,y,z) is close to 1, which means that it relies more on point cloud data. In areas with insufficient point cloud coverage, w(x,y,z) is close to 0, making the model rely more on data reconstructed from panoramic images. The specific calculation formula is:

[0087]

[0088] Among them, ρ max is the maximum value of the point cloud density in all patches, which is used for normalization to ensure that the weight w(x, y, z) varies between 0 and 1; ρ(x, y, z) is the point cloud density, which is calculated as the number of points N on a given patch divided by the area A of the patch:

[0089]

[0090] In step 5, three optimization measures are considered to ensure the visual realism and detail of the model, including the following:

[0091] (1) Priority selection of high-resolution areas

[0092] When doing texture modeling, give priority to areas with larger pixel areas on the image. These areas contain more pixel details, thus ensuring high resolution and clarity of the texture. For example, on the facade of a building, select those areas such as windows and door frames captured at high resolution for texture mapping. This approach can avoid blurring on the model surface and make the final generated 3D model more detailed and realistic. Mathematically, assuming that the resolution function of the image is R(x,y), when selecting the texture mapping area, give priority to the area with the largest resolution max(R(x,y)).

[0093] (2) Optimal Angle Patch Selection

[0094] In order to reduce the deformation and distortion of textures caused by large angles, the patch with the smallest angle with the photographic optical axis is selected for texture mapping. For example, when modeling the texture of a statue, we will give priority to mapping the image areas that are almost facing the camera. This selection is based on the principle of photogrammetry. By ensuring that the angle θ between the normal vector n of the patch and the photographic optical axis d is minimized (as shown in formula (9)), the accuracy and naturalness of the texture can be retained to the maximum extent, avoiding visual distortion caused by angle problems.

[0095]

[0096] (3) Prioritize the image center area

[0097] In order to improve the accuracy of texture mapping, try to select an area close to the center of the image for texture mapping. The optical distortion in the center of the image is minimal and can provide the most realistic texture information. For example, when processing panoramic images of urban street scenes, priority is given to using building facades located in the center of the image and avoiding using edge areas. This method ensures the accuracy and consistency of the texture, reduces image distortion caused by lens distortion, and thus improves the visual effect of the entire model.

[0098] Assume that the center point of the image is (x c ,y c ), we select the area with the smallest distance from the center point (x, y) for texture mapping:

[0099]

[0100] Example

[0101] The present invention is now described in detail by taking the present invention being tried in a certain city to reconstruct a scene of a certain building as an example, which also has a guiding role in applying the present invention to scene reconstruction in other cities.

[0102] This example selects a building in a certain city as the modeling object and uses a ground backpack device to collect data. During the collection process, five cameras were used to take 2048×2048 resolution photos, totaling 1891 photos, and the ground resolution of each photo was 0.0025 meters. In addition, 15 ground laser point cloud files were collected, with a point cloud density of 10,000 points per square meter. The panoramic image and ground laser point cloud data are the input data of this example. The process of generating a fine 3D model is as shown in the attached figure. Figure 1 As shown, the following steps are included:

[0103] Step 1: Perform POS-assisted aerial triangulation on the 1891 panoramic images collected from 5 lenses, calculate the initial orientation parameters (including internal and external orientation elements) and camera distortion parameters of the images, and generate a sparse feature point cloud, as shown in the attached figure. Figure 2 As shown;

[0104] Step 2: Use the ICP algorithm to align the 15 collected ground laser point clouds with the panoramic image sparse feature point cloud generated in step 1;

[0105] Step 3: Under the constraint of ground laser point control information, perform regional block adjustment to optimize the positioning parameters of 1891 panoramic images;

[0106] Step 4: Combine the ground laser point cloud with the 1,891 corrected panoramic images to perform fine geometric modeling and generate a high-precision 3D mesh.

[0107] Step 5: Texture modeling of the 3D Mesh. Based on the 1,891 corrected panoramic images, texture information is added to the 3D Mesh to improve the visual realism and detail expression of the model, and the final 3D model is output.

[0108] The final output of the overall three-dimensional model is as follows: Figure 3 As shown, attached Figure 4 , 5 6 is a local comparison diagram of the model output by the method of the present invention and the three-dimensional reconstruction based only on the panoramic image.

[0109] Figure 4 The left figure in the figure is a model effect diagram output by the method of the present invention. Figure 4 The right picture in the figure is the rendering of wall modeling based only on panoramic images;

[0110] Figure 5 The left figure in the figure is a model effect diagram output by the method of the present invention. Figure 5 The right picture is the effect of modeling the stone tablet based only on the panoramic image;

[0111] Figure 6 The left figure in the figure is a model effect diagram output by the method of the present invention. Figure 6 The right picture in the figure is the effect of courtyard modeling based only on panoramic images;

[0112] contrast Figure 4 , Figure 5 , Figure 6 It is found that compared with the existing three-dimensional reconstruction of panoramic images, the method proposed in the present invention performs better in terms of details, especially in corners and shadows (such as Figure 4 The present invention retains more details, the boundary lines are clearer, and the overall surface texture is smoother, showing a higher reconstruction quality. At the same time, for typical objects such as stone tablets, the present invention can achieve scene reconstruction more completely, while the method based only on panoramic images ultimately fails to successfully reconstruct (such as Figure 5Finally, in terms of the reconstruction effect of the front courtyard, whether it is the reconstruction effect of the ground or the windows and walls, the method of the present invention is visually much better than the comparison method (such as Figure 6 (Images of the courtyard modeling in the middle left and right images).

[0113] Complex urban environment mainly refers to the inability of conventional drones to perform close-up 3D reconstruction or poor image reconstruction due to environmental restrictions. The walls of a certain building, the walls and windows of the stone tablet and the courtyard behind the stone tablet in this embodiment are all relatively complex urban situations. Compared with the typical square high-rise buildings, the reconstruction difficulty of a certain building in this embodiment is much higher, and the objects are easily too dense, which is very simple for backpack collection, but difficult to implement with currently popular data collection methods such as drones. This embodiment achieves high-quality 3D reconstruction of a certain building through the method of the present invention (such as retaining more details in the corners and shadows, clearer boundary lines, and more complete scene reconstruction of typical objects such as the walls and windows of the stone tablet and the courtyard behind the stone tablet), thereby achieving high-quality 3D reconstruction of a complex urban environment.

[0114] The embodiment of the present invention is in a certain building in a certain city. The embodiment adopts the method of the present invention to effectively reconstruct the detailed texture of the building, such as the effective re-addition of discrete small objects in the corners and courtyards, and realizes high-quality three-dimensional reconstruction of a large scale;

[0115] It can be seen that the embodiment achieves high-quality three-dimensional reconstruction of a large-scale complex urban environment through the method of the present invention.

[0116] The present invention mainly solves the problem that it is difficult to achieve high-precision, fine and complete modeling of single-source panoramic images in complex urban scenarios. The present invention proposes a method for accurately geometrically positioning the panoramic images collected at the same time based on the ground laser point cloud obtained by the backpack acquisition system, and combining the ground laser point cloud and the corrected panoramic image to reconstruct the complex scene with high precision. The laser point cloud is used to correct the geometric errors of the regional network panoramic image, and the complementary characteristics of the ground laser point cloud and the panoramic image are fully considered to achieve higher-quality three-dimensional reconstruction of complex urban scenes, and complete and fine restoration of the fine structure of large-scale complex urban areas. High-precision geometric positioning of panoramic images based on ground laser point clouds is the key technology of the present invention. The above-mentioned complex urban scenes refer to urban environments with high-density and diverse buildings and infrastructures. These environments usually include high-rise buildings, narrow streets, complex road networks, overpasses, trees, wires and other different types of objects. Because the objects in the scene have different shapes, complex layouts and serious occlusion, higher requirements are placed on the acquisition and processing of sensor data. In such a complex scene, it may be difficult for drones to effectively obtain complete and accurate three-dimensional data due to factors such as limited viewing angles, flight altitude restrictions and signal occlusion. The method of combining ground laser point cloud and panoramic imaging can more comprehensively capture the details in complex urban environments and achieve high-quality three-dimensional reconstruction of complex urban scenes.

[0117] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

[0118] Other parts not described belong to the prior art.

Claims

1. A method for urban scene reconstruction by integrating ground laser point cloud and panoramic image, characterized in that: The following steps are involved: Step 1: aerial triangulation processing of ground panoramic images; Step 2: Registration of the panoramic image and the ground laser point cloud. The registration method of the panoramic image and the ground laser point cloud includes: The ground laser point cloud is registered with the sparse feature point cloud generated in step 1 through the iterative nearest neighbor point ICP algorithm; the panoramic image sparse feature point cloud is rigidly transformed into the ground laser point cloud coordinate system through the ICP algorithm to eliminate the systematic deviation between the panoramic image and the ground laser point cloud, and the exterior orientation elements of the panoramic image and the object coordinates of the sparse feature points are updated, thereby promoting the geometric consistency between the panoramic image sparse feature point cloud and the laser point cloud at the overall level; Step 3: Optimize the regional network adjustment of panoramic images based on ground laser point cloud constraints; Step 3: Based on step 2, under the constraint of the ground laser point control information, perform regional network adjustment solution, optimize the orientation parameters of the panoramic image, and update the object coordinates of the sparse feature points of the panoramic image until the error in the panoramic image orientation meets the standard; Under the constraint of ground laser point control information, regional network adjustment is performed to optimize the positioning parameters of the panoramic image and obtain the corrected panoramic image. Step 4: Combine the ground laser point cloud and the panoramic image to perform geometric modeling; Combine the ground laser point cloud with the rectified panoramic image to perform detailed geometric modeling and generate a high-precision 3D mesh; Step 5: Texture modeling of 3D mesh.

2. The urban scene reconstruction method of integrating ground laser point cloud and panoramic image according to claim 1 is characterized in that: In step 1, the ground panoramic image aerial triangulation processing method includes: POS-assisted aerial triangulation is performed on the panoramic images collected by the panoramic camera to calculate the initial orientation parameters and camera distortion parameters of the panoramic image and generate a sparse feature point cloud.

3. The urban scene reconstruction method of integrating ground laser point cloud and panoramic image according to claim 2 is characterized in that: In step 1, the strict imaging model of the panoramic camera is expressed as formula (1): Right now: In formulas (1) to (2), (x, y, z) are the image point coordinates in the panoramic spherical coordinate system, (X S ,Y S ,Z S ) is the coordinate of the object point in the panoramic spherical coordinate system, (X W ,Y W ,Z W ) are the geodetic coordinates of the object point, a1, a2, a3, b1, b2, b3, c1, c2, c3 and ΔX, ΔY, ΔZ are the coefficients of the exterior orientation elements of the spherical coordinate system in the geodetic coordinate system; [T x ,T y ,T z ] T is the translation vector of the exterior orientation of a lens in the spherical coordinate system; Taking the image point coordinates as the observed values ​​and the object coordinates as the unknown parameters, linearize equation (2) to obtain the error equation: In the formula, x 0 and 0 is the coordinate of the image point calculated by substituting the image exterior orientation elements and the approximate value of the object coordinates into equation (2); and for the posture information provided by the POS system, the positioning constraint v is obtained D With attitude angle constraint v angle , as shown in formula (4): In the formula, (X G ,Y G ,Z G ) is the geodetic coordinate of the object point observed by the POS system; is the shooting attitude angle of the panoramic camera; It is the camera shooting attitude angle observed by the POS system.

4. The urban scene reconstruction method of fusing ground laser point cloud and panoramic image according to claim 1 is characterized in that: In step 4, dense matching is used to generate the image geometric model and the laser point cloud data is finely registered using the ICP algorithm to obtain the point cloud geometric model; for the finely reconstructed geometric model M, the model shown in formula (5) is defined: M(x,y,z)=w(x,y,z)·P(x,y,z)+(1-w(x,y,z))·G(x,y,z) (5) Wherein, M(x,y,z) is the data point of the final geometric model at the spatial coordinates (x,y,z); P(x,y,z) is the geometric model function of the laser point cloud at the same spatial coordinates (x,y,z), which provides the main geometric structure; G(x,y,z) is the geometric model function of the panoramic image at the same spatial coordinates (x,y,z), which is responsible for providing the geometric structure in the area where the point cloud data is missing; w(x,y,z) is the weight function, which determines the proportion of point cloud reconstruction data P or panoramic image reconstruction data G at a given position (x,y,z); The w(x,y,z) function is based on the coverage density calculation of the point cloud. The specific calculation formula is: Among them, ρ max is the maximum value of the point cloud density in all patches, which is used for normalization to ensure that the weight w(x, y, z) varies between 0 and 1; ρ(x, y, z) is the point cloud density, which is calculated as the number of points N on a given patch divided by the area A of the patch:

5. The urban scene reconstruction method of fusing ground laser point cloud and panoramic image according to claim 4 is characterized in that: In step 5, the texture modeling method of the three-dimensional mesh includes: Based on panoramic images, texture information is added to the 3D mesh to improve the visual realism and detail of the model, and the final 3D model is output.

6. The urban scene reconstruction method of fusing ground laser point cloud and panoramic image according to claim 5 is characterized in that: In step 5, in order to ensure the effectiveness of texture modeling, three optimization measures are comprehensively considered, including the following parts: (1) Priority selection of high-resolution areas; When performing texture modeling, priority is given to areas with large pixel areas on the image; assuming that the resolution function of the image is R(x,y), priority is given to the area with the largest resolution max(R(x,y)) when selecting the texture mapping area; (2) Selection of optimal angle patches; Based on the principle of photogrammetry, the angle θ between the normal vector n of the facet and the photographic optical axis d is ensured to be the smallest. The specific calculation method is shown in formula (8): (3) Prioritize the central area of ​​the image; Assume that the center point of the image is (x c ,y c ), the area with the smallest distance from the center point (x, y) is preferentially selected for texture mapping, as shown in formula (9):

Images