Three-dimensional reconstruction method for splicing projection plane measurement and multi-view panorama

Abstract

According to the projection surface measurement splicing multi-view panorama three-dimensional reconstruction method, a seamless multi-view panorama image two-dimensional reproduction form is obtained through parallel splicing, the multi-view panorama image generated through two-dimensional parallel splicing is a gallery-shaped two-dimensional image, scenery information on the two sides of a street is completely covered, continuous browsing of a landscape by a user is supported, and the user experience is improved. And the method has the advantages of low data volume, simple data structure and convenient browsing mode. According to the multi-section plane axis projection tunnel image splicing method, the visual impression requirement of people on a multi-view panorama is met, measurement information is introduced, and the purpose of the multi-view panorama is further widened; and 2, a three-dimensional scanning projection surface full-streetscape splicing method: a measured value projection facade is constructed based on a scene model, streetscape projection data is resampled, three-dimensional projection panoramic streetscape splicing with good expression of measurement information of images is realized, the panoramic image three-dimensional reconstruction precision and efficiency are high, and the method can be used for panoramic photogrammetry calculation of accurate measurement.

Description

Technical Field

[0001] This application relates to a multi-view panoramic Figure 3 dimensional reconstruction method, particularly a method for projecting surface measurement and stitching multi-viewpoint panorama Figure 3 dimensional reconstruction method, which belongs to panoramic Figure 3 The field of maintenance and reconstruction technology. Background Art

[0002] In recent years, computer vision technology has rapidly advanced, and the demand for landscape reconstruction has become more urgent across various fields. Providing users with clear, data-intensive, and easy-to-understand modeling scenarios has been a research and development hotspot. With the massive construction of infrastructure, maintenance and monitoring are enormous undertakings. Timely disaster predictions for roads, bridges, and tunnels are crucial to ensuring the timely and efficient transportation of personnel and supplies. Existing technologies, when processing data from disparate sources, are unable to generate continuously navigable multi-viewpoint panoramas, nor can they accurately and timely locate target information, hindering future repairs and maintenance.

[0003] In addition to being used for disaster prediction in tunnels, bridges, and roads, virtual reality applications are increasingly demanding. Street view is a prime example. Current street view image stitching technologies can be broadly divided into two categories. The first is 360° panoramic images, which are projected using spherical, cubic, or cylindrical projections. Users can view the streetscape from the geometric center of the sphere or cylinder in 360°. However, 360° panoramic images make it difficult for users to determine the shape and position of textures. Furthermore, transitions between stations in this type of street view are not smooth, significantly impacting the user's immersive experience. 360° panoramic images are derived from images captured by a camera at the same viewpoint, while multi-view panoramic images are derived from images captured from different viewpoints. Currently, multi-view panoramic images are derived from a sequence of images or videos captured by a digital camera mounted on a mobile platform as it moves along the street. The camera's principal optical axis is perpendicular to the platform's trajectory, and the normal vectors corresponding to the imaging planes of each frame are roughly coplanar. It is convenient and efficient to collect data using a panoramic gimbal collection vehicle. Then, this method can be used to obtain multi-viewpoint stitching of street scenes in a short time, which will have a better application market in the future.

[0004] However, in the development of panoramic images, many people prioritize the visual effects of images in computer vision while neglecting the information provided by surveying and mapping. During field work, photographs can capture local information about structural defects or damage, but images lack measurement capabilities, so the extent of the damage cannot be determined. Measurements must be made manually or with other equipment, increasing the workload in the field. Surveying and mapping fundamentally provides location-based services. Whether traditional surveying or modern photogrammetry, these lack vitality without a relationship to location. Maps produced in traditional surveying and mapping use images to provide a mathematical relationship with the real world. The Gauss-Krüger projection is based on a rigorous mathematical model, allowing the longitude and latitude of a point on a map to accurately identify a point in a three-dimensional scene. This provides significant convenience in real life, and this method is still used today to explore the relationship between maps and the real world. By examining panoramic images from this perspective, exploring a new approach beyond visual expression and incorporating mathematical information into panoramas, we can create a stronger connection between panoramas and the real three-dimensional world, broadening the prospects for their development.

[0005] Multi-view panorama of existing technology Figure 3 The problems that need to be solved in the maintenance and reconstruction and the key technical difficulties of this application include:

[0006] (1) The visual effect of panoramic images in computer vision of existing technologies ignores the information of surveying and mapping. In the field process, local information of structures with defects or diseases such as tunnels can be obtained by taking pictures, but the pictures have no measurement capabilities, so the scope of the disease cannot be obtained. It needs to be measured manually or completed by other equipment and instruments, which increases the workload of field work. The most basic thing about surveying and mapping is to provide location services. Whether it is traditional surveying and mapping or today's photogrammetry, it lacks vitality if it does not have a relationship with location. The maps produced in traditional surveying and mapping use images to provide mathematical relationships with the real world. The Gauss Krüger projection is based on a strict mathematical model, so the longitude and latitude of a point on the map can accurately find a point in the three-dimensional scene, but multi-viewpoint panorama Figure 3 3D reconstruction does not seek the relationship between the map and the real world, does not allow the panorama to carry mathematical information, and the panorama has a weak connection with the real three-dimensional world. It lacks strict mathematical operations, cannot guarantee seamless splicing between images and strict mathematical measurement relationships, and cannot realize photogrammetric calculations based on panoramas.

[0007] (2) The existing image stitching technology lies in the processing of images. When obtaining pictures, it is necessary to image both sides of the landscape separately, which results in data redundancy. In addition, the images need to be processed in advance, such as color correction, and then the matching algorithm is used for image stitching. The whole process requires a long period of preparation in the early stage. In the later stage, various constraints are added to the image matching algorithm based on different scenes, and the processing results are also difficult to be satisfactory. There is a lack of reconstruction of two-dimensional scenes through rigorous mathematical transformation. The data processing work in the early stage of the existing technology is cumbersome. The measurement value projection surface is not selected and established. There is a lack of establishing a strict mathematical relationship between different projections. There is no determination of the mathematical relationship from the three-dimensional scene to the projection surface. There is no design of different projection algorithms according to different scenes. It is impossible to obtain a multi-viewpoint stitching that can be seamlessly stitched. There is a lack of a mathematical relationship model from the projection model to the stitched image. There is a lack of a mapped tunnel three-dimensional mesh or a successfully stitched spherical panorama. It is impossible to resample according to the corresponding texture value of the index point. Multi-viewpoint panorama Figure 3 The 3D reconstruction accuracy is poor and the efficiency is low, so it cannot be used for panoramic photogrammetry calculations for precise measurement.

[0008] (3) The existing texture resampling technology does not create a seamless image, does not determine the mathematical relationship between the three-dimensional scene and the projection surface, and does not determine the mathematical relationship between the projection surface and the stitched image. It is impossible to reconstruct a multi-viewpoint panoramic image from a 360° street view image. Given that the texture of tunnel images is weak and the brightness inside the tunnel is low, the traditional image-based matching algorithm is not suitable for image stitching of tunnel scenes. The existing technology lacks a projection surface stitching method based on rigorous mathematical derivation of measurement values, and cannot guarantee seamless stitching between tunnel stations, which brings many difficulties to actual engineering applications. The texture of tunnel stitching images is similar, and it is impossible to effectively use the existing image matching algorithm for image stitching. Tunnels are not strictly axial shapes. In real life, tunnels are often undulating and appear as curved polygons with the axis as a curve. There is no good method for two-dimensional flattening of three-dimensional tunnel scenes. The existing image stitching method based on tunnel images has weak tunnel texture information, few feature points, poor effect, lack of tunnel mesh and texture information, and generates a seamless tunnel stitching through processes such as measurement value projection surface determination, coordinate transformation, and texture resampling, but cannot achieve seamless stitching based on strict mathematical relationships. Summary of the Invention

[0009] Given the weak texture of tunnel images and the low brightness inside tunnels, traditional image-based matching algorithms are not suitable for image stitching in tunnel scenes. This application proposes a stitching method based on rigorous mathematical derivation of measured value projection surfaces. The entire process is based on strict mathematical operations, ultimately ensuring seamless stitching between tunnel stations, which has important practical significance in practical engineering applications. Due to the similar textures of tunnel stitching images, existing image matching algorithms cannot effectively stitch images. Tunnels are not strictly axial shapes. In real life, tunnels often have ups and downs and appear as curved polygons with a curved axis. Therefore, a two-dimensional flattening method for three-dimensional tunnel scenes is established. Using the tunnel's mesh and texture information, a seamless tunnel stitching is generated through processes such as measurement value projection surface determination, coordinate transformation, and texture resampling. For street scenes, this application uses a 360-degree rectangular panorama as a data source to generate multi-viewpoint panoramas along the street based on the Markov energy function. This achieves the transition from single-viewpoint panoramas to multi-viewpoint panoramas, achieving seamless stitching based on strict mathematical relationships.

[0010] To achieve the above technical effects, the technical solutions adopted in this application are as follows:

[0011] Projection surface measurement and stitching of multi-view panoramas Figure 3 A 2D reconstruction method is established to establish a multi-viewpoint parallel stitching method for landscape reproduction. Through parallel stitching, a seamless 2D reproduction form of multi-viewpoint panoramic images is obtained. The multi-viewpoint panoramic image generated by 2D parallel stitching is a long corridor-shaped 2D image that fully covers the scenery information on both sides of the street, supports users' continuous browsing of the landscape, and has low data volume, simple data structure and convenient browsing method.

[0012] Multi-segment axis projection tunnel image stitching method: The scene is reconstructed in three dimensions through all-round pictures. The measurement value projection surface suitable for the scene is calculated based on the model. The data is then projected and resampled to generate multi-viewpoint stitching. Based on the mathematical model of data mesh to projection surface mesh and the conversion model of projection surface mesh to plane mesh, a set of mathematical relationships from 3D model scene points to 2D image planes is supervised. Through multi-segment axis projection texture resampling, the measurement relationship between the 3D scene and the 2D image plane is established. This not only meets people's visual perception requirements for multi-viewpoint panoramas, but also introduces measurement information, further broadening the use of multi-viewpoint panoramas.

[0013] 3D scanning projection surface full street view stitching method: Based on the mathematical relationship between scene models, the measurement value projection facade is constructed, and the coordinate relationship model between the 360-degree spherical panorama and the spatial street view facade and the coordinate relationship model between the 360-degree cylindrical panorama and the spatial street view facade are established respectively. The mathematical relationship between the scenes ensures a strict stitching relationship between the two, and the coordinate conversion relationship between the 3D model scene and the 2D multi-viewpoint panorama is established. The street view projection data is resampled to establish a 3D projection panoramic street view stitching that can well express the measurement information of the image.

[0014] Preferably, multi-segment axial projection tunnel image stitching data acquisition and processing: the initial image data of the tunnel is collected by a mobile cart and uploaded to the cloud for processing. The data only contains the data collected by the panoramic pan-tilt head. The cloud-based panoramic pan-tilt head image acquisition and processing process is as follows:

[0015] 1) Image preprocessing: including image distortion correction, image enhancement, and contrast enhancement of preprocessed images;

[0016] 2) Relative orientation: Import the pre-processed image into the aerial triangulation software to automatically match the same-name points, and obtain the relative orientation result after free network adjustment;

[0017] 3) Absolute orientation: Based on the relative orientation result, the absolute orientation result is obtained by adding control points to the image and jointly solving them;

[0018] 4) Generate dense point cloud: Based on the results of absolute orientation, point cloud encryption is performed to obtain high-density and high-precision dense point cloud data. Each image is assigned real 3D spatial position and posture information, and each matching point also has real 3D coordinate information. Every pixel in each image corresponds to a real geographic coordinate position.

[0019] 5) Output result data: Output the processed image and the absolutely oriented dense point cloud according to the actual application requirements or convert them into a specified format for archiving and future use;

[0020] DP-Smart automated image modeling uses DP-Capture to collect data and the DP-Smart program automatically creates models. The DP-Smart program groups and categorizes the data, then automatically models the mesh and selects the tunnel mesh data and corresponding texture information that has been successfully constructed.

[0021] The tunnel data is divided into sub-stations. The division rule is to make the angle between the axis of each tunnel section and the axis of the next tunnel section the largest. The model data divided at each station is conducive to fitting the measurement value projection surface using the regular axis.

[0022] Preferably, a projection surface is selected and established: mesh data of several tunnels and a texture map corresponding to the tunnels are used, wherein the mesh data includes three parts of information: triangle vertex information starting with 'v', corresponding texture coordinates starting with 'vt', and triangle information starting with 'f', and the three-dimensional model point coordinates, texture coordinates and triangles are associated;

[0023] A projection surface is selected and established based on the 3D model scene of the tunnel, and the axial surface that best fits the tunnel is constructed as the measurement value projection surface. First, the vertex information of the 3D model mesh is extracted. Then, noise points that are separated from the tunnel body are filtered and the remaining points are further stored. The normal direction of the filtered points is then estimated to prepare data for subsequent normal segmentation. Based on the input parameters, the near-axial portion of the tunnel side and top and the planar portion of the tunnel ground are obtained respectively. The data of the near-axial portion of the tunnel side and top are selected for axial fitting, and the radius range of the axial projection surface is set. The measurement value projection surface is fitted with the vertex information in the mesh data to obtain the parameter values ​​of the measurement value projection surface so that the projection surface can be mathematically represented.

[0024] The obtained measurement value projection surface is an axial surface. The axial projection surface contains seven parameters, of which the first three represent a point on the axis of the axial projection surface, the middle three represent the axial direction vector of the axial projection surface, and the last one represents the radius of the fitted axial surface. Knowing that any spatial axis can be determined by the axial direction, the point on the axis, and the radius, the axial projection surface is established and determined.

[0025] Verify whether the fitting parameters are accurate: apply the axis fitting function in PCL to perform parameter fitting, compare the difference between the two, save the vertex information of the mesh data as a .pcd file and import it into the software for axis fitting, and compare the fitting effects of the three;

[0026] In addition, the shape detector in CloudCompare is used to detect the best fitting axis for the point cloud and give the parameters of the fitting axis. The fitting parameters of the axis are given in the form of a matrix.

[0027] Preferably, a mathematical model of data mesh to projection surface mesh is established: let point p(X,Y,Z) be the vertex coordinates of the initial mesh, let the perpendicular line through point p intersect the projection surface at point p'(X1,Y1,Z1), and let the intersection point with the axis be m(X2,Y2,Z2), and point p' is the projection point of the point on the projection surface;

[0028] The seven parameters of the axis are the axial vector set to (a, b, c), the coordinates of any point on the axis are (x0, y0, z0), and the radius of the axis is uniformly set to r. The equation of the axis is:

[0029]

[0030] Knowing that m is a point on the axis, Y2 and Z2 are expressed by X2:

[0031]

[0032] Taking into account that the vector pp' formed by points p and p' is perpendicular to the axis vector, it can be further deduced that:

[0033] a*(X-X2)+b*(Y-Y2)+c*(Z-Z2)=0 Equation 3

[0034] get:

[0035]

[0036] Point p is the vertex coordinate of the mesh data, and p' is the projection point of point p on the projection surface, resulting in:

[0037]

[0038] The coordinates of the projection point are:

[0039] X1=X2+Δx

[0040] At this point, the mathematical transformation relationship from the vertices of the 3D model mesh to the vertices of the projection surface mesh is completed;

[0041] Axis fitting and projection processing are performed on each tunnel mesh. There is no fixed mathematical relationship between the projection surface of each tunnel, so the projection surface mesh is processed again.

[0042] In real engineering, the axes of axis A and axis B are a pair of skew lines, which are set as L1 and L respectively. The translation of the axis involves the translation and splicing of the skew lines. Let the equation of line L1 be Let the equation of line L be The direction vectors of the two straight lines are v = (a, b, c) and v1 = (a1, b1, c1). Let the direction vector of the common perpendicular be n = (x1, y1, z1). We get Solve for the direction vector of the common perpendicular line, and then find the equation of the common perpendicular line based on a vertex on the axis (the vertex on the axis can be found by searching), and then combine the equation of the common perpendicular line with the equation of the other axis to find the optimal connection point of the two non-planar axes based on two non-planar straight lines;

[0043] Vertical rotation of the projection mesh: Plane ABC is a plane with the projection mesh axis as its normal and passing through a point on the axis. The seven parameters of the axis are the radius r, a point on the axis (x0, y0, z0), and the axis vector (a, b, c). Write the equation of plane ABC, then calculate the rotation angle to describe the angle between the projection mesh and the z-axis.

[0044] Several tunnel projection meshes are obtained, each of which has an axial direction that is consistent with the z-axis. The axial direction of the mesh projection surface after rotation is strictly checked, and the axial direction vector of the vertical axis is calculated using the axis fitting program.

[0045] Preferably, the conversion model from the projection surface mesh to the plane mesh is as follows: based on a projection surface mesh whose axis coincides with the z-axis, the coordinate relationship between any point p on the projection surface mesh and the plane mesh is found, and the coordinate system of the plane mesh is defined with the intersection B1 of the projection surface mesh and the positive semi-axis of the y-axis as the origin, the positive direction of the z-axis defines the positive direction of the y-axis of the plane mesh, and the x-axis direction of the plane mesh is defined with the x-direction of the original three-dimensional coordinate system;

[0046] Looking down from the positive direction of the z-axis to get a top view, the length of the horizontal coordinate of the plane projection point p is Assume that the angle between the line op connecting point p and the axis center and the positive y-axis of the xoy section is θ. The length of the ordinate of p is the z-coordinate of the projected mesh point, that is, the length of OH. Expand the projected surface mesh along the axis generatrix where point B1 is located to obtain the plane mesh. The relationship between the coordinates is as follows, completing the mathematical conversion from the projected surface mesh to the plane mesh.

[0047]

[0048] When expanding the projection surface mesh along a generatrix on the axial plane, select the location of the expanded generatrix, detect the direction of the generatrix at the top of the tunnel, and make sure that the dividing line must be at the top of the tunnel;

[0049] When projecting onto a plane mesh, the topological information of the triangles is removed. The triangles that the split line passes through need to be indexed and deleted in the original mesh.

[0050] After the triangle on the dividing line is divided, the three vertices are located on both sides of the image, and the distance between the vertices is greater than or equal to the circumference of the projection surface. The triangle information of the projection surface mesh is traversed. For each triangle, the three vertices are pointl(x1, y1, z1), point2(x2, y2, z2), and point3(x3, y3, z3). The radius of the axis projection surface is R. The distances between the three points are calculated respectively, and the sizes of the three distances l1, l2, and l3 are compared with R. If the distance is greater than R, the dividing line passes through the triangle and the triangle is deleted.

[0051] The last step is to unfold the projection surface mesh into a plane mesh projection. A fixed projection base is selected that is suitable for several tunnel sections in the scene. After the plane unfolded mesh is generated, the texture is attached and the measurement value projection is completed.

[0052] Preferably, multi-segment plane axis projection texture resampling: construct a mathematical relationship between the plane unfolding mesh and the final multi-view panoramic image, and convert the relationship between the plane unfolding mesh and the pixel position. The origin of the image coordinate is at the upper left corner of the computer screen. Point A (m, n) represents the position of a pixel on the image. Assuming that the pixel size of the image is pixel, the coordinates of point A on the plane mesh are (x, y). The relationship between the two is:

[0053] n = [x / pixel]

[0054] m = [row.pixel - y] / pixel Formula 10

[0055] Find the coordinates (x, y) on the corresponding plane mesh according to the rows and columns of the multi-view panorama, and use the affine transformation matrix to inversely solve the pixel values ​​of the corresponding positions in the rows and columns of the panorama;

[0056] Get an unfolded plane mesh, find the mapping relationship between the plane mesh's triangles and the texture image UV coordinates, and use affine transformation to obtain the affine transformation matrix between the plane triangles and the texture coordinates. Each triangle and the corresponding texture coordinates have an affine transformation matrix R. The pixel values ​​of the pixels contained in a triangle are further resampled through the inverse solution method.

[0057] Find the minimum bounding rectangle of a triangular facet of a planar mesh, calculate the two-dimensional texture coordinates corresponding to the pixels according to the number of rows and columns of pixels contained in the minimum bounding rectangle, continuously look for the minimum circumscribed polygon of each triangular facet in a mesh plane to define the rows and columns of the search graph, use bilinear interpolation method to assign pixel values, and generate a multi-viewpoint plane image from the three-dimensional mesh and texture information.

[0058] In addition, in the data processing part, software cutting tools are used to decompose the tunnel into several parts. The incisions of each tunnel section are not regular circles. The incisions of most tunnels are ellipses. Local stretching and other processing or channel transparency processing are performed on local parts of the image to establish the relationship between the pixel coordinates of the two-dimensional multi-viewpoint panorama and the three-dimensional scene coordinates.

[0059] Optimally, the key points for capturing 3D scanning street view source data are as follows: the camera rotation angle is determined based on the panoramic gimbal used to capture images to ensure full coverage of the scene and photo overlap. The lens is tilted upwards by approximately 5 degrees with its optical center as the axis during shooting. After capturing the image, the stitched image is loaded and the software is used to obtain a successfully stitched panorama. The image is then processed into a rectangular panorama of uniform size by removing the content of the capture vehicle and tripod that is irrelevant to the captured content. The captured data is then stitched using DP-Smart.

[0060] Preferably, construct the measured value projection facade: assuming that the position of the main plane is the position of the multi-viewpoint facade, and the row tree a is not in the main plane, the images of the row tree a from different viewpoints are not in the same position in the main plane. When determining the multi-viewpoint projection facade, select the facade where the building falls to ensure that the information of interest can be spliced. When selecting the projection facade, consider the target of interest in the entire panoramic image, and then determine the location of the facade;

[0061] Determine the projection surface through the initial image data, obtain sparse spatial point cloud data based on SFM, fit the projection facade according to the 3D scene reflected by the point cloud data, and explain the specific process of solving the spatial point cloud through the mathematical relationship between the intersection of the 3D spatial point and the spatial front of the photograph;

[0062] The sparse point cloud generated by the SFM method is used to fit the measurement value projection surface. The measurement value projection surface is then solved based on the 3D scene restored from the image. At each station, a panoramic gimbal is used to acquire image data, obtain the image's focal length and initial POS information, and input the image, focal length, initial POS information, sensor size, and control points into smart3D. After initial aerial triangulation, the control points are added and the solution is recalculated to assess the aerial triangulation accuracy and produce a visual model.

[0063] Determine the facade equation and the motion trajectory of the 3D camera based on the 3D model of the scene. First, import the generated 3D model into Smart3DCapture Viewer and select the corresponding coordinate system based on the location of the shooting scene.

[0064] Use measurement tools to determine the coordinates of 3D points in the 3D scene. Select multiple points along the street to fit the facade equation. Then select the highest point of the building in the scene to determine the facade height. The street facade is perpendicular to the ground along the street. Fit the equation of the street facade in the xoy plane to determine the facade equation. Use Origin8 to fit the selected points.

[0065] For straight streets, the facade solution is simplified to the facade along the street. The coordinates of three points are (x1, y1, y1), (x2, y2, z2), (x3, y3, z3), and the facade equation is solved as: a = 1

[0066]

[0067] Where (a, b, c) represents the normal vector of the plane, and determines the facade with the best projection effect for a scene.

[0068] Preferably, the relationship model between the 360-degree spherical panorama and the spatial street view facade coordinates is as follows: Assume that the pixel coordinates of the rectangular panoramic image are M(m,n), and the center O of the panoramic sphere corresponds to G of the panoramic image. The direction from point O to point G is defined as the y-axis of the panoramic sphere. The x-axis should be located in the equatorial plane, requiring the x-axis to be perpendicular to the y-axis. The direction defined by the z-axis is perpendicular to the xoy plane. The relationship between a point M(m,n) on the image plane and the corresponding mapping point P(x,y,z) on the panoramic sphere is as follows:

[0069]

[0070] Finally, the relationship between P(x,y,z) and M(m,n) is:

[0071]

[0072] Where W is the length of the photo and H is the width of the photo, and the transformation from the rectangular panoramic image plane to the panoramic spherical image space coordinate system is obtained, and then the transformation from the panoramic spherical image space coordinate system to the panoramic spherical image space auxiliary coordinate system is obtained;

[0073] Convert from the panoramic spherical image space coordinate system to the panoramic spherical auxiliary coordinate system, use the three angles in photogrammetry to control the transformation of the coordinate system, and find the transformation relationship between the coordinates according to the transformation matrix. Assume that the coordinates of the mapping point in the panoramic spherical image space auxiliary coordinate system are P'(X, Y, Z), and the coordinates in the panoramic spherical image space coordinate system are set to P(x, y, z). The transformation matrix between the two is set to R. The transformation matrix and ω, The relationship between κ is as follows:

[0074]

[0075] Finally, consider the coordinates P of a space object point in the panoramic spherical auxiliary coordinate system. c (X c ,Y c ,Z c ) and the world coordinate system, let the coordinates of the object point in the world coordinate be P d (X d ,Y d ,Z d ), s' is the coordinate point of the panoramic sphere center in the world coordinate system, and p represents a point on the object side; the relationship between the panoramic spherical image space auxiliary coordinate system and the world coordinate system is:

[0076]

[0077] where X s ,Y s ,Z s is the coordinate of the photography center in the world coordinate system, P c (X c ,Y c ,Z c ) and P`(X, Y, Z) satisfy the collinear equation, and we get:

[0078]

[0079] The value of λ is the ratio of the distance from the image space auxiliary coordinates to the center of the panoramic sphere to the radius of the panoramic sphere. Define the radius of the panoramic sphere as 1, starting from a point P in space d (X d ,Y d ,Z d ) to the 360-degree rectangular panoramic image coordinates M(m,n):

[0080]

[0081] Preferably, the street view projection data is resampled: given an undirected graph defined as G = (V, ε), where V represents a pixel point, ε represents the relationship between a pixel and its adjacent pixels, there is a pixel point set v = {p1, p2, K, p N}, where the mark field ω={w p1 ,w p2 ,K,w pN},w v ={1,2,K,K L ), L is the symbol set, K L is the total number of labels, and image segmentation is performed as a maximum a posteriori estimate The parameter F is expressed as a random variable, P(w) represents the segmentation result of the next instance based on F, P(w|F) represents the posterior probability of the segmentation result under this condition, P(F / w) is the probability distribution of the random variable F, P(F) is the prior probability, if when F is an instance, its probability is a certain value, P(w|F) and P(w)P(F|w) are proportional to each other. After obtaining the prior probability P(w) and the likelihood function P(F|w), it is converted to U(w|F)=U(F|w)+U(w)+Uc through w=arg max(P(w)P(F|w));

[0082] U(w|F) represents the posterior energy, U C Represents a constant. The three-dimensional coordinates of each pixel point on the multi-view panoramic image are set to p(x, y, z). Each pixel point position in the multi-view panoramic image has a normal vector S(p) perpendicular to the plane. The position coordinates of each camera in the photography are set to pos(x0, y0, z0).

[0083] Calculate the normal vector of each pixel as S(p), set the line vector between the pixel and the viewpoint as a, and calculate the angle between each viewpoint and the normal of each pixel The one with the smallest angle is the one that is at the front of the search;

[0084] The cost function D(p, L(p))=|pp is selected to represent the two-dimensional plane distance between the projection point of each viewpoint on the facade and the pixel point. L(p) |, where p represents the position coordinate of the pixel point, p L(p) Represents the two-dimensional coordinates of the projection of each viewpoint on the facade, and D(P, L(P)) represents the coordinates of the projection point to the pixel point on the two-dimensional plane;

[0085] Assuming that the coordinates of the projection points of each two panoramic sphere centers are (x0, y0, z0) and (x1, y1, z1), calculate the perpendicular bisector of the line connecting the centers of the two panoramic spheres:

[0086]

[0087] Calculate the projection facade equation and the perpendicular bisector to divide the entire projection facade into several blocks. Combined with the characteristics of multi-viewpoint panoramic images, the field of view and scenery in front of you are always at the forefront when browsing the street view. Ensure that the pixel collection of each area is at the forefront. During the specific projection, ensure that adjacent projection areas have overlapping parts. Subsequent steps will further determine the projection stitching line to obtain an image that minimizes the energy function.

[0088] Compared with the existing technology, the innovation and advantages of this application are:

[0089] (1) The panoramic images of this application use surveying and mapping information in computer vision effects. Surveying and mapping provide location services and establish relationships with locations. The maps produced use images to provide mathematical relationships with the real world. The latitude and longitude of a point on the map generated based on a strict mathematical model can accurately find a point in the three-dimensional scene, which provides great convenience for real life. In addition to visual expression, panoramas are examined in a different way, allowing panoramas to carry mathematical information and have a stronger connection with the real three-dimensional world. A new splicing method is established to process the captured scene images to reconstruct the three-dimensional scene in different forms. For the tunnel data of the first experiment in this application, the mesh and texture map of a section of tunnel data are obtained by modeling the pictures taken from all directions of the tunnel. The tunnel model is used to determine the projection surface of the measurement value, and then the model is flattened to obtain seamless splicing. Street view data is also obtained by taking photos of the street in all directions, processing the pictures to obtain a three-dimensional model, and stitching the pictures to obtain a 360-degree panoramic sphere. Then, the measurement values ​​are determined based on the three-dimensional model and projected onto the facade to obtain a multi-viewpoint panorama of the street. The two-dimensional scene is reconstructed through strict mathematical transformation to ensure seamless stitching between images and strict mathematical measurement relationships, thereby realizing photogrammetric calculations based on the panorama.

[0090] (2) This application establishes a multi-viewpoint parallel stitching method for landscape reproduction, which obtains a seamless two-dimensional reproduction form of a multi-viewpoint panoramic image through parallel stitching. The multi-viewpoint panoramic image generated by two-dimensional parallel stitching is a two-dimensional corridor-shaped image, which completely covers the scenery information on both sides of the street, supports users to browse the landscape continuously, and has low data volume, simple data structure and convenient browsing method. The first is a multi-segment face axis projection tunnel image stitching method: the scene is reconstructed in three dimensions through all-round pictures, and the measurement value projection surface suitable for the scene is calculated according to the model, and then the data is projected and resampled to generate multi-viewpoint stitching, which not only solves people's visual perception requirements for multi-viewpoint panoramas, but also introduces measurement information to further broaden the use of multi-viewpoint panoramas; the second is a three-dimensional scanning projection surface full street view stitching method: based on the mathematical relationship between scene models, a measurement value projection facade is constructed, and a coordinate conversion relationship between the three-dimensional model scene and the two-dimensional multi-viewpoint panorama is established. The street view projection data is resampled to establish a three-dimensional projection panoramic street view stitching that has a good expression of the image's measurement information. Multi-viewpoint panorama Figure 3 The 3D reconstruction has high accuracy and efficiency, and can be used for panoramic photogrammetry calculations for precise measurement.

[0091] (3) In view of the weak texture of tunnel images and the low brightness inside the tunnel, the traditional image-based matching algorithm is not suitable for image stitching of tunnel scenes. This application proposes a method for stitching tunnel scenes based on a strictly mathematically derived measurement value projection surface. The entire process is based on strict mathematical operations, and finally ensures seamless stitching between tunnel stations. It has important practical significance in actual engineering applications. In view of the similarity of tunnel splicing image textures, it is impossible to effectively use the existing image matching algorithm for image stitching. Tunnels are not strictly axial shapes. In real life, tunnels are often undulating and appear as curved polygons with the axis as a curve. A two-dimensional flattening method for the three-dimensional scene of the tunnel is established. Using the mesh and texture information of the tunnel, a seamless tunnel stitching is generated through the processes of measurement value projection surface determination, coordinate conversion, texture resampling, etc. For street scenes, this application uses 360-degree rectangular panorama as the data source to generate multi-viewpoint panoramas along the street based on the Markov energy function, realizing the transition from single-viewpoint panorama to multi-viewpoint panorama, and achieving seamless stitching based on strict mathematical relationships. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] Figure 1 It is to select and establish a schematic diagram of the projection surface tunnel fitting parameters.

[0093] Figure 2 It is a schematic diagram of the mathematical model for building data mesh to projection surface mesh.

[0094] Figure 3 It is a schematic diagram of the mathematical model projection point coordinate calculation from the data mesh to the projection surface mesh.

[0095] Figure 4 It is a schematic diagram of the processing of the axis of the parametric projection surface.

[0096] Figure 5 It is the conversion model from projection surface mesh to plane mesh, and the projection surface plane shows the intention.

[0097] Figure 6 This is a schematic diagram of the splitting line selection for the conversion model from projection surface mesh to plane mesh.

[0098] Figure 7 This is a multi-view panoramic view of a tunnel obtained using measured data from a tunnel in Yunnan.

[0099] Figure 8 This is a diagram showing the relationship between facade selection and projection.

[0100] Figure 9 It is a relationship diagram between spherical panorama and rectangular panorama mapping.

[0101] Figure 10 It is a flow chart of 3D projection panoramic street view stitching.

[0102] Figure 11 It is a schematic diagram of the rectangular panoramic sphere of the experimental input data.

[0103] Figure 12 It is a street mosaic graph considering the energy function in the experiment. DETAILED DESCRIPTION

[0104] The following is a multi-view panoramic view of the projection surface measurement provided by this application with reference to the accompanying drawings. Figure 3 The technical solution of the dimensional reconstruction method is further described so that those skilled in the art can better understand the present application and implement it.

[0105] In response to the current situation, this application establishes a multi-viewpoint parallel stitching method for landscape reproduction. Through parallel stitching, a seamless two-dimensional reproduction form of a multi-viewpoint panoramic image is obtained. The multi-viewpoint panoramic image generated by two-dimensional parallel stitching is a two-dimensional corridor-shaped image, which completely covers the scenery information on both sides of the street, supports users' continuous browsing of the landscape, and has low data volume, simple data structure and convenient browsing method.

[0106] Multi-segment axis projection tunnel image stitching method: The scene is reconstructed in three dimensions through all-round pictures. The measurement value projection surface suitable for the scene is calculated based on the model. The data is then projected and resampled to generate multi-viewpoint stitching. Based on the mathematical model of data mesh to projection surface mesh and the conversion model of projection surface mesh to plane mesh, a set of mathematical relationships from 3D model scene points to 2D image planes is supervised. Through multi-segment axis projection texture resampling, the measurement relationship between the 3D scene and the 2D image plane is established. This not only meets people's visual perception requirements for multi-viewpoint panoramas, but also introduces measurement information, further broadening the use of multi-viewpoint panoramas.

[0107] 3D scanning projection surface full street view stitching method: Based on the mathematical relationship between scene models, the measurement value projection facade is constructed, and the coordinate relationship model between the 360-degree spherical panorama and the spatial street view facade and the coordinate relationship model between the 360-degree cylindrical panorama and the spatial street view facade are established respectively. The mathematical relationship between the scenes ensures a strict stitching relationship between the two, and the coordinate conversion relationship between the 3D model scene and the 2D multi-viewpoint panorama is established. The street view projection data is resampled to establish a 3D projection panoramic street view stitching that can well express the measurement information of the image.

[0108] 1. Multi-segment axial projection tunnel image stitching method

[0109] Given the weak texture and low brightness of tunnel images, traditional image-based matching algorithms are not suitable for image stitching in tunnel scenes. Therefore, this application proposes a method for stitching projected surfaces based on rigorous mathematical derivation of measured values. This method, based on rigorous mathematical calculations throughout the entire process, ultimately ensures seamless stitching between tunnel stations, which has important practical significance in practical engineering applications.

[0110] One of the difficulties is that the textures of tunnel stitching images are similar, and it is impossible to effectively use existing image matching algorithms for image stitching. Secondly, tunnels are not strictly axial shapes. In real life, tunnels are often undulating and appear as curved polygons with the axis as a curve. Therefore, how to flatten the three-dimensional scene of the tunnel in two dimensions is a very meaningful topic.

[0111] Based on the above analysis, this application no longer uses ordinary tunnel images as initial data. Tunnel image-based image stitching methods have poor results due to weak tunnel texture information and few feature points. Therefore, this application uses the tunnel mesh and texture information to generate a seamless tunnel stitching through a process of measurement value projection surface determination, coordinate conversion, and texture resampling. The specific process is as follows:

[0112] Step 1: Based on the full-scale shooting of the tunnel, the pictures are processed by software to perform 3D reconstruction operations to obtain the required tunnel mesh and corresponding texture map;

[0113] Step 2: Fit and establish the measurement value projection surface based on the provided mesh data. Since the research object of this application is a tunnel and most tunnels are paraxial, an axis closest to the tunnel axis is fitted using the initial data as the projection surface;

[0114] Step 3: Based on the calculated measurement value projection surface, the mesh is projected onto the calculated projection surface to obtain the projection surface mesh; the mesh on the projection surface is rotated so that the axis of each tunnel section coincides with the z-axis. This strictly guarantees the mathematical relationship between the front and back of each tunnel section after projection. Strictly ensuring this mathematical relationship has an important influence on achieving seamless splicing of images, and the vertical mesh is obtained;

[0115] Step 4: Define the image plane coordinate system, realize the mathematical transformation between the vertical back projection surface mesh and the plane mesh, and obtain the plane mesh;

[0116] Step 5: Resample the image data. Calculate the affine transformation matrix T (a 2-row, 3-column matrix) between the 2D coordinates of each triangle in the mesh and the texture coordinates of the three vertices in the texture image. A search area is determined based on the minimum bounding rectangle of each triangle in the mesh, and each pixel in this area is resampled.

[0117] (1) Data collection and processing

[0118] The initial image data of the tunnel is collected by a mobile cart and uploaded to the cloud for processing. The data only contains data collected by the panoramic pan-tilt head. The cloud-based panoramic pan-tilt head image processing process is as follows:

[0119] 1) Image preprocessing: This includes image distortion correction, image enhancement, and contrast enhancement of preprocessed images, providing high-quality data for subsequent aerial triangulation.

[0120] 2) Relative orientation: Import the pre-processed image into the aerial triangulation software to automatically match the same-name points, and obtain the relative orientation result after free network adjustment;

[0121] 3) Absolute orientation: Based on the relative orientation result, the absolute orientation result is obtained by adding control points to the image and jointly solving them;

[0122] 4) Generate dense point cloud: Based on the results of absolute orientation, point cloud encryption is performed to obtain high-density and high-precision dense point cloud data. Each image is assigned real 3D spatial position and posture information, and each matching point also has real 3D coordinate information. Every pixel in each image corresponds to a real geographic coordinate position.

[0123] 5) Output result data: Output the processed image and the absolutely oriented dense point cloud according to the actual application requirements or convert them into a specified format for archiving and future use;

[0124] DP-Smart automated image modeling uses DP-Capture to collect data and the DP-Smart program automatically creates models. The DP-Smart program groups and categorizes the data, then automatically models the mesh and selects the tunnel mesh data and corresponding texture information that has been successfully constructed.

[0125] The tunnel data is divided into sub-stations. The division rule is to make the angle between the axis of each tunnel section and the axis of the next tunnel section the largest. The model data divided at each station is conducive to fitting the measurement value projection surface using the regular axis.

[0126] (2) Select and create the projection surface

[0127] The mesh data of several tunnels and the texture maps corresponding to the tunnels are used. The mesh data includes three parts of information: triangle vertex information starting with 'v', corresponding texture coordinates starting with 'vt', and triangle information starting with 'f', which associates the 3D model point coordinates, texture coordinates and triangles.

[0128] A projection surface is selected and established based on the 3D model scene of the tunnel, and the axial surface that best fits the tunnel is constructed as the measurement value projection surface. First, the vertex information of the 3D model mesh is extracted. Then, noise points that are separated from the tunnel body are filtered and the remaining points are further stored. The normal direction of the filtered points is then estimated to prepare data for subsequent normal segmentation. Based on the input parameters, the near-axial portion of the tunnel side and top and the planar portion of the tunnel ground are obtained respectively. The data of the near-axial portion of the tunnel side and top are selected for axial fitting, and the radius range of the axial projection surface is set. The measurement value projection surface is fitted with the vertex information in the mesh data to obtain the parameter values ​​of the measurement value projection surface so that the projection surface can be mathematically represented.

[0129] The obtained measurement value projection surface is an axial surface. The axial projection surface contains seven parameters, of which the first three represent a point on the axis of the axial projection surface, the middle three represent the axial direction vector of the axial projection surface, and the last one represents the radius of the fitted axial surface. Knowing that any spatial axis can be determined by the axial direction, the point on the axis, and the radius, the axial projection surface is established and determined, such as Figure 1 shown.

[0130] Verify whether the fitting parameters are accurate: apply the axis fitting function in PCL to perform parameter fitting, compare the difference between the two, save the vertex information of the mesh data into a .pcd file (storing the vertex information of the mesh) and import it into the software for axis fitting, and compare the fitting effects of the three;

[0131] In addition, the shape detector in CloudCompare is used to detect the best fitting axis for the point cloud and give the parameters of the fitting axis. The fitting parameters of the axis are given in the form of a matrix.

[0132] (3) Establishing a mathematical model from data mesh to projection surface mesh

[0133] like Figure 2 Let point p(X,Y,Z) be the vertex coordinates of the initial mesh, let the perpendicular line through point p intersect the projection plane at point p'(X1,Y1,Z1), and let the intersection with the axis be m(X2,Y2,Z2). Point p' is the projection point of the point on the projection plane.

[0134] The seven parameters of the axis are the axial vector set to (a, b, c), the coordinates of any point on the axis are (x0, y0, z0), and the radius of the axis is uniformly set to r. The equation of the axis is:

[0135]

[0136] Knowing that m is a point on the axis, Y2 and Z2 are expressed by X2:

[0137]

[0138] Taking into account Figure 3 The vector pp' formed by the points p and p' shown is perpendicular to the axis vector and further deduced:

[0139] a*(X-X2)+b*(Y-Y2)+c*(Z-Z2)=0 Equation 3

[0140] get:

[0141]

[0142] Point p is the vertex coordinate of the mesh data, and p' is the projection point of point p on the projection surface, resulting in:

[0143]

[0144] The coordinates of the projection point are:

[0145]

[0146] At this point, the mathematical transformation relationship from the vertices of the 3D model mesh to the vertices of the projection surface mesh is completed;

[0147] Axis fitting and projection processing are performed on each tunnel mesh. However, the fitting of each projection surface destroys the seamless connection of the initial 3D scene. There is no fixed mathematical relationship between the projection surfaces of each tunnel, so the projection surface mesh is processed again.

[0148] like Figure 4 It can be seen that the axis of axis A and axis B are a pair of non-planar straight lines in real engineering, which are set as L1 and L respectively. The translation of the axis involves the translation and splicing of non-planar straight lines. The equation of the straight line Ll is Let the equation of line L be The direction vectors of the two straight lines are v = (a, b, c) and v1 = (a1, b1, c1). Let the direction vector of the common perpendicular be n = (x1, y1, z1). We get Solve for the direction vector of the common perpendicular line, and then find the equation of the common perpendicular line based on a vertex on the axis (the vertex on the axis can be found by searching), and then combine the equation of the common perpendicular line with the equation of the other axis to find the optimal connection point of the two non-planar axes based on two non-planar straight lines;

[0149] Vertical rotation of the projection mesh: Plane ABC is a plane with the projection mesh axis as its normal and passing through a point on the axis. The seven parameters of the axis are the radius r, a point on the axis (x0, y0, z0), and the axis vector (a, b, c). Write the equation of plane ABC, then calculate the rotation angle to describe the angle between the projection mesh and the z-axis.

[0150] Several tunnel projection meshes are obtained, each of which has an axial direction that is consistent with the z-axis. The axial direction of the mesh projection surface after rotation is strictly checked, and the axial direction vector of the vertical axis is calculated using the axis fitting program.

[0151] (IV) Conversion model from projection surface mesh to plane mesh

[0152] like Figure 5 The left side shows a projection surface mesh whose axis coincides with the z-axis. Find the coordinate relationship between any point p on the projection surface mesh and the plane mesh. The coordinate system of the plane mesh is defined with the intersection B1 of the projection surface mesh and the positive semi-axis of the y-axis as the origin, the positive direction of the z-axis defines the positive direction of the plane mesh's y-axis, and the x-axis direction of the plane mesh is defined by the x-direction of the original three-dimensional coordinate system.

[0153] Figure 5 In the upper right corner, looking down from the positive direction of the z-axis, we get a top view. The length of the horizontal coordinate of the plane projection point p is (length of arc BlP), let the angle between the line op connecting point p and the axis center and the positive y-axis of the xoy section be θ, the length of the ordinate of p is the z-coordinate of the projection mesh point, that is, the length of OH, and the projection surface mesh is expanded along the axis generatrix where point B1 is located to obtain the plane mesh. The relationship between the coordinates is as follows, completing the mathematical conversion from the projection surface mesh to the plane mesh;

[0154]

[0155] When expanding the projection surface mesh along a generatrix on the axial plane, select the location of the expanded generatrix, detect the direction of the generatrix at the top of the tunnel, and make sure that the dividing line must be at the top of the tunnel;

[0156] like Figure 6 The experimental results are different because of the incorrect selection of the tunnel dividing line. This experimental data is for a tunnel mesh without ground, such as Figure 6 The left side shows the dividing line on the ground side, and the right side shows the dividing line on the top of the tunnel.

[0157] When a split line is artificially defined to expand the projected mesh into a flat mesh, the initial 3D mesh structure is destroyed, destroying the original mesh structure. When projecting onto a flat mesh, the topological information of the triangles is removed, and the triangles that the split line passes through need to be indexed and deleted from the original mesh.

[0158] After the triangle on the dividing line is divided, the three vertices are located on both sides of the image, and the distance between the vertices is greater than or equal to the circumference of the projection surface. The triangle information of the projection surface mesh is traversed. For each triangle, the three vertices are point1 (x1, y1, z1), point2 (x2, y2, z2), and point3 (x3, y3, z3). The radius of the axis projection surface is R. The distance between the three points is calculated respectively, and the size of the three distances l1, l2, and l3 is compared with R. If the distance is greater than R, the dividing line passes through the triangle and the triangle is deleted.

[0159] The last step is to unfold the projection surface mesh into a plane mesh projection. A fixed projection base is selected that is suitable for several tunnel sections in the scene. After the plane unfolded mesh is generated, the texture is attached and the measurement value projection is completed.

[0160] (5) Multi-segment axis projection texture resampling

[0161] Construct a mathematical relationship between the plane unfolding mesh and the resulting multi-view panorama, and convert the relationship between the plane unfolding mesh and the pixel position. The origin of the image coordinates is at the upper left corner of the computer screen. Point A (m, n) represents the position of a pixel on the image. Assuming that the pixel size of the image is pixel, let the coordinates of point A on the plane mesh be (x, y). The relationship between the two is:

[0162] n = [x / pixel]

[0163] m = [row.pixel - y] / pixel Formula 10

[0164] Find the coordinates (x, y) on the corresponding plane mesh according to the rows and columns of the multi-view panorama, and use the affine transformation matrix to inversely solve the pixel values ​​of the corresponding positions in the rows and columns of the panorama;

[0165] Get an unfolded plane mesh, find the mapping relationship between the plane mesh's triangles and the texture image UV coordinates, and use affine transformation to obtain the affine transformation matrix between the plane triangles and the texture coordinates. Each triangle and the corresponding texture coordinates have an affine transformation matrix R. The pixel values ​​of the pixels contained in a triangle are further resampled through the inverse solution method.

[0166] Find the minimum bounding rectangle of a triangular facet of a planar mesh, calculate the two-dimensional texture coordinates corresponding to the pixels according to the number of rows and columns of pixels contained in the minimum bounding rectangle, continuously look for the minimum circumscribed polygon of each triangular facet in a mesh plane to define the rows and columns of the search graph, use bilinear interpolation method to assign pixel values, and generate a multi-viewpoint plane image from the three-dimensional mesh and texture information.

[0167] In addition, in the data processing part, software cutting tools are used to decompose the tunnel into several parts. The incisions of each tunnel section are not regular circles. The incisions of most tunnels are ellipses. Local stretching and other processing or channel transparency processing are performed on local parts of the image to establish the relationship between the pixel coordinates of the two-dimensional multi-viewpoint panorama and the three-dimensional scene coordinates.

[0168] (6) Tunnel flattening test

[0169] The experimental data is the generated tunnel mesh (triangular patch) data and the corresponding tunnel texture map. The goal is to generate a seamless flattened map of several tunnel stations. This application realizes that only the mesh data and texture information of the initial tunnel are required to automatically generate a seamless flattened map.

[0170] like Figure 7 , Figure 7 Experiment 1 uses measured data from a tunnel in Yunnan to create a multi-view panoramic image of the tunnel. The test results allow for a simple interpretation of tunnel hazards, such as water seepage and cracks on the tunnel roof. The figure shows the joints of the tunnel, demonstrating excellent results and achieving seamless tunnel connection.

[0171] The tunnel experiment in this application used data from two tunnel sections in Yunnan and Changsha. By processing all-around tunnel images, mesh and texture maps were generated. A multi-viewpoint panoramic image was generated through processes such as fitting measurement value projection surfaces, surface projection, surface expansion, and texture resampling. Disaster prediction and detection were performed on the tunnel sidewalls and top. The panoramic image also has a measurement function. After a series of mathematical derivations, the final panoramic image was placed in GlobalMapper as an orthophoto image with coordinates. This provides measurement information for surveying and mapping, and the entire tunnel scene can be viewed from a wide angle from the panoramic angle.

[0172] 2.3D Scanning Projection Surface Full Street View Stitching Method

[0173] Combining street view multi-viewpoint panoramas, a street view stitching method based on the measurement value projection surface is established. The street view is a straight road with houses built along the street. The street view vehicle trajectory is also along the street. The projection surface is a vertical scanning projection surface. First, a series of 360-degree panoramic images are collected and stitched together by the panoramic gimbal acquisition vehicle. The transformation from a single-viewpoint panorama (referring to a 360-degree panoramic sphere) to a multi-viewpoint panorama is performed. The collected image data is processed and stitched to generate a rectangular 360-degree panoramic sphere and processed into a 3D scene model. The measurement value projection surface is solved in combination with the 3D model, projected based on the global optimization rule, and texture resampling is performed.

[0174] (1) Street View Source Data Processing

[0175] Key points for street view photography: Determine the camera's rotation angle based on the panoramic gimbal used to capture images, ensuring full coverage of the scene and photo overlap. With the lens' optical center as the axis, tilt the lens approximately 5 degrees when shooting. After acquiring the image, load the stitched image. Use software to create a successfully stitched panorama. Remove any irrelevant content from the image, including the vehicle and tripod, to create a uniformly sized rectangular panorama. Use DP-Smart to stitch the captured data.

[0176] (2) Constructing the measured value projection facade

[0177] like Figure 8 As shown, assuming that the position of the main plane is the position of the multi-viewpoint facade, the row tree a is not in the main plane, and the imaging of the row tree a from different viewpoints is not in the same position in the main plane. When determining the multi-viewpoint projection facade, select the facade where the building falls to ensure that the information of interest can be spliced. When selecting the projection facade, consider the target of interest in the entire panoramic image, and then determine the position of the facade;

[0178] Determine the projection surface through the initial image data, obtain sparse spatial point cloud data based on SFM, fit the projection facade according to the 3D scene reflected by the point cloud data, and explain the specific process of solving the spatial point cloud through the mathematical relationship between the intersection of the 3D spatial point and the spatial front of the photograph;

[0179] The sparse point cloud generated by the SFM method is used to fit the measurement value projection surface. The measurement value projection surface is then solved based on the 3D scene restored from the image. At each station, a panoramic gimbal is used to acquire image data, obtain the image's focal length and initial POS information, and input the image, focal length, initial POS information, sensor size, and control points into smart3D. After initial aerial triangulation, the control points are added and the solution is recalculated to assess the aerial triangulation accuracy and produce a visual model.

[0180] Determine the facade equation and the motion trajectory of the 3D camera based on the 3D model of the scene. First, import the generated 3D model into Smart3DCapture Viewer and select the corresponding coordinate system based on the location of the shooting scene.

[0181] Use measurement tools to determine the coordinates of 3D points in the 3D scene. Select multiple points along the street to fit the facade equation. Then select the highest point of the building in the scene to determine the facade height. The street facade is perpendicular to the ground along the street. Fit the equation of the street facade in the xoy plane to determine the facade equation. Use Origin8 to fit the selected points.

[0182] For straight streets, the facade solution is simplified to the facade along the street. The coordinates of three points are (x1, y1, y1), (x2, y2, z2), (x3, y3, z3), and the facade equation is solved as: a = 1

[0183]

[0184] Where (a, b, c) represents the normal vector of the plane, and determines the facade with the best projection effect for a scene.

[0185] (3) Street View Jigsaw Projection

[0186] Because the final result is a multi-viewpoint panoramic image along the street, after determining the measurement value projection surface, the next step is to establish the relationship between the 360° rectangular panoramic image and the multi-viewpoint panoramic image. Therefore, the first thing to do is to establish the mathematical relationship between the 360° rectangular panoramic image and the facade coordinates and perform texture resampling.

[0187] 1- Coordinate conversion from rectangular panorama to panoramic sphere

[0188] (1) 360-degree spherical panorama and streetscape facade coordinate relationship model

[0189] Assume that the pixel coordinates of the rectangular panoramic image are M(m,n), and the center O of the panoramic sphere corresponds to G of the panoramic image. The direction from point O to point G is defined as the y-axis of the panoramic sphere. The x-axis should be in the equatorial plane and perpendicular to the y-axis. The direction defined by the z-axis is perpendicular to the xoy plane. Figure 9 The relationship between a point M(m,n) on the image plane and the corresponding mapping point P(x,y,z) on the panoramic sphere is:

[0190]

[0191] Finally, the relationship between P(x,y,z) and M(m,n) is:

[0192]

[0193] Where W is the length of the photo and H is the width of the photo, and the transformation from the rectangular panoramic image plane to the panoramic spherical image space coordinate system is obtained, and then the transformation from the panoramic spherical image space coordinate system to the panoramic spherical image space auxiliary coordinate system is obtained;

[0194] Convert from the panoramic spherical image space coordinate system to the panoramic spherical auxiliary coordinate system, use the three angles in photogrammetry to control the transformation of the coordinate system, and find the transformation relationship between the coordinates according to the transformation matrix. Assume that the coordinates of the mapping point in the panoramic spherical image space auxiliary coordinate system are P'(X, Y, Z), and the coordinates in the panoramic spherical image space coordinate system are set to P(x, y, z). The transformation matrix between the two is set to R. The transformation matrix and ω, The relationship between κ is as follows:

[0195]

[0196] Finally, consider the coordinates P of a space object point in the panoramic spherical auxiliary coordinate system. c (X c ,Y c ,Z c ) and the world coordinate system, let the coordinates of the object point in the world coordinate be P d (X d ,Y d ,Z d ), s' is the coordinate point of the panoramic sphere center in the world coordinate system, and p represents a point on the object side; the relationship between the panoramic spherical image space auxiliary coordinate system and the world coordinate system is:

[0197]

[0198] where X s ,Y s ,Z s is the coordinate of the photography center in the world coordinate system, P c (X c ,Y c ,Z c ) and P`(X, Y, Z) satisfy the collinear equation, and we get:

[0199]

[0200] The value of λ is the ratio of the distance from the image space auxiliary coordinates to the center of the panoramic sphere to the radius of the panoramic sphere. Define the radius of the panoramic sphere as 1, starting from a point P in space d (X d ,Y d ,Z d ) to the 360-degree rectangular panoramic image coordinates M(m,n):

[0201]

[0202] (2) 360° cylindrical panorama and spatial street view facade coordinate relationship model

[0203] Assume that the radius of the axis is r, and the coordinates of the auxiliary coordinate system point O are O(X O ,Y O ,Z O ), the relationship between the two is:

[0204]

[0205] Assume that the coordinates of the image space coordinate system are (x1, y1, z1) and the relationship between the coordinates of the image space auxiliary coordinate system (x, y, z) should be:

[0206]

[0207] The values ​​of each element in the matrix are:

[0208]

[0209] The remaining relationship is between the texture coordinates (u, v) of the 360-degree rectangular panorama and the coordinates (x1, y1, z1) in the cylindrical panoramic image space coordinate system. Assume the coordinates (u, v) of a point on the photo, the radius of the axis is r, and the width of the photo is row.

[0210] 2-Street View Projection Data Resampling

[0211] Given an undirected graph defined as G = (V, ε), where V represents the pixel point and ε represents the relationship between the pixel and its adjacent pixels, there is a pixel point set v = {p1, p2, K, p N}, where the mark field ω={w p1 ,w p2 ,K,w pN},w v ={1,2,K,K L ), L is the symbol set, K L is the total number of labels, and image segmentation is performed as a maximum a posteriori estimate The parameter F is expressed as a random variable, P(w) represents the segmentation result of the next instance based on F, P(w|F) represents the posterior probability of the segmentation result under this condition, P(F / w) is the probability distribution of the random variable F, P(F) is the prior probability, if when F is an instance, its probability is a certain value, P(w|F) and P(w)P(F|w) are proportional to each other, after obtaining the prior probability P(w) and the likelihood function P(F|w), it is converted to U(w|F)=U(F|w)+U(w)+U through w=argmax(P(w)P(F|w)). c ;

[0212] U(w|F) represents the posterior energy, U CRepresents a constant. The three-dimensional coordinates of each pixel point on the multi-view panoramic image are set to p(x, y, z). Each pixel point position in the multi-view panoramic image has a normal vector S(p) perpendicular to the plane. The position coordinates of each camera in the photography are set to pos(x0, y0, z0).

[0213] Calculate the normal vector of each pixel as S(p), set the line vector between the pixel and the viewpoint as a, and calculate the angle between each viewpoint and the normal of each pixel The one with the smallest angle is the one that is at the front of the search;

[0214] The cost function D(p, L(p))=|pp is selected to represent the two-dimensional plane distance between the projection point of each viewpoint on the facade and the pixel point. L(p) |, where p represents the position coordinate of the pixel point, p L(p) Represents the two-dimensional coordinates of the projection of each viewpoint on the facade, and D(P, L(P)) represents the coordinates of the projection point to the pixel point on the two-dimensional plane.

[0215] Assuming that the coordinates of the projection points of each two panoramic sphere centers are (x0, y0, z0) and (x1, y1, z1), calculate the perpendicular bisector of the line connecting the centers of the two panoramic spheres:

[0216]

[0217] Calculate the projection facade equation and the perpendicular bisector to divide the entire projection facade into several blocks. Combined with the characteristics of multi-viewpoint panoramic images, the field of view and scenery in front of you are always at the forefront when browsing the street view. Ensure that the pixel collection of each area is at the forefront. During the specific projection, ensure that adjacent projection areas have overlapping parts. Subsequent steps will further determine the projection stitching line to obtain an image that minimizes the energy function.

[0218] (4) 3D projection panoramic street scene stitching

[0219] (1) Splicing process, such as Figure 10 ;

[0220] (2) Experimental input data, such as Figure 11 Rectangular panoramic sphere;

[0221] (3) projections of single-station photos on both sides of the street, including single-station panoramic spherical projections on both sides of the street;

[0222] (4) Consider the street mosaic graph of the energy function, such as Figure 12 ;

[0223] There are many intersections along the data, which seriously affect the stitching effect, resulting in unnatural stitching. In addition, the panoramic data collection time interval is too long, and the lighting conditions of each image are different. Some parts are obviously overexposed, while others are underexposed. Therefore, the final stitched image can be evened to achieve a connection between the two panoramas. This is a new method for generating multi-view panoramas. Compared with tunnel multi-view panoramas, street view panoramas are more difficult to stitch and have richer texture information. Tunnel multi-view panoramas have better measurement performance. Tunnel panoramas better meet the characteristics of multi-view panoramas. Street view images are more like multi-view perspective panoramas. Multi-view panoramas of both scenes are based on the mathematical relationship between spatial three-dimensional points and two-dimensional scene points based on the measurement value projection surface, and then the texture resampling method is used to obtain a multi-view panorama. However, the difficulties of each scene are different. Multi-view street view stitching will be widely used in map navigation, street view browsing and other functions. Tunnel multi-view panoramas will also be widely used in disaster control and prediction of underground pipelines, tunnels and bridges. Therefore, the research of this application is very meaningful.

[0224] The comparison shows that the generation of tunnel multi-view panoramas in this application will be more widely used in surveying and mapping, because a set of mathematical methods for conversion processes can present the relationship between three-dimensional model points and two-dimensional multi-view panoramas. Street view images provide a single-station 360° panoramic sphere, and the model does not contain point information of the three-dimensional model. Therefore, street view multi-view panoramas are more commonly used in visual services with a large field of view.

Claims

1. A method for 3D reconstruction of multi-view panoramic images by projected surface measurement and stitching, characterized in that: A multi-viewpoint parallel stitching method for landscape reconstruction is established. Through parallel stitching, a seamless two-dimensional multi-viewpoint panoramic image is obtained. The multi-viewpoint panoramic image generated by two-dimensional parallel stitching is a long corridor-shaped two-dimensional image that fully covers the scenery information on both sides of the street. It supports users' continuous browsing of the landscape and has a low data volume, simple data structure and convenient browsing method. Multi-segment axis projection tunnel image stitching method: The scene is reconstructed in three dimensions through all-round pictures. The measurement value projection surface suitable for the scene is calculated based on the model. The data is then projected and resampled to generate multi-viewpoint stitching. Based on the mathematical model of data mesh to projection surface mesh and the conversion model of projection surface mesh to plane mesh, a set of mathematical relationships from 3D model scene points to 2D image planes is supervised. Through multi-segment axis projection texture resampling, the measurement relationship between the 3D scene and the 2D image plane is established. This not only meets people's visual perception requirements for multi-viewpoint panoramas, but also introduces measurement information, further broadening the use of multi-viewpoint panoramas. 3D scanning projection surface full street view stitching method: Based on the mathematical relationship between scene models, the measurement value projection facade is constructed, and the coordinate relationship model between the 360-degree spherical panorama and the spatial street view facade and the coordinate relationship model between the 360-degree cylindrical panorama and the spatial street view facade are established respectively. The mathematical relationship between the scenes ensures a strict stitching relationship between the two, and the coordinate conversion relationship between the 3D model scene and the 2D multi-viewpoint panorama is established. The street view projection data is resampled to establish a 3D projection panoramic street view stitching that can well express the measurement information of the image.

2. The method for 3D reconstruction of a multi-view panoramic image by measuring and stitching a projection surface according to claim 1, characterized in that: Multi-segment axial projection tunnel image stitching data acquisition and processing: The initial image data of the tunnel is collected by a mobile cart and uploaded to the cloud for processing. The data only contains data collected by the panoramic pan-tilt head. The cloud-based panoramic pan-tilt head image processing process is as follows: 1) Image preprocessing: including image distortion correction, image enhancement, and contrast enhancement of preprocessed images; 2) Relative orientation: Import the pre-processed image into the aerial triangulation software to automatically match the same-name points, and obtain the relative orientation result after free network adjustment; 3) Absolute orientation: Based on the relative orientation result, the absolute orientation result is obtained by adding control points to the image and jointly solving them; 4) Generate dense point cloud: Based on the results of absolute orientation, point cloud encryption is performed to obtain high-density and high-precision dense point cloud data. Each image is assigned real 3D spatial position and posture information, and each matching point also has real 3D coordinate information. Every pixel in each image corresponds to a real geographic coordinate position. 5) Output result data: Output the processed image and the absolutely oriented dense point cloud according to the actual application requirements or convert them into a specified format for archiving and future use; DP-Smart automated image modeling uses DP-Capture to collect data and the DP-Smart program automatically creates models. The DP-Smart program groups and categorizes the data, then automatically models the mesh and selects the tunnel mesh data and corresponding texture information that has been successfully constructed. The tunnel data is divided into sub-stations. The division rule is to make the angle between the axis of each tunnel section and the axis of the next tunnel section the largest. The model data divided at each station is conducive to fitting the measurement value projection surface using the regular axis.

3. The method for 3D reconstruction of multi-view panoramic images by measuring and stitching projection surfaces according to claim 1, characterized in that: Select and create a projection surface: Use mesh data of several tunnels and the corresponding texture map of the tunnels. The mesh data consists of three parts of information: triangle vertex information starting with 'v', corresponding texture coordinates starting with 'vt', and triangle information starting with 'f'. Associate the 3D model point coordinates, texture coordinates and triangles. Based on the 3D model of the tunnel, a projection surface is selected and established. The axial surface that best fits the tunnel is constructed as the measurement projection surface. Vertex information of the 3D model mesh is first extracted. Noise points that are separated from the main body of the tunnel are filtered, and the remaining points are further stored. Normals are then estimated for the filtered points to prepare data for subsequent normal segmentation. Based on the input parameters, the near-axial portions of the tunnel side and top, as well as the planar portion of the tunnel floor, are obtained. Select some data of the tunnel side and top near axis for axis fitting, set the radius range of the axis projection surface, use the vertex information in the mesh data to fit the measurement value projection surface, obtain the parameter value of the measurement value projection surface, so that the projection surface can be represented mathematically; The obtained measurement value projection surface is an axial surface. The axial projection surface contains seven parameters, of which the first three represent a point on the axis of the axial projection surface, the middle three represent the axial direction vector of the axial projection surface, and the last one represents the radius of the fitted axial surface. Knowing that any spatial axis can be determined by the axial direction, the point on the axis, and the radius, the axial projection surface is established and determined. Verify whether the fitting parameters are accurate: apply the axis fitting function in PCL to perform parameter fitting, compare the difference between the two, save the vertex information of the mesh data as a .pcd file and import it into the software for axis fitting, and compare the fitting effects of the three; In addition, the shape detector in CloudCompare is used to detect the best fitting axis for the point cloud and give the parameters of the fitting axis. The fitting parameters of the axis are given in the form of a matrix.

4. The method for 3D reconstruction of multi-view panoramic images by measuring and stitching projection surfaces according to claim 1, characterized in that: Establish a mathematical model for transforming data mesh to projection surface mesh: Let point p(X,Y,Z) be the vertex coordinates of the initial mesh, let the perpendicular line through point p intersect the projection surface at point p'(X1,Y1,Z1), and let the intersection with the axis be m(X2,Y2,Z2). Point p' is the projection point of the point on the projection surface. The seven parameters of the axis are the axial vector set to (a, b, c), the coordinates of any point on the axis are (x0, y0, z0), and the radius of the axis is uniformly set to r. The equation of the axis is: Knowing that m is a point on the axis, Y2 and Z2 are expressed by X2: Taking into account that the vector pp' formed by points p and p' is perpendicular to the axis vector, it can be further deduced that: a*(X-X2)+b*(Y-Y2)+c*(Z-Z2)=0 Equation 3 get: Point p is the vertex coordinate of the mesh data, and p' is the projection point of point p on the projection surface, resulting in: The coordinates of the projection point are: X1=X2+Δx Y1=Y2+△y Z1=Z2+Δz Equation 8 At this point, the mathematical transformation relationship from the vertices of the 3D model mesh to the vertices of the projection surface mesh is completed; Axis fitting and projection processing are performed on each tunnel mesh. There is no fixed mathematical relationship between the projection surface of each tunnel, so the projection surface mesh is processed again. In real engineering, the axes of axis A and axis B are a pair of skew lines, which are set as L1 and L respectively. The translation of the axis involves the translation and splicing of the skew lines. Let the equation of line L1 be Let the equation of line L be The direction vectors of the two straight lines are v = (a, b, c) and v1 = (a1, b1, c1). Let the direction vector of the common perpendicular be n = (x1, y1, z1). get Solve for the direction vector of the common perpendicular line, and then find the equation of the common perpendicular line based on a vertex on the axis (the vertex on the axis can be found by searching), and then combine the equation of the common perpendicular line with the equation of the other axis to find the optimal connection point of the two non-planar axes based on two non-planar straight lines; Vertical rotation of the projection mesh: Plane ABC is a plane with the projection mesh axis as its normal and passing through a point on the axis. The seven parameters of the axis are the radius r, a point on the axis (x0, y0, z0), and the axis vector (a, b, c). Write the equation of plane ABC, then calculate the rotation angle to describe the angle between the projection mesh and the z-axis. Several tunnel projection meshes are obtained, each of which has an axial direction that is consistent with the z-axis. The axial direction of the mesh projection surface after rotation is strictly checked, and the axial direction vector of the vertical axis is calculated using the axis fitting program.

5. The method for 3D reconstruction of multi-view panoramic images by measuring and stitching projection surfaces according to claim 1, characterized in that: Conversion model from projection surface mesh to plane mesh: Based on a projection surface mesh whose axis coincides with the z-axis, find the coordinate relationship between any point p on the projection surface mesh and the plane mesh. Define the coordinate system of the plane mesh with the intersection B1 of the projection surface mesh and the positive semi-axis of the y-axis as the origin, define the positive direction of the y-axis of the plane mesh with the positive direction of the z-axis, and define the x-axis direction of the plane mesh with the x-direction of the original three-dimensional coordinate system. Looking down from the positive direction of the z-axis to get a top view, the length of the horizontal coordinate of the plane projection point p is Assume that the angle between the line op connecting point p and the axis center and the positive y-axis of the xoy section is θ. The length of the ordinate of p is the z-coordinate of the projected mesh point, that is, the length of OH. Expand the projected surface mesh along the axis generatrix where point B1 is located to obtain the plane mesh. The relationship between the coordinates is as follows, completing the mathematical conversion from the projected surface mesh to the plane mesh. X4=rθ Y4=Z3 Z4=r Equation 9 When expanding the projection surface mesh along a generatrix on the axial plane, select the location of the expanded generatrix, detect the direction of the generatrix at the top of the tunnel, and make sure that the dividing line must be at the top of the tunnel; When projecting onto a plane mesh, the topological information of the triangles is removed. The triangles that the split line passes through need to be indexed and deleted in the original mesh. After the triangle on the dividing line is divided, the three vertices are located on both sides of the image, and the distance between the vertices is greater than or equal to the circumference of the projection surface. The triangle information of the projection surface mesh is traversed. For each triangle, the three vertices are pointl(x1, y1, z1), point2(x2, y2, z2), and point3(x3, y3, z3). The radius of the axis projection surface is R. The distances between the three points are calculated respectively, and the sizes of the three distances l1, l2, and l3 are compared with R. If the distance is greater than R, the dividing line passes through the triangle and the triangle is deleted. The last step is to unfold the projection surface mesh into a plane mesh projection. A fixed projection base is selected that is suitable for several tunnel sections in the scene. After the plane unfolded mesh is generated, the texture is attached and the measurement value projection is completed.

6. The method for 3D reconstruction of multi-view panoramic images by measuring and stitching projection surfaces according to claim 1, characterized in that: Multi-segment axial projection texture resampling: Construct a mathematical relationship between the plane unfolding mesh and the resulting multi-view panorama, and convert the relationship between the plane unfolding mesh and the pixel position. The origin of the image coordinates is at the upper left corner of the computer screen. Point A (m, n) represents the position of a pixel on the image. Assuming the pixel size of the image is pixel, let the coordinates of point A on the plane mesh be (x, y). The relationship between the two is: n = [x / pixel] m = [row.pixel - y] / pixel Formula 10 Find the coordinates (x, y) on the corresponding plane mesh according to the rows and columns of the multi-view panorama, and use the affine transformation matrix to inversely solve the pixel values ​​of the corresponding positions in the rows and columns of the panorama; Get an unfolded plane mesh, find the mapping relationship between the plane mesh's triangles and the texture image UV coordinates, and use affine transformation to obtain the affine transformation matrix between the plane triangles and the texture coordinates. Each triangle and the corresponding texture coordinates have an affine transformation matrix R. The pixel values ​​of the pixels contained in a triangle are further resampled through the inverse solution method. Find the minimum bounding rectangle of a triangular facet of a planar mesh, calculate the two-dimensional texture coordinates corresponding to the pixels according to the number of rows and columns of pixels contained in the minimum bounding rectangle, continuously look for the minimum circumscribed polygon of each triangular facet in a mesh plane to define the rows and columns of the search graph, use bilinear interpolation method to assign pixel values, and generate a multi-viewpoint plane image from the three-dimensional mesh and texture information. In addition, in the data processing part, software cutting tools are used to decompose the tunnel into several parts. The incisions of each tunnel section are not regular circles. The incisions of most tunnels are ellipses. Local stretching and other processing or channel transparency processing are performed on local parts of the image to establish the relationship between the pixel coordinates of the two-dimensional multi-viewpoint panorama and the three-dimensional scene coordinates.

7. The method for 3D reconstruction of multi-view panoramic images by measuring and stitching projection surfaces according to claim 1, characterized in that: Key points for capturing 3D scanning street view source data: Determine the camera's rotation angle based on the panoramic gimbal used to capture images to ensure full coverage of the scene and photo overlap. With the lens' optical center as the axis, tilt the lens upward approximately 5 degrees during capture. After acquiring the image, load the stitched image. Use software to obtain a successfully stitched panorama. Remove any irrelevant content from the image, including the capture vehicle and tripod, to create a uniformly sized rectangular panorama. Use DP-Smart to stitch the captured data.

8. The method for 3D reconstruction of multi-view panoramic images by measuring and stitching projection surfaces according to claim 1, characterized in that: Construct the measured value projection facade: Assume that the position of the main plane is the position of the multi-viewpoint facade. The row tree a is not in the main plane. The imaging of the row tree a from different viewpoints is not in the same position in the main plane. When determining the multi-viewpoint projection facade, select the facade where the building falls to ensure that the information of interest can be spliced. Select the projection facade considering the target of interest in the entire panoramic image, and then Then determine the location of the facade; Determine the projection surface through the initial image data, obtain sparse spatial point cloud data based on SFM, fit the projection facade according to the 3D scene reflected by the point cloud data, and explain the specific process of solving the spatial point cloud through the mathematical relationship between the intersection of the 3D spatial point and the spatial front of the photograph; The sparse point cloud generated by the SFM method is used to fit the measurement value projection surface. The measurement value projection surface is then solved based on the 3D scene restored from the image. At each station, a panoramic gimbal is used to acquire image data, obtain the image's focal length and initial POS information, and input the image, focal length, initial POS information, sensor size, and control points into smart3D. After initial aerial triangulation, the control points are added and the solution is recalculated to assess the aerial triangulation accuracy and produce a visual model. Determine the facade equation and the motion trajectory of the 3D camera based on the 3D model of the scene. First, import the generated 3D model into Smart3DCapture Viewer and select the corresponding coordinate system based on the location of the shooting scene. Use measurement tools to determine the coordinates of 3D points in the 3D scene. Select multiple points along the street to fit the facade equation. Then select the highest point of the building in the scene to determine the facade height. The street facade is perpendicular to the ground along the street. Fit the equation of the street facade in the xoy plane to determine the facade equation. Use Origin8 to fit the selected points. For straight streets, the facade solution is simplified to the facade along the street. The coordinates of three points are (x1, y1, y1), (x2, y2, z2), (x3, y3, z3), and the facade equation is solved as: a = 1 a(x-x1)+b(y-y1)+c(z-z1)=0 Equation 11 Where (a, b, c) represents the normal vector of the plane, and determines the facade with the best projection effect for a scene.

9. The method for 3D reconstruction of multi-view panoramic images by measuring and stitching projection surfaces according to claim 1, characterized in that: 360-degree spherical panorama and spatial street view facade coordinate relationship model: Assume that the pixel coordinates of the rectangular panoramic image are M(m,n), and the center O of the panoramic sphere corresponds to G of the panoramic image. The direction from point O to point G is defined as the y-axis of the panoramic sphere. The x-axis should be located in the equatorial plane and is required to be perpendicular to the y-axis. The direction defined by the z-axis is perpendicular to the xoy plane. The relationship between a point M(m,n) on the image plane and the corresponding mapping point P(x,y,z) on the panoramic sphere is as follows: m＝r(α+π) z=rsinβ Equation 12 Finally, the relationship between P(x,y,z) and M(m,n) is: Where W is the length of the photo and H is the width of the photo, and the transformation from the rectangular panoramic image plane to the panoramic spherical image space coordinate system is obtained, and then the transformation from the panoramic spherical image space coordinate system to the panoramic spherical image space auxiliary coordinate system is obtained; Convert from the panoramic spherical image space coordinate system to the panoramic spherical auxiliary coordinate system, use the three angles in photogrammetry to control the transformation of the coordinate system, and find the transformation relationship between the coordinates according to the transformation matrix. Assume that the coordinates of the mapping point in the panoramic spherical image space auxiliary coordinate system are P'(X, Y, Z), and the coordinates in the panoramic spherical image space coordinate system are set to P(x, y, z). The transformation matrix between the two is set to R. The transformation matrix and ω, The relationship between κ is as follows: b1=cosωsinκ b2=cosωcosκ b3=-sinω Finally, consider the coordinates P of a space object point in the panoramic spherical auxiliary coordinate system. c (X c ,Y c ,Z c ) and the world coordinate system Relationship, let the coordinate of the object point in the world coordinate be P d (X d ,Y d ,Z d ), s' is the coordinate point of the panoramic sphere center in the world coordinate system, and p represents a point on the object side; the relationship between the panoramic spherical image space auxiliary coordinate system and the world coordinate system is: X c =X d -X s AND c =And d -AND s Z c = Z d -Z s Equation 15 where X s ,Y s ,Z s is the coordinate of the photography center in the world coordinate system, P c (X c ,Y c ,Z c ) and P`(X, Y, Z) satisfy the collinear equation, and we get: The value of λ is the ratio of the distance from the image space auxiliary coordinates to the center of the panoramic sphere to the radius of the panoramic sphere. Define the radius of the panoramic sphere as 1, starting from a point P in space d (X d ,Y d ,Z d ) to the 360-degree rectangular panoramic image coordinates M(m,n): m＝r(α+π) 10. The method for 3D reconstruction of multi-view panoramic images by measuring and stitching projection surfaces according to claim 1, characterized in that: Street view projection data resampling: Given an undirected graph defined as G = (V, ε), where V represents the pixel point and ε represents the relationship between the pixel and the adjacent pixels, there is a pixel point set v = {p1, p2, K, p N }, where the mark field ω={w p1 ,w p2 ,K,w pN },w v ={1,2,K,K L ), L is the symbol set, K L is the total number of labels, and image segmentation is performed as a maximum a posteriori estimate The parameter F is expressed as a random variable, P(w) represents the segmentation result of the next instance based on F, P(w|F) represents the posterior probability of the segmentation result under this condition, P(F / w) is the probability distribution of the random variable F, P(F) is the prior probability, if when F is an instance, its probability is a certain value, P(w|F) and P(w)P(F|w) are proportional to each other, after obtaining the prior probability P(w) and the likelihood function P(F|w), it is converted to U(w|F)=U(F|w)+U(w)+U through w=argmax(P(w)P(F|w)). c ; U(w|F) represents the posterior energy, U C Represents a constant. The three-dimensional coordinates of each pixel point on the multi-view panoramic image are set to p(x, y, z). Each pixel point position in the multi-view panoramic image has a normal vector S(p) perpendicular to the plane. The position coordinates of each camera in the photography are set to pos(x0, y0, z0). Calculate the normal vector of each pixel as S(p), set the line vector between the pixel and the viewpoint as a, and calculate the angle between each viewpoint and the normal of each pixel The one with the smallest angle is the one that is at the front of the search; The cost function D(p, L(p))=|pp is selected to represent the two-dimensional plane distance between the projection point of each viewpoint on the facade and the pixel point. L (p)|, where p represents the position coordinate of the pixel point, p L(p) Represents the two-dimensional coordinates of the projection of each viewpoint on the facade, and D(P, L(P)) represents the coordinates of the projection point to the pixel point on the two-dimensional plane; Assuming that the coordinates of the projection points of each two panoramic sphere centers are (x0, y0, z0) and (x1, y1, z1), calculate the perpendicular bisector of the line connecting the centers of the two panoramic spheres: a=(x1-x0,y1-y0,z1-z0) Calculate the projection facade equation and the perpendicular bisector to divide the entire projection facade into several blocks. Combined with the characteristics of multi-viewpoint panoramic images, the field of view and scenery in front of you are always at the forefront when browsing the street view. Ensure that the pixel collection of each area is at the forefront. During the specific projection, ensure that adjacent projection areas have overlapping parts. Subsequent steps will further determine the projection stitching line to obtain an image that minimizes the energy function.

Images