A method and system for automatically stratifying high-rise building oblique photography models
The automatic layering method for high-rise building oblique photography models uses a convolutional neural network to detect and convert window key points, fit contour planes, and cluster based on elevation, addressing low efficiency in manual stratification and improving accuracy and speed.
Patent Information
- Application Number
- CN202211274533.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-10-18
- Publication Date
- 2024-12-24
- Estimated Expiration
- 2042-10-18
AI Technical Summary
The existing method for extracting elevation information from oblique photography models of high-rise buildings relies heavily on manual labor, leading to low work efficiency in stratifying building layers.
An automatic layering method using a convolutional neural network to detect window key points, construct a three-dimensional model, fit connection points to obtain contour planes, project and convert coordinates, and use mean clustering to establish stratification layers based on elevation information.
Automatically stratifies high-rise building models, enhancing work efficiency by reducing manual intervention and improving the accuracy and speed of layer extraction.
Smart Images
Figure CN115496908B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of real-scene three-dimensional Chinese construction, and in particular relates to an automatic layering method and system for an oblique photography model of a high-rise building. Background Art
[0002] Realistic 3D is a digital virtual space that reflects and expresses human production, life and ecological space in a real, three-dimensional and time-sequential manner. It is a new type of basic surveying and mapping standardized product and an important part of the country's new infrastructure construction. It provides a unified spatial foundation for economic and social development and the informatization of various departments. Realistic 3D is usually divided into terrain level, city level and component level according to the expression content.
[0003] Oblique photography technology acquires rich high-resolution textures of the top and side views of buildings by synchronously collecting images from different perspectives, and generates a real 3D city model by combining advanced positioning, fusion, modeling and other technologies. It can not only truly reflect the ground conditions, but also obtain the side texture information of buildings.
[0004] In the construction of real-life 3D China, 3D real estate is an important component of the basic geographic entity database. During the construction of the 3D real estate database, due to historical reasons, there is no accurate elevation information, so it is necessary to extract elevation information from the oblique photography model. Among them, automated extraction of sub-levels is a key step in realizing the construction of a 3D real estate database.
[0005] At present, the extraction of artificial building layering mainly relies on the human eye to recognize the layered information of the house in the oblique photography model, and generates the layered surface of the house by marking it one by one, but this method requires a lot of manpower and material resources. Summary of the invention
[0006] In order to solve the above problems, the present invention proposes a method for automatically stratifying a high-rise building oblique photography model, which can solve the problem of low work efficiency when stratifying a building. The method is as follows:
[0007] Based on the convolutional neural network, the original image obtained from the oblique photography is detected to obtain all the key points of the windows in the original image and extract the contours of all the windows;
[0008] Constructing an oblique photography three-dimensional model and generating connection points, wherein the connection points are used to record the correlation relationship between the coordinate value of a certain point in the three-dimensional model space and the image coordinates of the same-name point in the original image;
[0009] Extract all the connection points in the window contours from the original image, and use a fitting algorithm to fit the connection points in the window contours to obtain a contour plane;
[0010] Project the connection point onto the contour plane, obtain its projected coordinates, establish a conversion relationship between the original image pixel points of the oblique photography and the projection points of the contour plane, convert the coordinates of the key points of the window into the three-dimensional model space using the conversion relationship, and mark the coordinates of the center of gravity of each key point of the window after conversion as the center point of the window;
[0011] According to the elevation information, the center points of windows on the same floor are obtained through the mean clustering method and the sub-layers are established;
[0012] According to the elevation values of each layer, a vertical plane of the z-axis is made to obtain the inclined model layer. The data sandwiched between two inclined model layer surfaces are cut to achieve automatic stratification.
[0013] Furthermore, the original image obtained from the oblique photography is detected based on the convolutional neural network to obtain all the key points of the windows in the original image, specifically:
[0014] First, the four corner points of each window in the original image are marked using a calibration program. The coordinates of the four corner points of each window are recorded and converted into proportional values based on the size of the entire image with the width and height as the base.
[0015] Then, a window four-corner point detection model is built based on the convolutional neural network;
[0016] Finally, the original image obtained from oblique photography is input into the four-corner point detection model to obtain the coordinates of the four corner points of all windows.
[0017] Furthermore, the backbone network of the convolutional neural network adopts a classic convolutional structure plus a fully connected layer, and the output layer uses a fully connected layer.
[0018] Furthermore, the fitting algorithm adopts the RANSAC algorithm.
[0019] Furthermore, the conversion relationship is:
[0020]
[0021] Among them, X A , Y A , Z A To represent the corresponding coordinates on the contour plane after the connection point is transformed; f , γ are the photographic focal length and scale factor of the internal orientation element of the original image; x, y are the image plane coordinates of the image point of the original image; X S , Y S , Z S is the object space coordinate of the camera station; a i , b i , c i(i=1,2,3) are the 9 direction cosines composed of the 3 exterior azimuth elements of the original image.
[0022] Furthermore, the mean clustering calculation process is as follows:
[0023] Setting a classification threshold for the center point of the window, dividing the maximum value and the minimum value of the z coordinate according to the interval of the classification threshold, and obtaining a classification centroid with the classification threshold as the interval;
[0024] Calculate the distance from the coordinate point to each centroid, and classify the window center points into the set to which the centroid is closest.
[0025] Recalculate the classification centroid of each set after classification;
[0026] If the distance between the recalculated centroid and the original centroid is less than the classification threshold, it is considered that convergence has been achieved and clustering is complete;
[0027] If the distance between the new centroid and the original centroid changes greatly and exceeds the classification threshold, the above steps are repeated using the new centroid until convergence.
[0028] The present invention also provides an automatic layering system for high-rise building oblique photography models, the system comprising:
[0029] The contour extraction module detects the original image obtained from oblique photography based on the convolutional neural network, obtains the key points of all windows in the original image, and extracts the contours of all windows;
[0030] Obtaining a contour plane module, constructing an oblique photography three-dimensional model, generating connection points, wherein the connection points are used to record the correlation between the coordinate value of a point in the three-dimensional model space and the image coordinates of the same-name point in the original image, extracting the connection points in all window contours from the original image, and fitting the connection points in all window contours using a fitting algorithm to obtain a contour plane;
[0031] A module for obtaining the center point of a window projects the connection point onto the contour plane, obtains the coordinates after projection, establishes a conversion relationship between the original image pixel point of the oblique photography and the projection point of the contour plane, converts the coordinates of the key points of the window into the three-dimensional model space using the conversion relationship, and marks the coordinates of the center of gravity of each key point of the window after conversion as the center point of the window;
[0032] The automatic stratification module obtains the center points of the windows on the same floor and establishes stratification layers based on the elevation information through the mean clustering method. The vertical plane of the z-axis is made according to the elevation values of each stratification layer to obtain the inclined model stratification layer. The data sandwiched between the two inclined model stratification layers are cut to achieve automatic stratification.
[0033] This method proposes an automatic layering method for high-rise building oblique photography models, which can extract layered surfaces from the high-rise building oblique photography model to automatically layer artificial buildings, effectively solving the problem of low work efficiency when layering houses. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.
[0035] Figure 1 A framework diagram of an automatic layering method for high-rise building oblique photography models.
[0036] Figure 2 shows the extracted outline of the house windows.
[0037] Figure 3 This is an example diagram of the connection points.
[0038] Figure 4 For connecting the dots infographic.
[0039] Figure 5 This is an example diagram of the results of the RANSAC plane fitting algorithm.
[0040] Figure 6 Layered cut surface fitting diagram for windows.
[0041] Figure 7 This is a display of the real-life 3D effect of Z-axis layer cutting.
[0042] Figure 8 This is a block diagram of an automatic layering system for high-rise building oblique photography models. DETAILED DESCRIPTION
[0043] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0044] In this embodiment, a method for automatically layering a high-rise building oblique photography model is demonstrated. The system block diagram is as follows: Figure 1 As shown, the method comprises the following steps:
[0045] In the first step, the original image obtained from oblique photography is detected based on a convolutional neural network to obtain the key points of all windows in the original image and extract the contours of all windows.
[0046] In Figure 2, (a) is the original oblique image, and (b) is the window outline image, where the marked yellow polygon is the window outline extracted from the original oblique photography image.
[0047] The window outline extraction in the original image uses the key point detection technology based on machine learning. The main work includes three parts: calibration of the key points (i.e. corner points) of the window position in the original image, model training, and model application. The specific contents are as follows:
[0048] Original image calibration: The four corner points of the window in the original image are marked using a calibration program to form an irregular window calibration quadrilateral. The coordinates of the four corner points of the quadrilateral are recorded separately and converted into proportional values based on the width and height according to the size of the entire image.
[0049] Model training: The commonly used CNN network structure is used to build a window four-corner point detection model: the backbone network uses a classic convolutional structure plus a fully connected layer, and the output layer uses a fully connected layer to construct 8 coordinates (x, y coordinates) output values based on 4 coordinate points. The network structure is as follows:
[0050] a. Resample the model input image size (resampling size is 416x416);
[0051] b. The output is four coordinate points of the window frame, each of which contains two coordinates, x and y. The fully connected layer of the model output part is adjusted to 8 output parameters, representing the coordinate values of the upper left corner, upper right corner, lower right corner, and lower left corner respectively;
[0052] c. Send the annotated source image into the model, train the model using the error back propagation principle and save it as a model structure in pb format.
[0053] Model application: Use the open source opencv4 open source library to load the trained pb format model. When using it, input the original image of oblique photography and the program will output the coordinates of the four corner points of the detected window.
[0054] The second step is to use the existing oblique photography modeling technology to construct an oblique photography three-dimensional model and generate connection points, which are used to record the association between the coordinate value of a point in the three-dimensional model space and the image coordinates of the same-name point in the original image.
[0055] The connection points are high-density connection points, and their density should be no less than 30 / ㎡. Figure 3 This is a connection point example diagram. Figure 4 For connecting the dots infographic.
[0056] The third step is to extract all the connection points within the window contours from the original image, and use the fitting algorithm to fit the connection points within the window contours to obtain the contour plane.
[0057] If the number of connection points is less than 3, the plane where the window contour line is located cannot be determined, and this contour line is discarded and does not participate in subsequent calculations. When the number of connection points is greater than 3, the RANSAC algorithm is used to fit the plane where the window contour line is located, and the plane equation Ax + By + Cz + D = 0 of the plane where the contour line is located is calculated. The number of connection points in the test data is much greater than 3, so the RANSAC algorithm is used to fit the window plane. The maximum number of iterations is selected as 100, and the threshold is selected as 10. The fitting results are as follows:
[0058] Step (3) extracts all connection points inside the window contour line from the original image. If the number of connection points is less than 3, the plane where the window contour line is located cannot be determined, and this contour line is discarded and does not participate in subsequent calculations. When the number of connection points is greater than 3, the RANSAC algorithm is used to fit the plane where the window contour line is located, and the plane equation of the plane where the contour line is located is calculated. Figure 5 It is an example diagram of the results of the RANSAC plane fitting algorithm, and the main process is as follows:
[0059] a. Determine the maximum number of iterations n = 100.
[0060] b. Perform the following operations within the current iteration number i:
[0061] 1) Randomly select three points from the extracted point set to form a plane;
[0062] 2) Calculate the four parameters ABCD in the plane equation Ax + By + Cz + D = 0 and form the plane equation;
[0063] 3) Calculate the distance value d from all the remaining points to the plane;
[0064] 4) Screen the distance d from the point to the plane according to the threshold threshold (taking the point cloud sampling distance). When d < threshold, it is considered that the point is in the window plane, and the counter is incremented by 1;
[0065] 5) Compare the result of the current counter with the result of the previous counter. When the value of the current counter is larger, it is considered that the fitted plane is better than the previously fitted plane, and it is considered the best plane iterated to the current time and is retained. Otherwise, the previously extracted plane is used for the next calculation.
[0066] c. After the iteration ends, output the plane equation of the best fit.
[0067] In the fourth step, project the connection points onto the fitted contour line plane to obtain their projected coordinates. Establish the conversion relationship between the pixel points of the original oblique photography image and the projected points of the contour line plane (i.e., the homologous points of the points in the three-dimensional space), and use this conversion relationship to convert the coordinates of the key points of the detected oblique image window to the three-dimensional model space. Denote the centroid coordinates of the converted key points as the window center point.
[0068] First, calculate the projection point of the connection point on the contour plane. The calculation formula is as follows:
[0069]
[0070]
[0071]
[0072] A, B, C, and D represent the parameters of the plane equation obtained in the third step, respectively. 0， y 0， z0 represents the original coordinate, X A , Y A , Z A The coordinates of the connection point projected onto the contour plane.
[0073] Secondly, establish the coordinate conversion relationship between oblique photography and the same-name points projected onto the contour surface in three-dimensional space:
[0074]
[0075] Among them, x, y are the image plane coordinates of the original image point;
[0076] f , γ is the photographic focal length and scale factor of the internal orientation element of the original image;
[0077] X S , Y S , Z S is the object space coordinate of the camera station;
[0078] a i , b i , c i (i = 1, 2, 3) are the 9 direction cosines composed of the 3 exterior azimuth elements of the image.
[0079] The image plane coordinate system is a right-handed plane coordinate system with the principal point as the origin, represented by o-xy, and used to represent the position of the image point on the film. However, in practical applications, the right-handed plane coordinate system P-XY with the intersection of the frame mark lines as the origin is often used, which is called the frame mark plane coordinate system. The image plane coordinate system o-xy is a two-dimensional coordinate system with the geometric center o of the image as the origin, and the x and y axis directions are parallel to the edge lines of the image frame. The image space coordinate system is used to describe the spatial position of the image point and the projection center. The origin of the image space coordinate system is the projection center S of the image, and the positive z axis of the coordinate is the opposite direction of the photographic direction and coincides with oS. The axis parallel to the x and y axes on the film through point S is the x and y axis of the image space coordinate system, which together with the z axis form an image space rectangular coordinate system S-xyz. In this coordinate system, the z coordinate of each image point is equal to the photographic principal distance f0 (when there is no special requirement, it is generally replaced by the photographic focal length f), but the sign is negative.
[0080] Using the above formula, the coordinates of the four corner points of the window outline (arranged clockwise from the upper left corner) are calculated as follows:
[0081] Table 1 Example of the four-corner coordinates of contour points
[0082] Roll Call X coordinate Y coordinate Z coordinate <![CDATA[P1(x1,y1,z1)]]> 355160.73 2977578.78 104.56 <![CDATA[P2(x2,y2,z2)]]> 355163.36 2977578.77 104.49 <![CDATA[P3(x3,y3,z3)]]> 355161.43 2977578.86 101.32 <![CDATA[P4(x4,y4,z4)]]> 355163.94 2977578.77 101.65
[0083] Finally, the center of gravity coordinates are calculated using the center of gravity calculation formula:
[0084] X=(x1+x2+x3+x4) / 4
[0085] Y=(y1+y2+y3+y4) / 4
[0086] Z=(z1+z2+z3+z4) / 4
[0087] The calculated center of gravity coordinates are (355162.365, 2977578.80, 103.00). Repeat the above process to complete the center of gravity coordinates, i.e., center coordinates, of all windows.
[0088] The fifth step is to obtain the center points of windows on the same floor and establish a layered surface based on the elevation information using the mean clustering method.
[0089] The coordinates of the center point of the window are shown in Table 2.
[0090] Table 2 Midpoint coordinates example
[0091] Roll Call X coordinate Y coordinate Z coordinate Pcen1 355162.36 2977578.80 103.00 Pcen2 355163.50 2977578.80 103.21 Pcen3 355164.95 2977578.78 102.98 Pcen4 355165.79 2977579.05 103.55
[0092] The mean clustering calculation process is as follows:
[0093] 1) Set the center point classification threshold to 0.2m, divide the maximum and minimum values of the z coordinate at an interval of 0.2, and obtain the classification centroid with an interval of 0.2.
[0094] 2) Calculate the distance from all coordinate points to each centroid, and classify them into the set to which the centroid they are closest to, and classify all center points.
[0095] 3) Recalculate the classification centroid of each set after classification.
[0096] 4) If the distance between the recalculated centroid and the original centroid is less than the threshold of 0.2m, it is considered that convergence has been achieved and clustering is complete.
[0097] 5) If the distance between the new centroid and the original centroid changes significantly and exceeds the threshold of 0.2m, repeat steps 2-4 using the new centroid until convergence.
[0098] The window center point classification obtained by the mean clustering algorithm is as follows Figure 6 shown.
[0099] The sixth step is to make a vertical plane of the z-axis according to the elevation values of each layer to obtain the inclined model layer, and cut the data sandwiched between two inclined model layers to achieve automatic stratification.
[0100] The middle position between the two adjacent layers of windows is the layering surface for the upper and lower floors of the house. The vertical plane of the z-axis is made according to the elevation value of each layer, which is the layering surface of the inclined model. The data between the two layers is cut using the layering information to achieve the purpose of automatic layering. The real-life 3D effect of the Z-axis layering is shown as follows: Figure 7 shown.
[0101] Figure 8 A high-rise building oblique photography model automatic stratification system is also demonstrated, the system comprising:
[0102] The contour extraction module detects the original image obtained from oblique photography based on the convolutional neural network, obtains the key points of all windows in the original image, and extracts the contours of all windows;
[0103] Obtaining a contour plane module, constructing an oblique photography three-dimensional model, generating connection points, wherein the connection points are used to record the correlation between the coordinate value of a point in the three-dimensional model space and the image coordinates of the same-name point in the original image, extracting the connection points in all window contours from the original image, and fitting the connection points in all window contours using a fitting algorithm to obtain a contour plane;
[0104] A module for obtaining the center point of a window projects the connection point onto the contour plane, obtains the coordinates after projection, establishes a conversion relationship between the original image pixel point of the oblique photography and the projection point of the contour plane, converts the coordinates of the key points of the window into the three-dimensional model space using the conversion relationship, and marks the coordinates of the center of gravity of each key point of the window after conversion as the center point of the window;
[0105] The automatic stratification module obtains the center points of the windows on the same floor and establishes stratification layers based on the elevation information through the mean clustering method. The vertical plane of the z-axis is made according to the elevation values of each stratification layer to obtain the inclined model stratification layer. The data sandwiched between the two inclined model stratification layers are cut to achieve automatic stratification.
[0106] According to the embodiments of the present invention as described above, these embodiments do not describe all the details in detail, nor do they limit the invention to the specific embodiments described. According to the above description, many modifications and changes can be made. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can make good use of the present invention and the modified use based on the present invention. The present invention is only limited by the claims and their full scope and equivalents.
Claims
1. A method for automatically stratifying a high-rise building oblique photography model, characterized in that: The method includes the following steps: Detect the original images obtained from oblique photography based on a convolutional neural network to obtain all the key points of the windows in the original images, and extract the contour lines of all the windows; Construct an oblique photography three-dimensional model and generate connection points. The connection points are used to record the association relationship between the coordinate values of a certain point in the three-dimensional model space and the image coordinates of the corresponding point in the original image. The connection points are high-density connection points, and their density should not be less than 30 per square meter; Extract the connection points within all the window contour lines from the original images, and use a fitting algorithm to fit the connection points within all the window contour lines to obtain the contour plane. If the number of connection points is less than 3, the plane where the window contour line is located cannot be determined, and this contour line is discarded and does not participate in subsequent calculations. When the number of connection points is greater than 3, use the RANSAC algorithm to fit the plane where the window contour line is located, and calculate the plane equation of the contour plane; The main process is as follows: a. Determine the maximum number of iterations n = 100; b. Perform the following operations within the current iteration number i: 1) Randomly select three points from the extracted point set to form a plane; 2) Calculate the four parameters A, B, C, and D in the plane equation Ax + By + Cz + D = 0 to form the plane equation; 3) Calculate the distance values d of all the remaining points to the plane; 4) Screen the distance d of the points to the plane according to the threshold threshold (taking the point cloud sampling distance). When d < threshold, it is considered that the point is within the window plane, and the counter is incremented by 1; 5) Compare the result of the current counter with the result of the previous counter. When the value of the current counter is larger, it is considered that the fitted plane is better than the previous fitted plane, and it is considered the best plane iterated to the current time and is retained. Otherwise, use the previously extracted plane for the next calculation; c. After the iteration ends, output the plane equation with the best fit; Project the connection points onto the fitted contour plane to obtain their projected coordinates. Establish the conversion relationship between the pixel points of the original oblique photography image and the projected points on the contour plane, and use this conversion relationship to convert the coordinates of the key points of the windows in the detected oblique image to the three-dimensional model space. Record the centroid coordinates of the converted key points as the window center points; According to the elevation information, obtain the window center points on the same layer through the mean clustering method and establish a stratified plane; Make a vertical plane along the z-axis according to the elevation values of each stratified plane to obtain the inclined model stratified planes, and cut the data sandwiched between two inclined model stratified planes to achieve automatic stratification; The detection of all the key points of the windows in the original images obtained from oblique photography based on the convolutional neural network is specifically as follows: First, use a calibration program to label the four corner points of each window in the original image, record the coordinates of the four corner points of each window respectively, and convert them into proportional values based on the width and height according to the size of the entire image; Then, construct a window four-corner point detection model based on the convolutional neural network; Finally, input the original images obtained from oblique photography into the four-corner point detection model to obtain the coordinates of the four corner points of all the windows; The backbone network of the convolutional neural network adopts a classic convolutional structure plus a fully connected layer, and the output layer uses a fully connected layer; The fitting algorithm adopts RANSAC algorithm; The conversion relationship is: ， Among them, X A , Y A , Z A To represent the corresponding coordinates on the contour plane after the connection point is transformed; f , γ are the photographic focal length and scale factor of the internal orientation element of the original image respectively; x, y are the image plane coordinates of the image point of the original image; X S , Y S , Z S is the object space coordinate of the camera station; a i , b i , c i (i=1,2,3) are the 9 direction cosines composed of the 3 exterior azimuth elements of the original image; The mean clustering calculation process is as follows: Setting a classification threshold for the center point of the window, dividing the maximum value and the minimum value of the z coordinate according to the interval of the classification threshold, and obtaining a classification centroid with the classification threshold as the interval; Calculate the distance from the coordinate point to each centroid, and classify the window center points into the set to which the centroid is closest. Recalculate the classification centroid of each set after classification; If the distance between the recalculated centroid and the original centroid is less than the classification threshold, it is considered that convergence has been achieved and clustering is complete; If the distance between the new centroid and the original centroid changes greatly and exceeds the classification threshold, the above steps are repeated using the new centroid until convergence.
2. A high-rise building oblique photography model automatic stratification system for implementing the method described in claim 1, characterized in that: The system comprises: The contour extraction module detects the original image obtained from oblique photography based on the convolutional neural network, obtains the key points of all windows in the original image, and extracts the contours of all windows; Obtaining a contour plane module, constructing an oblique photography three-dimensional model, generating connection points, wherein the connection points are used to record the correlation between the coordinate value of a point in the three-dimensional model space and the image coordinates of the same-name point in the original image, extracting the connection points in all window contours from the original image, and fitting the connection points in all window contours using a fitting algorithm to obtain a contour plane; A module for obtaining the center point of a window projects the connection point onto the contour plane, obtains the coordinates after projection, establishes a conversion relationship between the original image pixel point of the oblique photography and the projection point of the contour plane, converts the coordinates of the key points of the window into the three-dimensional model space using the conversion relationship, and marks the coordinates of the center of gravity of each key point of the window after conversion as the center point of the window; The automatic stratification module obtains the center points of the windows on the same floor and establishes stratification layers based on the elevation information through the mean clustering method. The vertical plane of the z-axis is made according to the elevation values of each stratification layer to obtain the inclined model stratification layer. The data sandwiched between the two inclined model stratification layers are cut to achieve automatic stratification.