Method for automatically generating historic building elevation map by using multi-mode neural network

Abstract

The invention discloses a method for automatically generating an ancient building elevation by using a multi-modal neural network, and the method comprises the steps: obtaining and preprocessing a three-dimensional inclined point cloud model and a three-dimensional laser point cloud model of a target ancient building, and obtaining the indoor and outdoor point cloud data of the ancient building; performing feature point matching and rigid body transformation fusion on the ancient building indoor and outdoor point cloud data to obtain a three-dimensional ancient building point cloud model; performing semantic segmentation on the three-dimensional ancient building point cloud model through projection and a multi-modal neural network to obtain each structure area of the ancient building; and according to each structural region of the ancient building, extracting each component contour from the three-dimensional point cloud model to generate an ancient building elevation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of ancient building facade drawing generation, and particularly relates to a method for automatically generating an ancient building facade drawing by using a multi-modal neural network. BACKGROUND

[0002] The building facade drawing is important drawing reflecting the external form, proportion, structural member distribution and decoration details of the building, and is the core basic data in building design, construction, repair, cultural relic protection, digital modeling and virtual display.

[0003] At present, the building facade drawing and the building interior drawing are mainly obtained in the following ways: one is to be surveyed and drawn manually by professional personnel, which depends on manual experience, is large in workload, low in efficiency, and is prone to subjective errors; and the other is to obtain three-dimensional point cloud data of the building by means of a three-dimensional laser scanner, a total station, a drone and the like, and then to perform segmentation, contour extraction and drawing generation manually or semi-automatically. However, the building interior drawing (i.e. a two-dimensional or three-dimensional display drawing reflecting the indoor wall surface, doors and windows, structural layout and the like) under the above process more depends on the laser point cloud measurement of the indoor space, and needs to be combined with the actual site investigation and manual post-processing.

[0004] The existing point cloud segmentation and contour extraction methods often depend on the geometric features of the point cloud, and it is difficult to achieve high-precision segmentation of the wall, railings, window mullions, carvings and other components of a complex and richly decorated ancient building, which is prone to misclassification and omission, and affects the completeness of the subsequent facade or interior drawing. The traditional process mainly focuses on structural contour information, lacks effective decoration protection and fine segmentation mechanism, so that many building component details reflecting the historical and aesthetic value (such as eaves brackets, brackets, reliefs and the like) are lost or simplified in the facade and interior drawings. The processes of data segmentation, contour extraction and drawing generation highly depend on manual operation and experience judgment, and it is difficult to achieve efficient automation, and subjective factors are prone to introduce errors. The current segmentation and facade (interior) extraction process does not utilize the mapping, texture and spatial constraint relationship between the point cloud and the image, resulting in limited accuracy in texture detail restoration and component differentiation.

[0005] Aiming at the problems existing in the prior art, a method for automatically generating an ancient building facade drawing by using a multi-modal neural network is designed. SUMMARY

[0006] Therefore, the purpose of the present application is to provide a method for automatically generating an ancient building facade drawing by using a multi-modal neural network, which can solve the above problems.

[0007] The present application provides a method for automatically generating an ancient building facade drawing by using a multi-modal neural network, comprising: Obtain and pre-process the three-dimensional oblique point cloud model and the three-dimensional laser point cloud model of the target ancient building, to obtain indoor and outdoor point cloud data of the ancient building; Fuse the indoor and outdoor point cloud data of the ancient building through feature point matching and rigid body transformation to obtain a three-dimensional ancient building point cloud model; Perform semantic segmentation on the three-dimensional ancient building point cloud model through projection and a multi-modal neural network to obtain each structural region of the ancient building; According to each structural region of the ancient building, extract the outlines of each component from the three-dimensional point cloud model to generate an ancient building elevation drawing.

[0008] The present application has the following advantages: First, through the cooperation of unmanned aerial vehicle oblique photography, panoramic image and laser point cloud, high-density collection of three-dimensional information of the ancient building is realized. Based on space threshold and region growing, a large number of low furniture, temporary obstacles and other non-building entities in the original point cloud data are removed, which affects the subsequent structure recognition.

[0009] Second, the registration accuracy is improved through feature point pairs and normal constraints, the local mismatch probability is reduced by introducing the consistency of the zenith direction and the local normal restriction, and the structure restoration ability and geometric accuracy are greatly improved by fitting enhancement for complex areas such as roofs and walls.

[0010] Third, by combining multi-angle rotation projection and Transformer deep network segmentation, multi-space information is fully expressed, the segmentation is more accurate and the details are more complete. The point cloud and pixel label space mapping and inverse mapping ensure the class label in the three-dimensional space of each point, effectively eliminating subjective intervention. The unsegmented points are automatically completed by using space KNN algorithm, ensuring the continuity of three-dimensional structure segmentation, not missing details, and enhancing the automation and robustness of the model.

[0011] Fourth, through automatic detection and completion of regional occlusion / loss, decoration protection and hierarchical processing, key components are avoided from being misclassified, lost or damaged, and the real structure of the elevation is completely restored. Through contour line graph theory completion, curve smoothing and automatic geometric alignment and straightening, continuous, smooth and standard elevation boundaries are obtained, realizing high-quality vector diagrams that can be used for digital modeling. The hierarchical control of main structure and decoration is supported, which is suitable for various downstream needs. BRIEF DESCRIPTION OF DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiment or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0013] Figure 1is a method flowchart of the embodiment.

[0014] Figure 2 is a step S3 and S4 flowchart of the embodiment.

[0015] Figure 3 is a building facade drawing obtained by the embodiment.

[0016] Figure 4 is a building facade drawing acquisition comparison drawing of the embodiment. DETAILED DESCRIPTION

[0017] For the convenience of those skilled in the art to understand, the structure of the present application will be further described in combination with the drawings of the embodiments, and it should be understood that, in the embodiments mentioned, the order of the steps can be adjusted according to actual needs, even simultaneously or partially simultaneously, except for the order specified.

[0018] As shown in Figure 1 , the embodiment of the present application provides a method for automatically generating an ancient building facade drawing by using a multi-modal neural network, comprising: S1, obtaining and preprocessing a three-dimensional inclined point cloud model and a three-dimensional laser point cloud model of a target ancient building to obtain indoor and outdoor point cloud data of the ancient building; S101, obtaining multi-angle image data of the ancient building by surrounding shooting with a drone, and processing the multi-angle image data to generate a three-dimensional inclined point cloud model of the ancient building; In this step, the multi-view image data of the building can be captured by surrounding shooting from different angles with a drone, and a three-dimensional inclined point cloud model of the outdoor of the ancient building is generated through data processing. The multi-view image data needs to meet the image overlap requirement to ensure the integrity and accuracy of the three-dimensional model.

[0019] S102, obtaining a three-dimensional laser point cloud model of the main body of the ancient building by panoramic photography and three-dimensional laser scanning, and preprocessing the three-dimensional laser point cloud model.

[0020] In this step, the three-dimensional laser point cloud model of the main body of the building is obtained by using panoramic photography and three-dimensional laser scanning technology. The panoramic image is used for color assignment of the three-dimensional point cloud, and the three-dimensional laser scanning provides the spatial structure information of the indoor of the ancient building. In order to improve the accuracy of the building structure extraction, the indoor laser point cloud is preprocessed as follows: S1021, performing preliminary point cloud filtering on the three-dimensional laser point cloud model based on a height threshold to remove low furniture and temporary obstacles, and the calculation formula is as follows: , wherein, represents the point cloud data after preliminary point cloud filtering, represents the point cloud data of the three-dimensional laser point cloud model, represents the coordinate value of the point cloud data on the z-axis, represents a height threshold value; S1022 extracts regular building surfaces of the three-dimensional laser point cloud model by removing low furniture and temporary obstacles through a region growing and plane fitting method, and removes irregular objects; S1023 performs semantic classification on the three-dimensional laser point cloud model of the irregular object through a point cloud semantic segmentation model.

[0021] In this step, the point cloud semantic segmentation model can use RandLA-Net (Random Sampling and Local Aggregation Network) and the like, which can effectively retain key structural points such as walls, columns, and floors of the three-dimensional laser point cloud model.

[0022] S2 fuses the indoor and outdoor point cloud data of the ancient building through feature point matching and rigid body transformation to obtain a three-dimensional ancient building point cloud model; S201 selects the same name feature point pairs in the three-dimensional inclined point cloud model and the three-dimensional laser point cloud model, solves the rotation matrix R and the translation vector T, and the calculation formula is as follows: , wherein, represents the coordinates of the feature points of the three-dimensional laser point cloud model, represents the coordinates of the feature points of the three-dimensional inclined point cloud model; In this step, by selecting the same name point pairs with obvious features and easy to identify (such as corner points, texture points or edge points, etc.) in the two groups of point clouds, the rotation and translation are solved to preliminarily align the two groups of point clouds, and a rough registration result is obtained, which ensures that the point clouds are roughly aligned as a whole, and prepares for subsequent local detailed correction.

[0023] S202 queries the neighborhood k nearest points in each pair of same name feature points in the two point clouds respectively using Kd-Tree, and uses the least square method to fit the plane where each neighborhood point cloud is located to minimize the sum of squared errors, to obtain the plane equation, and the calculation formula is as follows: , , wherein, , , represents the coordinates of the i-th neighborhood point, represents the plane parameters, the laser point cloud normal vector: , the inclined point cloud normal vector: , and M represents the number of points in the neighborhood of a certain feature point participating in fitting; In this step, the local plane of the feature point neighborhood is used to extract normal information, enhance the understanding of the local geometric structure, and extract the normal vector, which helps to judge the consistency of local area registration, surface morphology, etc., and prepare for subsequent point pair direction consistency filtering.

[0024] S203 sets the zenith direction as the Z axis and calculates the angle between the normal vector of the laser point cloud and the zenith direction. , the angle between the normal vector of the tilted point cloud and the zenith direction , and retain The characteristic points are calculated as follows: , , in, represents the unit vector in the zenith direction, Indicates the angle tolerance, setting the threshold to ensure that the normal directions of feature points are similar; In this step, the ground normal (zenith direction) is used as the standard to measure the consistency of the feature point's normal upward / downward direction to avoid the same point pointing in different directions due to noise or mismatching. If the normals of the two points are consistent, they are retained; otherwise, they are discarded. This improves the robustness and accuracy of subsequent registration and reduces the interference of abnormal data.

[0025] S204 uses the retained valid feature point pairs to re-solve the rotation matrix based on the minimum residual criterion and the translation vector T, the calculation formula of the optimization objective function is as follows: , in, Represents the feature point coordinates of a valid 3D laser point cloud model, Represents the coordinates of the feature points of a valid 3D tilted point cloud model, and N represents the number of point clouds; In this step, the rigid transformation (transformation matrix) is calculated again using the point pairs with verified directional consistency and high discrimination. The optimization objective function is the same as that of the aforementioned S201, but the point pairing is more accurate, resulting in a higher precision registration.

[0026] S205 through laser point cloud normal vector and the tilted point cloud normal vector Calculate the angle between the normal vectors of building walls , the calculation formula is as follows: ; S206 calculates the geometric center of the building, compares the symmetry deviation, and fits the wall and ground planes to calculate the residual standard deviation , the calculation formula is as follows: , Wherein, represents the zenith direction unit vector, represents the number of point clouds; In this step, the angle between the normal vectors of the building wall is calculated to verify the verticality of the building structure, and the angle is required to be close to 90 degrees. At the same time, the geometric center of the building is calculated, and the symmetry deviation is compared to find the geometric deformation or offset of the building, and the wall and ground planes are fitted, and the abnormal deformation is found by monitoring the structural verticality. Further, the wall / ground detection quality is judged to provide a basis for subsequent reconstruction and building disease analysis.

[0027] S207 will register and verify the point cloud data, and according to the local structure characteristics of the ancient building, it will be adaptively fused according to the region; S2071 carries out corresponding point cloud data fitting according to the roof area type of the ancient building, and the calculation formula is as follows: , Wherein, , ,f represents the quadratic surface coefficient, , represents the two-dimensional plane projection coordinates of a certain point, represents the zenith direction space coordinate value corresponding to a certain point; In this step, the roof modeling is very critical to the overall style and structural safety of the ancient building, so the type fusion can significantly improve the authenticity of the roof geometric expression and the fitting quality of the point cloud model. Adaptive classification and fitting can avoid rough fitting of all roofs into a single plane, reducing modeling errors.

[0028] The structure of the slope roof is usually composed of several planes (such as four-pitch roof), and the multi-plane (several intersecting planes) fitting method can be used to restore the true structure of the roof.

[0029] The overall platform roof is relatively flat, and a single plane can be used for fitting to improve the degree of simplification and fitting accuracy.

[0030] The dome / curved roof is a continuous curved surface, which needs to be fitted with a quadratic surface, a spherical surface, an ellipsoidal surface, or even a high-order free-form surface to more truly capture its spatial form.

[0031] S2072 carries out corresponding segmented fitting of point cloud data and / or irregular surface processing of point cloud data according to different wall region types of the ancient building; In this step, irregular structures are processed in areas with windows, cavities, uneven surfaces, and local deformations (such as settlement, bulging, and cracking). Some walls can be treated as a whole plane, while local areas such as window sills and steps can be treated as local curved surfaces or polyhedrons. By segmenting, each segment is fitted with the most suitable geometric surface (plane or curved surface), and the concave and convex changes of the detail parts are preserved. Many ancient building walls have complex real conditions, such as windows, reinforcement, repair, or uneven foundation deformation. This segmentation and irregular surface processing ensures that the model has good adaptability and restoration ability to real structural changes. Avoid distortion or loss of details caused by overall plane fitting, and improve the reliability of subsequent three-dimensional analysis (such as disease detection and deformation analysis).

[0032] S2073 The point cloud fusion weight is distributed according to the complexity of the regional structure of the ancient building, and the calculation formula is as follows: , wherein, represents a weight factor, represents the three-dimensional coordinates of the i-th feature point of the target point cloud, represents the i-th feature point coordinates of the source point cloud (the point cloud that needs to be registered and rigidly transformed).

[0033] In this step, higher weights should be given to areas with complex structures or significant geometric features of the ancient building (such as roof ridges, corners, and carved parts) to pursue higher fitting accuracy. Data sparsity or non-critical areas can be moderately reduced in weight to optimize overall modeling efficiency. The target point cloud is the registered point cloud, generally the reference system of registration, and the source point cloud is the point cloud that needs to be registered and rigidly transformed.

[0034] As shown in Figure 2 , after obtaining the three-dimensional ancient building point cloud model, the point cloud model needs to be split into components. The specific steps are as follows: S3 performs semantic segmentation on the three-dimensional ancient building point cloud model through projection and a multi-modal neural network to obtain each structural region of the ancient building; S301 projects the point cloud data of the three-dimensional ancient building point cloud model into a graphics space and constructs a mapping relationship between the point cloud data and the projected image; In this step, to ensure multi-view expression effect, projection operations are performed at different rotation angles. Specifically as follows: S3011 rotates the point cloud data around three axes (X, Y, Z) by 45° each time, generating 8 projection images at different angles on each axis. The calculation formula of the rotation matrix is: , , , , wherein, represents a rotation matrix rotating by an angle of around the X-axis, represents a rotation matrix rotating by an angle of around the Y-axis, represents a rotation matrix rotating by an angle of around the Z-axis; In this step, 8 projection images of different angles are generated on each axis, each rotating by 45°, to ensure that the geometric features of the point cloud are adequately reflected and exhibited at different viewing angles. The rotation operation adopts the Euler angle rotation method.

[0035] S3012 maps the rotated point cloud data onto a two-dimensional image to obtain a projection image, and calculates the image coordinates, with the calculation formula being as follows:

[0036] wherein, represents a pixel / projection coordinate of the two-dimensional image, represents a three-dimensional space coordinate of a point in the rotated point cloud, represents a focal length of the camera in the x direction, represents a pixel value of the x coordinate of the image principal point, represents a focal length of the camera in the y direction, represents the y coordinate of the image principal point.

[0037] In this step, each point of the rotated point cloud data is located in the 2D projection, while a mapping table (image coordinate) is reserved to facilitate backtracking to the original point cloud data after segmentation.

[0038] S302 records the mapping relationship between the position of each pixel point in the projection image and the corresponding point in the point cloud during the projection process, with the calculation formula being as follows: , wherein, represents a projection mapping matrix, represents an index number of the point cloud data; In this step, a projection mapping matrix or a mapping table records the corresponding relationship between the pixel points in the two-dimensional projection image and the original point cloud data. Saving the corresponding relationship between each 2D pixel (projected position) and its original point in the 3D point data facilitates subsequent semantic segmentation and label mapping tasks.

[0039] S303 uses a deep neural network based on the Transformer structure to perform semantic segmentation on the projection image, and obtains the structure class label of each pixel. The calculation formula is as follows: , wherein, represents an attention function, represents a query vector, represents a key vector, represents a value vector, represents the dimension of the vector, and softmax represents a normalization function; In this step, the image segmentation network with powerful Transformer structure is used to perform pixel-level semantic segmentation on multiple projection images, and the semantic class (such as wall, roof) to which each pixel belongs is output. The advantage of the Transformer is that it can process large-scale context information and has strong expression ability, and the precision is high.

[0040] S304 uses the mapping relationship and the class label of each pixel to remap the class label of each pixel back to the original point cloud data, and realizes the pre-segmentation of the point cloud. The calculation formula is as follows: , wherein, represents that the label of pixel i is assigned to the i-th point of the original point cloud data through the projection mapping matrix, represents a projection mapping matrix, represents the class of each pixel after segmentation; In this step, the pixel label obtained by the network segmentation is transmitted back through the mapping table and is assigned to the corresponding three-dimensional point in the source point cloud, so as to realize the pre-segmentation of the three-dimensional space. If a point is labeled multiple times by projection, voting or the class with the highest confidence can be adopted.

[0041] S305 uses a classification algorithm to perform secondary segmentation on the point cloud data that has not been mapped, and obtains the corresponding structure type.

[0042] In this step, some points (such as occlusion and projection outside) may not be segmented during projection. These “unlabeled points” are classified by a classification algorithm (K-Nearest Neighbor (KNN), Support Vector Machine (SVM), Random Forest (RF)) to classify the point cloud data. For example, the KNN (K-Nearest Neighbor) algorithm: find the nearest K points with segmentation labels around it, and see which class has the most, then assign it to that class. This ensures that the three-dimensional space segmentation is continuous and complete, and the details are more complete, further improving the integrity and accuracy of the segmentation. The specific steps are as follows: S3051 calculates the point cloud data that has not been mapped and all the point cloud data that has been mapped​ The distance between the point and the nearest point is calculated, and the nearest point is selected as the neighbor point The calculation formula is as follows: , Wherein, The Euclidean distance is represented; S3052 For each structure category, the number of sample categories in the field is counted, and the sample category with the most times is the category of the point, and the calculation formula is as follows: , Wherein, The number of points selected as neighbors is represented, The structure category is represented.

[0043] In this step, the distance from the remaining points to be segmented to all segmented points is calculated, and the nearest K is selected as the neighbor. Count the categories to which the K neighbors belong, and take the most as the category of the current unsegmented point.

[0044] S4 According to the structure region of the ancient building, the outline of each component is extracted from the three-dimensional point cloud model to generate the ancient building elevation drawing.

[0045] S401 Detect and remove the occlusion and missing area of each structure region of the ancient building; In this step, in order to avoid misjudging temporary occlusion (such as eaves, balcony) as building boundary or misprocessing scanning blind area as structure opening in the contour extraction process, the occlusion and data missing area in the point cloud data need to be explicitly identified. The specific steps are as follows: S4011 By calculating the point cloud density and the change of the normal vector angle in the local area, comparing the density threshold and the normal threshold, identifying the occlusion area and the missing area, and the calculation formula of the point cloud density and the change of the normal vector angle is as follows: , , Wherein, The local point density of the i-th point in the point cloud is represented, The number of points in the sphere with point i as the center and radius r is represented, The volume of the three-dimensional sphere, i.e. the neighborhood size, is represented, The angle between the normal vectors of points i and j is represented, The normal vector of points i and j is represented; In this step, the density and normal vector variation characteristics of a local region around a point in the point cloud are calculated through a spherical neighborhood or k-nearest neighbors. Occlusion regions typically have abrupt changes in point density but continuous normals, while missing regions typically have sparse points and unclear normal directions. The density and normal thresholds can be adaptively adjusted to accommodate varying scene complexities.

[0046] S4012 performs multi-angle scanning on the same component area. If data is missing from all viewing angles in the area, it can be determined as real data missing. If data is missing from only some viewing angles in the area, it can be marked as an occlusion area. The calculation formula is as follows: , in, represents the data integrity judgment result of point i, represents the i-th point of the point cloud, Indicates a specific viewing angle number of the scan / collection. represents the set of all points collected from the k-th perspective, Represents the set of all points collected from the j-th perspective; In this step, if part of the view is blocked by local components, that is, if some of the view is missing data, it will be marked as an occluded area. This step can introduce dense point cloud voxel reconstruction technology to improve the judgment accuracy, so as to better distinguish the root cause of the data problem and reduce manual post-repair.

[0047] S4013 removes the projection of the blocked area before the outline is generated, and interpolates and completes the missing area based on the building structure rules. The calculation formula is as follows: , in, Represents about the axis Symmetric mirror complement point set, represents the coordinate value of the symmetry axis in the x direction, Represents the point set around the missing area that needs to be completed.

[0048] In this step, identified obstructing elements are projected out of the primary viewing direction to avoid inadvertent inclusion in wall contour extraction. Areas with missing data can be filled by interpolating the surrounding structural contours, for example, by leveraging the building's axial symmetry and planar continuity to reconstruct missing contours.

[0049] S402 selects the components that are not completely blocked from the point cloud data after removing occlusions and filling in missing points, converts them into a two-dimensional image, and uses morphological dilation to enhance the structure in the two-dimensional image, connecting the broken points to form a more complete area. The calculation formula is as follows: , in, denotes a binary image region to be processed, denotes a structuring element, denotes the position of the structuring element B after moving with the pixel point z as the origin; In this step, the morphological dilation image processing method is used to make the projection structure coherent, which is conducive to subsequent contour detection.

[0050] S403 performs decoration component protection on the projection image of the outer contour of the ancient building; In this step, since there are many decorative components in the ancient building, in order to prevent the decorative components (such as railings, reliefs, and window traceries) from being lost or deformed in the morphological processing, the following protection mechanism is introduced: S4031 selects a corresponding morphological structuring element according to the size and directionality of the decorative details of the ancient building; In this step, in the image morphological processing, the structuring element (also known as kernel, template, or mask) is a small, pre-defined pixel window used to slide on the image to guide the erosion, dilation, opening operation, closing operation, and other operations. For example, a smaller or directionally inclined convolution kernel, appropriate structuring element parameters are used for subsequent morphological processing.

[0051] S4032 identifies the decorative detail area through a detection algorithm, generates a protection mask image for the decorative detail area, and the calculation formula is as follows: , wherein, denotes a gradient amplitude image or a protection mask image, denotes an input two-dimensional projection image; In this step, the decorative components are marked separately to ensure that the subsequent operations only erode / dilate the main structure and weaken the processing of the decorative details, achieving spatial partition protection S4033 performs erosion and dilation operations on the contour image containing the protection mask, and removes isolated noise to retain edge features;

[0052] wherein, denotes a two-dimensional projection image to be processed ， denotes a morphological structuring element; In this step, spatial mask + opening operation is used to remove irrelevant noise, and since the decorative components have a mask, the decorative detail contour can be retained to avoid being eroded and dilated.

[0053] S4034 extracts a coarse contour layer and a decorative detail layer respectively, and synthesizes the coarse contour layer and the decorative detail layer to obtain a complete contour image.

[0054] In this step, the main structure and decorative features are not interfered with each other by using layered processing and post superposition, and the structure precision and details are obtained.

[0055] S404 uses a boundary tracking algorithm to extract the external contour of all connected regions of the complete contour image, counts the area, aspect ratio, and position information of each contour, and classifies the contours into main contours and secondary contours according to the semantic label to obtain main / secondary contour coordinate point sets, and the calculation formula is as follows:

[0056] Among them, represents the maximum area contour, that is, the main contour, represents the contour point set extracted by the ith connected region, and argmax represents the index of the maximum . represents the area surrounded by the ith contour. In this step, the contours are divided into main contours such as main structure boundaries and secondary contours such as doors, windows, and decorative components.

[0057] S405 optimizes the main / secondary contour coordinate point sets for topological consistency to obtain optimized contours; S4051 uses a graph theory method to repair the breakpoints in the main / secondary contour coordinate point sets, and the calculation formula is as follows: , Among them, path represents the possible connection path between breakpoints, represents two end points in the contour, represents a set composed of all possible edges in the graph, represents the weight of , which is usually the Euclidean distance between the two end points; In this step, the graph theory method such as the shortest path, edge weight pruning, etc. In actual point cloud and image contour extraction, due to acquisition noise, occlusion, segmentation errors, the obtained contour is often not closed, that is, “has breakpoints” “loop is not complete”, which will affect the subsequent CAD modeling and semantic analysis. Automatically complete all breaks to make the external contour a “closed continuous curve / polygon”, which is beneficial for further processing or output to CAD.

[0058] S4052 smoothes the irregular boundaries of the main / secondary contour coordinate point sets by a smoothing algorithm; ​In this step, the boundary extracted from the pixel or point cloud rough contour often has a "sawtooth" effect or irregular fluctuations, which will distort the facade drawing and is not convenient for architectural design or engineering drawing. The smoothing algorithm can use Bezier curve or B-spline algorithm. Bezier curve and B-spline are a commonly used smooth interpolation or fitting curve, which can pass through or close to the original point set, but produces a smooth and beautiful boundary line.

[0059] S4053 fusion building geometry prior automatic straightening alignment main contour line, get the optimized contour, the calculation formula is as follows:

[0060] Among them, is the straight line distance parameter in polar coordinates, which represents the distance from the origin to the straight line, is the two-dimensional point coordinate, represents the angle between the straight line normal vector and the x-axis.

[0061] In this step, the building facade contour is mostly horizontal, vertical, symmetric and other structural features, but when the point cloud is automatically segmented or projected, the extracted contour often has slight skew, offset or bending, which does not completely conform to the actual or architectural drawing specification. Therefore, it is necessary to combine the building geometry prior (straight line preference, horizontal and vertical alignment) to automatically straighten and align the main contour line, enhance the overall aesthetic appearance and actual building style consistency.

[0062] This embodiment uses morphological erosion and contour detection to extract each component contour from the three-dimensional point cloud model to generate a traditional building facade drawing. It can also directly identify the outer contour of the building through a convolutional neural network (CNN) based image segmentation network.

[0063] S406 polygon fitting and node reconstruction are performed on the optimized contour to generate an editable vector facade drawing.

[0064] In this step, the contour is converted to a node sequence by polygon fitting (such as the Ramer-Douglas-Peucker algorithm) and simplified to a standard CAD curve or polygon. The vector output format supports SVG, DXF or architectural CAD format, which can be directly read by any CAD, BIM, architectural design platform as a high-quality vector facade drawing for subsequent component identification, semantic modeling, repair analysis, etc.

[0065] As Figure 3 and Figure 4As shown, according to the label mapping technology between different modalities, the image segmentation and the point cloud semantic segmentation result are successfully fused, the accuracy and integrity of the elevation map generation are improved. The K nearest neighbor algorithm is used for optimization on the unsegmented points, and the integrity and accuracy of the segmentation result are improved. The present application combines point cloud segmentation and image processing technology, can automatically generate accurate and editable building elevation map, effectively retains the decorative details of the ancient building, reduces the need for manual intervention.

[0066] Those skilled in the art will appreciate that embodiments of the application can be provided as methods, systems, or computer program products. Accordingly, the application can be embodied in the form of complete hardware embodiments, complete software embodiments, or embodiments combining software and hardware aspects. Moreover, the application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) having computer usable program code embodied thereon.

[0067] The present application is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more functions specified in the flowchart or flows and / or blocks. Figure 1 an apparatus that implements one or more functions specified in the flowchart or flows and / or blocks.

[0068] These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including instruction apparatus, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more functions specified in the flowchart or flows and / or blocks. Figure 1 an apparatus that implements one or more functions specified in the flowchart or flows and / or blocks.

[0069] These computer program instructions can also be loaded into a computer or other programmable data processing apparatus, so that a series of operation steps are performed on the computer or other programmable data processing apparatus to produce a computer-implemented process, so that the instructions executed on the computer or other programmable data processing apparatus provide a process for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more functions specified in the flowchart or flows and / or blocks. Figure 1 an apparatus that implements one or more functions specified in the flowchart or flows and / or blocks.

[0070] It should be noted that in the claims the reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps other than those listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. It is further stated that a plurality of items, structural parts and / or components can be presented in a claim in the format of "at least one", for example "one or more". The application can be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the claims the word "comprising" does not exclude other elements being added. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The word "first", "second" and the like do not imply any order, quantity, or importance, but rather are used to distinguish one element from another. The terms "preferably", "preferred", "preferably", "desired", "desirable", "suitable", "in an embodiment", or "in some embodiments" are no means to restrict or narrow the application or any exemplified embodiment. The term "exemplary" in the context of this specification means "serving as an example, instance, or illustration;" and not "preferred" over other examples. The disclosure of at least one feature recited in any claim can not imply that all of the features and or objects have to be combined to provide the technical effect that is described in connection with that feature or object. The term "about" means "approximately" or "circa", that is, "almost exactly" that which it describes, but does not have to be exact. The term "consisting essentially of" to avoid the use of the term "comprising" to limit the scope of a claim. The term "consisting essentially of" means including, but not limited to, any elements or ingredients listed.

[0071] Although preferred embodiments of the application have been described herein, it will be apparent to those skilled in the art that various modifications can be made within the scope of the application. Accordingly, it is intended that all such modifications come within the scope of the claims appended hereto and their equivalents.

[0072] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described herein.

[0073] In the present application, unless otherwise explicitly specified and limited, the terms "mounting", "connection", "connecting", "fixing" and the like should be interpreted in a broad sense, for example, can be fixed connection, can also be detachable connection, or integral; can be mechanical connection, can also be electrical connection; can be directly connected, can also be indirectly connected through an intermediate medium, can be internal communication of two elements or interaction relationship between two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0074] In the description of the present application, the description of the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present application, the illustrative description of the above terms should not be understood as necessarily referring to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. Furthermore, those skilled in the art can combine and combine the different embodiments or examples described in the present application and the features of the different embodiments or examples without contradiction.

Claims

1. A method for automatically generating ancient building facades using a multimodal neural network, characterized in that: include: Acquire and pre-process the 3D tilt point cloud model and 3D laser point cloud model of the target ancient building to obtain the indoor and outdoor point cloud data of the ancient building; The indoor and outdoor point cloud data of the ancient building are fused through feature point matching and rigid body transformation to obtain a three-dimensional point cloud model of the ancient building; The semantic segmentation of the 3D ancient building point cloud model was performed through projection and multimodal neural network to obtain the structural areas of the ancient building. According to the structural areas of the ancient building, the outlines of each component are extracted from the 3D point cloud model to generate the facade drawing of the ancient building.

2. The method for automatically generating ancient building facades using a multimodal neural network according to claim 1, characterized in that: The acquisition and preprocessing of the three-dimensional tilted point cloud model and the three-dimensional laser point cloud model of the target ancient building to obtain indoor and outdoor point cloud data of the ancient building includes: Acquire multi-angle image data of the ancient building through drone surround shooting, and process the multi-angle image data to generate a three-dimensional inclined point cloud model of the ancient building; The three-dimensional laser point cloud model of the main body of the ancient building is obtained through panoramic photography and three-dimensional laser scanning, and the three-dimensional laser point cloud model is preprocessed.

3. The method for automatically generating ancient building facades using a multimodal neural network according to claim 1, characterized in that: The method of fusing the indoor and outdoor point cloud data of the ancient building through feature point matching and rigid body transformation to obtain a three-dimensional point cloud model of the ancient building comprises: Select the same-name feature point pairs in the 3D tilted point cloud model and the 3D laser point cloud model, and solve the rotation matrix R and translation vector T. The calculation formula is as follows: ， in, Represents the coordinates of the feature points of the 3D laser point cloud model, Represents the coordinates of feature points of a 3D tilted point cloud model; For each pair of feature points with the same name, use Kd-Tree to query the k nearest points in the neighborhood in the two point clouds, and use the least squares method to fit the plane where each neighborhood point cloud is located so that the sum of squared errors is minimized to obtain the plane equation. The calculation formula is as follows: ， ， in, , , represents the coordinates of the i-th neighborhood point, Represents plane parameters, laser point cloud normal vector: , tilt the point cloud normal vector: , M represents the number of points in the neighborhood of a feature point involved in fitting; Assume the zenith direction is the Z axis and calculate the angle between the normal vector of the laser point cloud and the zenith direction , the angle between the normal vector of the tilted point cloud and the zenith direction , and retain The characteristic points are calculated as follows: ， ， in, represents the unit vector in the zenith direction, Indicates the angle tolerance, setting the threshold to ensure that the normal directions of feature points are similar; Using the retained valid feature point pairs, the rotation matrix is ​​re-solved based on the minimum residual criterion and the translation vector T, the calculation formula of the optimization objective function is as follows: ， in, Represents the feature point coordinates of a valid 3D laser point cloud model, Represents the coordinates of the feature points of a valid 3D tilted point cloud model, and N represents the number of point clouds; By laser point cloud normal vector and the tilted point cloud normal vector Calculate the angle between the normal vectors of building walls , the calculation formula is as follows: ； Calculate the geometric center of the building, compare the symmetry deviation, and fit the wall and ground planes to calculate the residual standard deviation , the calculation formula is as follows: ， in, represents the unit vector in the zenith direction, Indicates the number of point clouds; The registered and verified point cloud data are adaptively fused by region according to the local structural characteristics of the ancient buildings.

4. The method for automatically generating ancient building facades using a multimodal neural network according to claim 1, characterized in that: The semantic segmentation of the three-dimensional ancient building point cloud model by projection and multimodal neural network is performed to obtain the structural areas of the ancient building, including: Project the point cloud data of the three-dimensional ancient building point cloud model into the graphic space, and construct a mapping relationship between the point cloud data and the projected image; During the projection process, the mapping relationship between the position of each pixel in the projected image and the corresponding point in the point cloud is recorded. The calculation formula is as follows; ， in, represents a projection mapping matrix, Indicates the index number of the point cloud data; Use a deep neural network based on the Transformer structure to perform semantic segmentation on the projected image and obtain the structural category label of each pixel. The calculation formula is as follows: ， in, represents the attention function, represents the query vector, represents the key vector, represents a value vector, Represents the dimension of the vector, and softmax represents the normalization function; Using the mapping relationship and the category label of each pixel, the category label of each pixel is remapped back to the original point cloud data to achieve pre-segmentation of the point cloud. The calculation formula is as follows: ， in, Indicates that the label of pixel i is assigned back to the first pixel of the original point cloud data through the projection mapping matrix. Points, represents the projection mapping matrix, Indicates the category of each pixel after segmentation; For the point cloud data that is not mapped, a classification algorithm is used to perform secondary segmentation to obtain the corresponding structure type.

5. The method for automatically generating ancient building facades using a multimodal neural network according to claim 4, characterized in that: The step of projecting the point cloud data of the three-dimensional ancient building point cloud model into the graphic space and establishing a mapping relationship between the point cloud data and the projected image includes: The point cloud data is rotated around three axes (X, Y, Z) by 45° each time, generating 8 projection images at different angles on each axis. The calculation formula of the rotation matrix is: ， ， ， ， in, Indicates rotation around the X axis The rotation matrix of the angle, Indicates rotation around the Y axis The rotation matrix of the angle, Indicates rotation around the Z axis The rotation matrix of the angle; Map the rotated point cloud data onto a two-dimensional image to obtain a projected image and calculate the image coordinates. The calculation formula is as follows: ， in, represents the pixel / projection coordinates of a two-dimensional image, ( ) represents the three-dimensional space coordinates of the point in the rotated point cloud, represents the focal length of the camera in the x direction, represents the pixel value of the x-coordinate of the principal point of the image, represents the focal length of the camera in the y direction, Indicates the y coordinate of the principal point of the image.

6. The method for automatically generating ancient building facades using a multimodal neural network according to claim 4, characterized in that: For the unmapped point cloud data, a classification algorithm is used for secondary segmentation, and the corresponding structure types include: Calculate unmapped point cloud data With all the mapped point cloud data points distance, select the one with the smallest distance The calculation formula is as follows: ， in, represents the Euclidean distance; For each structural category, the number of times the category of samples in the statistical field appears is counted. The category with the largest number of sample categories is the category of the point. The calculation formula is as follows: ， in, Indicates the number of points selected as neighbors, Indicates the structural category.

7. The method for automatically generating ancient building facades using a multimodal neural network according to claim 1, characterized in that: The method of extracting the outline of each component from the three-dimensional point cloud model according to each structural area of ​​the ancient building to generate the facade drawing of the ancient building includes: Detect and remove occluded and missing areas in various structural areas of ancient buildings; Filter out the components that are not completely blocked from the point cloud data by removing occlusions and filling missing points, convert them into a two-dimensional image, use morphological dilation to enhance the structure in the two-dimensional image, connect the broken points, and form a more complete area. The calculation formula is as follows: ， in, represents the binary image area to be processed, Represents a structural element, Indicates the position of the structural element B after moving with pixel point z as the origin; Protect the decorative components of the projected image of the ancient building's outer contour; The boundary tracking algorithm is used to extract the external contours of all connected areas of the complete contour image. The area, aspect ratio, and location information of each contour are counted. The contours are then classified into primary and secondary contours with reference to the semantic labels. The primary / secondary contour coordinate point sets are obtained. The calculation formula is as follows: ， in, represents the contour with the largest area, i.e. the main contour, Represents the contour point set extracted from the i-th connected region, and argmax represents returning the value that makes The biggest one Index represents the area enclosed by the i-th contour; Perform topological consistency optimization on the primary / secondary contour coordinate point set to obtain the optimized contour; Perform polygon fitting and node reconstruction on the optimized outline to generate an editable vector elevation drawing.

8. The method for automatically generating ancient building facades using a multimodal neural network according to claim 7, characterized in that: The detection and removal of blocked and missing areas in various structural areas of the ancient building include: By calculating the point cloud density and the change in the normal vector angle in the local area, and comparing the density threshold and the normal threshold, the occluded area and the missing area are identified. The calculation formula for the point cloud density and the change in the normal vector angle is as follows: ， ， in, represents the local point density of the i-th point in the point cloud, represents the number of points within a sphere with radius r and centered at point i. represents the volume of the three-dimensional sphere, i.e. the size of the neighborhood, represents the angle between the normal vectors of point i and point j, Represents the normal vector of point i, j; Perform multi-angle scanning on the same component area. If data is missing from all viewing angles in the area, it can be determined as real data missing. If data is missing from only some viewing angles in the area, it can be marked as an occlusion area. The calculation formula is as follows: , in, represents the data integrity judgment result of point i, represents the i-th point of the point cloud, Indicates a specific viewing angle number of the scan / collection. represents the set of all points collected from the k-th perspective, Represents the set of all points collected from the j-th perspective; The occluded area is projected out before the outline is generated, and the missing area is interpolated and completed based on the building structure rules. The calculation formula is as follows: ， in, Represents about the axis Symmetric mirror complement point set, represents the coordinate value of the symmetry axis in the x direction, Represents the point set around the missing area that needs to be completed.

9. The method for automatically generating ancient building facades using a multimodal neural network according to claim 7, characterized in that: The protection of decorative components of the projected image of the ancient building's outer contour includes: According to the scale and directionality of the decorative details of ancient buildings, corresponding morphological structural elements are selected; The decoration detail area is identified through the detection algorithm, and a protection mask image is generated for the decoration detail area. The calculation formula is as follows: ， in, represents the gradient magnitude map or protection mask image, Represents the input two-dimensional projection image; Perform erosion and dilation operations on the contour image containing the protection mask, remove isolated noise and retain edge features; ， in, Represents the two-dimensional projection image to be processed ， Represents morphological structural elements; The coarse contour layer and the decoration detail layer are extracted separately, and then the coarse contour layer and the decoration detail layer are synthesized to obtain a complete contour image.

10. The method for automatically generating ancient building facades using a multimodal neural network according to claim 7, characterized in that: The topological consistency optimization of the primary / secondary contour coordinate point set to obtain the optimized contour includes: Use graph theory to repair the breakpoints in the primary / secondary contour coordinate point set. The calculation formula is as follows: ， Among them, path represents the possible connection path between breakpoints. Represents the two endpoints in the table outline, represents the set of all possible edges in the graph, express The weight of , usually the Euclidean distance between the two endpoints; Smoothing the irregular boundaries of the primary / secondary contour coordinate point sets using a smoothing algorithm; Integrating the building geometry prior, the main contour lines are automatically straightened and aligned to obtain the optimized contour. The calculation formula is as follows: ， in, is the straight line distance parameter in polar coordinates, which represents the distance from the straight line to the origin. are the two-dimensional point coordinates, Represents the angle between the normal vector of the line and the x-axis.

Images