A Method and System for Automatic Extraction of Grid Model Components for Streetscape Facade Renovation

Abstract

This invention relates to the field of image data processing technology, specifically to a method and system for automatically extracting components from a gridded model for streetscape facade renovation. The method includes: acquiring a 3D point cloud and a 2D image of the streetscape facade; constructing a 3D grid model composed of multiple grid regions by gridding the 3D point cloud; determining the projection regions of the grid regions in the 2D image to obtain the semantic probability vectors of the grid regions; determining the adjacent edges and geometric discontinuities between adjacent grid regions; determining the overlapping pixel information of the component image edges and adjacent edges projected onto the corresponding 2D images; and combining the semantic probability vectors, geometric discontinuities, and overlapping pixel information on both sides of the adjacent edges to obtain the boundary coefficients of the adjacent edges, thereby obtaining the component segmentation result of the 3D grid model. Through the technical solution of this invention, the boundaries of each component in the 3D grid model are accurately located, achieving better segmentation results.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, specifically to a method and system for automatically extracting gridded model components for streetscape facade renovation. Background Technology

[0002] Streetscape facade renovation refers to the updating, beautification, functional optimization, or historical restoration of the building facades on both sides of urban streets. A realistic 3D mesh model is reconstructed after collecting data from real streetscapes using technologies such as oblique photogrammetry, laser scanning, or drone photography. This model realistically records the geometry, texture, and spatial relationships of building components. Building facades include functional and decorative elements such as windows, doors, and billboards. To extract these components, algorithms are used to automatically identify, segment, and classify these independent component objects from the complex overall realistic mesh model. The 3D mesh model reconstructed from real street scene facades is a continuous surface. The depth differences between components such as window glass and window frames, and window frames and walls in the street scene facades are small, and there are no obvious topological boundaries on the continuous surface of the 3D mesh model. The algorithm must identify and repair the boundaries from subtle geometric changes (such as small depth differences between glass and walls, and abrupt changes in normal), which is extremely difficult. Current component extraction schemes are prone to inaccurate boundary positioning or over-segmentation. Summary of the Invention

[0003] To address the technical problem of low component segmentation accuracy in current 3D mesh model component extraction schemes for streetscape facades, the present invention aims to provide an automatic extraction method and system for mesh model components used in streetscape facade renovation. The specific technical solution adopted is as follows: This invention provides a method for automatically extracting gridded model components for streetscape facade renovation, the method comprising: Obtain 3D point cloud and 2D images of street scene facades, and construct a 3D mesh model composed of multiple mesh regions by meshing the 3D point cloud; Determine the projection region of the grid region in the 2D image and obtain the semantic probability vector of the grid region based on the projection region. Determine the adjacent edges between adjacent grid regions in the 3D grid model and the geometric discontinuities on both sides of the adjacent edges. Determine the edge of the component image in the two-dimensional image, and determine the overlapping pixel information of the adjacent edge projection with the edge pixel of the component image in the corresponding two-dimensional image; The boundary coefficients of adjacent edges are obtained by combining the semantic probability vectors on both sides of the adjacent edges, the geometric discontinuities, and the overlapping pixel information. Based on the boundary coefficients of each adjacent edge, the segmentation boundary and component segmentation results of the three-dimensional mesh model are obtained.

[0004] Furthermore, the step of constructing a 3D mesh model composed of multiple mesh regions by meshing the 3D point cloud includes: The triangulation algorithm is used to establish connections between adjacent 3D point clouds to obtain multiple grid regions; The average curvature of the grid region in the 3D mesh model is determined by using the neighborhood curvature parameters of each vertex of the grid region.

[0005] Further, determining the projection region of the grid region in the two-dimensional image and obtaining the semantic probability vector of the grid region based on the projection region includes: Projecting the vertices of the grid region onto the pixel coordinate system of the corresponding two-dimensional image yields the projected region in the two-dimensional image. Determine the probability of each pixel in the projection area corresponding to the semantic category of each component and obtain the semantic probability vector of the pixel; The semantic probability vector of the grid region is obtained based on the semantic probability vector of each pixel in the projection region.

[0006] Further, obtaining the semantic probability vector of the grid region based on the semantic probability vector of each pixel in the projection region includes: The initial semantic probability vector of the grid region is obtained by averaging the semantic probability vectors of each pixel in the projection region. By utilizing the shooting perspective relationship between the grid region and the corresponding two-dimensional image, the quality weight of the grid region in the corresponding two-dimensional image representing perspective distortion is determined; By utilizing the quality weights of the grid regions in different corresponding two-dimensional images and the initial semantic probability vectors, the target semantic probability vector of the grid regions is obtained.

[0007] Furthermore, the step of determining the quality weight of the grid region in the corresponding two-dimensional image, based on the shooting perspective relationship between the grid region and the corresponding two-dimensional image, includes: Determine the projected area of ​​the grid region in the corresponding two-dimensional image, and determine the vector angle between the normal vector of the grid region and the shooting direction vector of the corresponding two-dimensional image; Using the projected area and the vector angle, the quality weight of the grid region in the corresponding two-dimensional image representing perspective distortion is calculated.

[0008] Furthermore, determining the geometric discontinuities on both sides of adjacent edges includes: Determine the unit normal difference and the average curvature difference of the grid regions on both sides of an adjacent edge; The geometric discontinuity on both sides of the adjacent edge is obtained by combining the unit normal vector difference and the average curvature difference.

[0009] Further, determining the overlapping pixel information between the adjacent edge projection and the edge pixels of the component image in the corresponding two-dimensional image includes: Determine the projected edges of adjacent edges in the corresponding 2D image, and determine the number of overlapping pixels between the projected edges and the edges of the component image; The edge pixel density of adjacent edges on the corresponding two-dimensional image is obtained by using the number of overlapping pixels and the length of the projected edge.

[0010] Further, the step of combining the semantic probability vectors on both sides of the adjacent edge, the geometric discontinuity, and the overlapping pixel information to obtain the boundary coefficient of the adjacent edge includes: Using the geometric discontinuities and the overlapping pixel information, the degree of semantic difference between the components on both sides of the adjacent edge is obtained; The boundary coefficient of the adjacent edge is obtained by combining the difference in semantic probability vectors on both sides of the adjacent edge and the degree of semantic difference of the components.

[0011] Furthermore, the step of obtaining the segmentation boundary and component segmentation results of the 3D mesh model based on the boundary coefficients of each adjacent edge includes: Each grid region is clustered according to its semantic probability vector to obtain multiple clusters, and adjacent clusters are used as source points and sink points respectively. The capacity of adjacent edges is obtained by using the boundary coefficient of adjacent edges. The obtained source points, sink points and capacity are then used to obtain the segmentation boundary and component segmentation results of the 3D mesh model using the minimum cut algorithm.

[0012] This invention also provides an automatic extraction system for gridded model components used in streetscape facade renovation, the system being used to implement the automatic extraction method for gridded model components used in streetscape facade renovation as described in any of the preceding claims; the system includes: The mesh construction module is used to acquire the 3D point cloud and 2D image of the street scene facade, and to construct a 3D mesh model composed of multiple mesh regions by meshing the 3D point cloud. The correlation analysis module is used to determine the projection region of the grid region in the 2D image and obtain the semantic probability vector of the grid region based on the projection region; determine the adjacent edges between adjacent grid regions in the 3D grid model and the geometric discontinuities on both sides of the adjacent edges; determine the component image edges in the 2D image; and determine the overlapping pixel information of the adjacent edge projection and the component image edge pixels in the corresponding 2D image. The boundary localization module is used to combine the semantic probability vectors on both sides of the adjacent edge, the geometric discontinuity, and the overlapping pixel information to obtain the boundary coefficient of the adjacent edge, and to obtain the segmentation boundary and component segmentation result of the three-dimensional mesh model based on the boundary coefficient of each adjacent edge.

[0013] The present invention has the following beneficial effects: To automatically extract components from a grid model for streetscape facade renovation, this invention uses 3D point cloud technology to collect 3D point cloud data and 2D images of building facades. A 3D mesh model is constructed using the collected 3D point cloud data. The faces (mesh regions) of the 3D mesh model are projected onto the corresponding 2D images, and semantic probability analysis of the faces is performed by analyzing the components in the 2D images, thus obtaining the 3D semantic probability vector of the faces. Furthermore, the boundary coefficients of adjacent edges are obtained by combining the geometric changes between faces with the overlap of adjacent edges with edge pixels in the 2D images and the 3D probability vector of the faces, thereby obtaining the segmentation boundary and completing the semantic segmentation result of the 3D mesh model. Compared to traditional algorithms that only automatically identify components based on the 3D mesh model, this invention obtains the 3D semantic probability vector of the 3D mesh model by projecting the 3D mesh model onto the 2D image, and reflects the semantic features of components in the 2D image into the 3D mesh model for correlation analysis. This makes the segmentation result more semantically meaningful, more accurately locating the boundaries of each component in the 3D mesh model from at least two dimensions: semantic features and morphological change features, thus achieving a more ideal segmentation effect. Attached Figure Description

[0014] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 The flowchart illustrates the steps of an automatic extraction method for gridded model components in streetscape facade renovation, as provided in an embodiment of the present invention. Figure 2 This is a detailed flowchart of step S2 in a method for automatically extracting gridded model components for streetscape facade renovation, provided in an embodiment of the present invention. Figure 3 This is a detailed flowchart of step S23 in a method for automatically extracting gridded model components for streetscape facade renovation provided in an embodiment of the present invention. Figure 4 This is a detailed flowchart of step S4 in a method for automatically extracting gridded model components for streetscape facade renovation provided in an embodiment of the present invention. Figure 5 A detailed flowchart of step S4 in a method for automatically extracting gridded model components for streetscape facade renovation provided in another embodiment of the present invention; Figure 6This is a schematic diagram of the hardware operating environment of the automatic extraction device for gridded model components used in streetscape facade renovation, as described in an embodiment of the present invention. Figure 7 This is a schematic diagram of the frame structure of the automatic extraction system for gridded model components used in streetscape facade renovation, which is part of an embodiment of the present invention. Detailed Implementation

[0016] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a method for automatically extracting gridded model components for streetscape facade renovation based on the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0017] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0018] The following description, in conjunction with the accompanying drawings, details the specific scheme of the automatic extraction method for gridded model components used in streetscape facade renovation provided by this invention.

[0019] Example 1: For the automatic extraction method of gridded model components for streetscape facade renovation provided by this invention, please refer to [link to relevant documentation]. Figure 1 The diagram illustrates a flowchart of the automatic extraction method for gridded model components for streetscape facade renovation provided by an embodiment of the present invention.

[0020] The method for automatically extracting gridded model components for streetscape facade renovation includes: Step S1: Obtain the 3D point cloud and 2D image of the street scene facade, and construct a 3D mesh model composed of multiple mesh regions by meshing the 3D point cloud. The data acquisition process for street view facades can be achieved by constructing an integrated multi-sensor mobile acquisition system. This system is equipped with a high-precision synchronous positioning and attitude determination module, a multispectral imaging array, a lidar, and an ambient light sensor. Before acquisition, a UWB (Ultra Wide Band) integrated indoor and outdoor positioning base station network is deployed around the target street view facade, and highly reflective coded markers are affixed to key feature points on the facade itself, thereby establishing an absolute spatial reference frame with centimeter-level accuracy. During operation, the data acquisition system performs multiple rounds of time-series data acquisition on the target facade along a preset path: First, the UAV adaptively adjusts its trajectory based on the preset facade geometric complexity model, performing multi-view oblique photography and LiDAR scanning within a range of 20 to 50 meters from the facade, simultaneously acquiring high-resolution RGB (Red, Green, Blue) images, near-infrared band images, and high-density point clouds to obtain two-dimensional images and three-dimensional point clouds; at the same time, the ground mobile platform performs supplementary scanning of the bottom and detailed areas of the facade at a close ground distance (2 to 10 meters), and its structured light depth camera and high dynamic range camera are specifically used to capture the microscopic geometric and textural features of fine components such as window frames and carvings; the spatiotemporal data of all sensors are generated through hardware-level synchronous triggering and real-time fusion with the base station network to generate a multimodal data stream with strictly aligned timestamps; during the process, the data acquisition system continuously records the ambient light intensity and color temperature, and automatically identifies and segments data of dynamic obstructions such as vehicles and pedestrians by performing differential calculations on the data collected within the same time series. For data gaps identified in time-series analysis due to long-term occlusion, the acquisition system will autonomously plan and drive the near-field acquisition unit equipped with supplementary lighting to perform a targeted, active, and detailed acquisition of the area to ensure data integrity. Finally, all raw data, along with synchronously generated positioning and orientation files, environmental parameter logs, and occlusion probability maps, are encapsulated to form a standardized data package that can be directly input into subsequent processing procedures.

[0021] Specifically, step S1 involves constructing a 3D mesh model composed of multiple mesh regions by meshing the 3D point cloud, including: The triangulation algorithm is used to establish connections between adjacent 3D point clouds to obtain multiple grid regions; The average curvature of the grid region in the 3D mesh model is determined by using the neighborhood curvature parameters of each vertex of the grid region.

[0022] In this embodiment, when automatically extracting components from the mesh model, since the street view facade contains building structures of different materials, feature extraction of the three-dimensional mesh model (hereinafter referred to as "mesh model") is required first when extracting components. For each data point in the acquired 3D point cloud data of the street scene facade, the system uses a triangulation algorithm to establish connections between adjacent points based on the spatial distribution of the point cloud, forming a network composed of countless triangles, thus creating a grid. Each triangular mesh region in the mesh model is extracted and denoted as a face, and each face consists of three vertices and three edges; For each facet, obtain the normal vector and the unit normal vector of that normal vector, and record it as the facet's orientation; To extract components from a meshed model, it is necessary to analyze the texture changes in different parts of the mesh model, which requires extracting the curvature data of the facets. For each vertex in the meshed model and the vertices in its one-ring neighborhood (all vertices sharing an edge with that vertex), a local smooth surface is obtained by fitting using the least squares method; For the fitted surface at each vertex, two principal curvatures k1 and k2 are obtained by solving the curvature of the surface normal and the tangent plane (there are existing methods for finding principal curvatures, which will not be elaborated on in detail). The average curvature k' of the fitted surface can be obtained by (k1+k2) / 2; The average curvature k' of the three vertices of the surface is denoted as the average curvature K of the surface.

[0023] In one embodiment, a three-dimensional mesh model can be projected onto a two-dimensional plane along the main normal direction of the building facade to generate an orthophoto image without perspective distortion. Each pixel in the orthophoto is a vertex in the mesh model, and edge detection is performed on the curvature K of the surface in the orthophoto to obtain the edges in the 3D mesh model; The edges of the 3D mesh model obtained by the above operations can represent the parts in the 3D mesh model with large curvature changes (curvature changes greater than a preset threshold are considered large changes, which can be an adaptive threshold or an empirically based fixed threshold), and can be used in subsequent segmentation construction.

[0024] Step S2: Determine the projection region of the grid region in the two-dimensional image and obtain the semantic probability vector of the grid region based on the projection region; determine the adjacent edges between adjacent grid regions in the three-dimensional grid model and the geometric discontinuities on both sides of the adjacent edges. Specifically, please refer to Figure 2 Step S2, determining the projection region of the grid region in the two-dimensional image and obtaining the semantic probability vector of the grid region based on the projection region, includes: Step S21: Project the vertices of the grid region onto the pixel coordinate system of the corresponding two-dimensional image to obtain the projection region in the two-dimensional image; Step S22: Determine the probability of each component semantic category corresponding to the pixel in the projection area and obtain the semantic probability vector of the pixel. Step S23: Based on the semantic probability vector of each pixel in the projection region, obtain the semantic probability vector of the grid region.

[0025] More specifically, please refer to Figure 3 Step S23 includes: Step S231: Calculate the initial semantic probability vector of the grid region by using the semantic probability vector of each pixel in the projection region. In this embodiment, the edges in the three-dimensional mesh model obtained in the above embodiment are not entirely the edges of components due to the complexity of the components on the street scene facade surface. Therefore, it is necessary to segment them by combining the semantics of the components. Acquire the collected two-dimensional images and filter out those that clearly show the facade of the target building; Each selected 2D image is input into a pre-trained deep convolutional neural network optimized for building facades. Through network training, a probability map identical to the image is output (the acquisition of the probability map is a current technology and will not be elaborated on here). In the two-dimensional image, each pixel contains a c-dimensional (semantic probability) vector, where c is the number of semantic categories (e.g., window, door, decoration); the c-dimensional vector represents the probability that the pixel belongs to each component category, and the sum of the probabilities is 1. Furthermore, Canny edge detection is used on each filtered 2D image to obtain the component image edges in the image; For each triangular facet in the above 3D mesh, the depth buffer (Z-buffer) technique or ray-casting algorithm is used to determine whether the mesh facet is occluded by a foreground object in the current camera view; semantic probability vectors are extracted and mapped only for the unoccluded visible faces, and the unoccluded faces are recorded as visible faces. For a visible face, its three vertices are projected onto the pixel coordinate system of the corresponding two-dimensional image to obtain a projected triangular region. It should be noted that the two-dimensional image corresponding to the face here refers to the two-dimensional image containing the projection of the face or the image information of the face, and not other unrelated images.

[0026] Obtain the c-dimensional vector of each pixel in the projection triangle region. Take the mean of each dimension of the vector of each pixel in the projection triangle region as a new c-dimensional vector in the projection triangle region, denoted as the initial semantic probability vector C. Each dimension in the probability vector C indicates the probability of different component categories.

[0027] Step S232: Using the shooting perspective relationship between the grid region and the corresponding two-dimensional image, determine the quality weight of the grid region in the corresponding two-dimensional image to represent perspective distortion. More specifically, step S232 includes: Determine the projected area of ​​the grid region in the corresponding two-dimensional image, and determine the vector angle between the normal vector of the grid region and the shooting direction vector of the corresponding two-dimensional image; Using the projected area and the vector angle, the quality weight of the grid region in the corresponding two-dimensional image representing perspective distortion is calculated.

[0028] In this embodiment, based on the above embodiment, the initial semantic probability vector of the corresponding patch in the two-dimensional image can be obtained. Since the patch will exist in different two-dimensional images during the acquisition process, it is also necessary to analyze the semantic probability vector of the patch in different two-dimensional images. During the acquisition process, two-dimensional images are captured from different perspectives of the street scene facade. Since the angle between the facade and the camera changes from different perspectives, perspective distortion occurs, which leads to distortion of the texture geometry information and may result in inaccurate recognition. The larger the projected area of ​​a patch in different images, the more complete the texture information of the patch in the image will be; Obtain the projected triangular region of the current patch in different images, and obtain the number of pixels n in it to reflect the projected area; Obtain the normal vector of the facet; and obtain the camera intrinsic parameters of the current image, and obtain the shooting direction vector of the current image based on the camera intrinsic parameters; Obtain the angle between the normal vector of the facet and the shooting direction vector. (The angle value is linearly normalized according to 0°-90°, and the value is [0,1]). When the camera is facing the front of the facade during image acquisition, perspective distortion is reduced, resulting in higher quality texture geometry information of materials in the image; Therefore, the quality weight ZQ of the patch in different images can be obtained, specifically through... The results are then normalized to obtain ZQ (unless otherwise specified in this invention, all normalizations are performed to [0,1] using Max-Min), where... It is a very small positive number, such as 0.01, to prevent the denominator from being 0; and for The larger the value, the more pixels the patch has in the projected triangular region of the current image and the smaller the angle between the patch normal vector and the shooting direction. Therefore, it indicates that the current shooting direction is more aligned with the direction of the patch, so the image acquired at this time can better reflect the texture geometry of the patch, and thus the patch has a greater quality weight in the image.

[0029] It should be noted that since all images are at the same scale, the more pixels there are, the larger their proportion and the greater their quality weight.

[0030] Step S233: Using the quality weights of the grid regions in the corresponding two-dimensional images and the initial semantic probability vectors, the target semantic probability vector of the grid regions is obtained.

[0031] Since each patch possesses an (initial) semantic probability vector C in multiple images v And the quality weights ZQ in different images v Where v represents the v-th viewpoint image of the patch; Therefore, we can obtain the three-dimensional probability vector C' of the patch through weighting, specifically through C'= We obtain V, which is the number of viewpoint images of the patch. This allows us to obtain the three-dimensional probability vector C' of each facet in the mesh model, which is also the target semantic probability vector. If a grid region is occluded in all corresponding 2D images, resulting in a viewpoint image count of 0, then extract the target semantic probability vector of its adjacent ring-shaped neighborhood grid region in 3D space, and calculate the mean value as the target semantic probability vector of the occluded grid region.

[0032] Each facet in the 3D mesh model is clustered according to its 3D probability vector (e.g., DBSCAN), resulting in several clusters, where the 3D probability vectors of the faces in each cluster are similar. Specifically, superpixel segmentation can be used to segment the three-dimensional probability vector C' of each patch, resulting in several clusters, where the three-dimensional probability vectors of the patches in each cluster are similar.

[0033] In one embodiment, step S2, determining the geometric discontinuity on both sides of an adjacent edge, includes: Determine the unit normal difference and the average curvature difference of the grid regions on both sides of an adjacent edge; The geometric discontinuity on both sides of the adjacent edge is obtained by combining the unit normal vector difference and the average curvature difference.

[0034] The three-dimensional probability vector C' of each facet in the mesh model obtained in the above embodiments can be used to directly segment the components in the three-dimensional mesh model through the clustering results; During the segmentation and extraction process, the edge region of the component may not have an accurate 3D probability vector due to its connection with two types of components, and there may be cases of component adhesion. Therefore, this embodiment requires further analysis of the component edges in the three-dimensional mesh model; For any connected triangular facet in a 3D mesh model, the connected edges are denoted as adjacent edges; Adjacent edges may be the boundaries of components in a streetscape facade. There may be discontinuities between the edges of different components, which will cause changes in the curvature and angle of the boundary. Obtain the unit normal vectors f1 and f2 of the two side faces of the adjacent edge, and normalize the difference between the obtained unit normal vectors to obtain f'; Furthermore, the average curvatures K1 and K2 of the two side panels of the adjacent edge are obtained, and the difference between the obtained average curvatures is normalized to obtain K'; Therefore, the geometric discontinuity JX on both sides of the adjacent edge can be obtained, specifically after normalization by K'×f'. The larger the value of K'×f', the larger the difference in the normal vectors of the two side faces of the adjacent edge and the larger the difference in the average curvature of the two side faces. Therefore, it indicates that the difference in the orientation and curvature of the two side faces of the adjacent edge is greater, and thus the geometric discontinuity on both sides of the adjacent edge is greater.

[0035] Step S3: Determine the component image edge in the two-dimensional image, and determine the overlapping pixel information of the adjacent edge projection with the component image edge pixel in the corresponding two-dimensional image; Specifically, step S3, determining the overlapping pixel information between the adjacent edge projection and the edge pixels of the component image in the corresponding two-dimensional image, includes: Determine the projected edges of adjacent edges in the corresponding 2D image, and determine the number of overlapping pixels between the projected edges and the edges of the component image; The edge pixel density of adjacent edges on the corresponding two-dimensional image is obtained by using the number of overlapping pixels and the length of the projected edge.

[0036] In this embodiment, the geometric discontinuity of the two side panels of the adjacent edge in the above embodiment represents the geometric shape change on both sides of the component edge. However, in the actual street scene facade renovation, there may be no obvious geometric change between different components (such as glass and walls), which will result in no obvious geometric discontinuity at the edge of the component. For each image containing the current adjacent edges, obtain the component image edges in that image; Get the projected edge of the current adjacent edge in each image, and get the number of edge pixels n' on the projected edge, that is, the number of overlapping pixels between the projected edge and the edge of the component image. Then, the edge pixel density N of adjacent edges per unit projection length on each image is obtained by combining the projection edge length.

[0037] Step S4: Combine the semantic probability vectors on both sides of the adjacent edge, the geometric discontinuity, and the overlapping pixel information to obtain the boundary coefficient of the adjacent edge. Based on the boundary coefficient of each adjacent edge, obtain the segmentation boundary and component segmentation result of the three-dimensional mesh model.

[0038] Specifically, please refer to Figure 4 In one embodiment, step S4, which combines the semantic probability vectors on both sides of the adjacent edge, the geometric discontinuity, and the overlapping pixel information to obtain the boundary coefficient of the adjacent edge, includes: Step S41: Using the geometric discontinuity and the overlapping pixel information, obtain the degree of semantic difference between the components on both sides of the adjacent edge; Step S42: Combine the difference in semantic probability vectors on both sides of the adjacent edge with the degree of semantic difference of the components to obtain the boundary coefficient of the adjacent edge.

[0039] Based on the above embodiments, in this embodiment, components with the same texture structure will have obvious differences in the two-dimensional image due to the different component types, resulting in edges between the components; Therefore, the semantic difference (degree of semantic difference of components) HB on both sides of each adjacent edge can be obtained, specifically by normalizing the result after applying 0.5×JX+0.5×N. For the edge pixel density N, the larger the value, the more edge pixels the adjacent edge has on the image projection. Therefore, the more the surface adjacent edge belongs to the edge in the image, the greater its semantic difference. The two terms 0.5 represent the weights assigned to the geometric discontinuity JX and the edge pixel density N, respectively. The weights of the two terms can be equal or different. This is just an example.

[0040] Based on the semantic difference HB, adjacent edges with a semantic difference HB greater than 0.7 (a preset difference threshold, which is an empirical reference value obtained by statistical analysis of the real labeled boundaries of the historical street scene facade sample set; or a dynamic threshold obtained by adaptive calculation of the semantic difference values ​​of all adjacent edges in the 3D mesh model using the Otsu method) can be selected as candidate boundaries for components. When extracting components, it is necessary to segment them according to the adjacent edges with higher candidate boundary coefficients as much as possible.

[0041] Therefore, the candidate boundary coefficient E of the adjacent edge can be obtained, specifically by normalizing the result of 0.5×HB+0.5×C'', where C'' is the difference in the three-dimensional probability vectors of the two side faces of the adjacent edge, and the two terms 0.5 are the corresponding calculation weights. The larger the value of 0.5×HB+0.5×C'', the larger the candidate boundary coefficient of the current adjacent edge and the larger the difference in the three-dimensional probability vectors of the two sides of the adjacent edge. Therefore, it indicates that the material difference between the two side faces of the adjacent edge is more obvious. Thus, the larger the candidate boundary coefficient, the more likely the corresponding adjacent edge is to be the real component boundary.

[0042] Specifically, please refer to Figure 5 In another embodiment, step S4, obtaining the segmentation boundary and component segmentation results of the 3D mesh model based on the boundary coefficients of each adjacent edge, includes: Step S401: Cluster each grid region according to the semantic probability vector to obtain multiple clusters, and use adjacent clusters as source points and sink points respectively. Step S402: Use the boundary coefficient of adjacent edges to obtain the capacity of adjacent edges, and use the obtained source point, sink point and capacity to obtain the segmentation boundary and component segmentation results of the three-dimensional mesh model using the minimum cut algorithm.

[0043] In this embodiment, based on the clustering and boundary coefficients obtained from the above embodiments, the minimum cut algorithm (Min-Cut) is used to select adjacent edges; The algorithm divides the image into two parts by finding a cut: source point (a specific construction label, such as door) and sink point (other category labels that do not contain this label, such as window, billboard, etc.). Here, the adjacent clusters mentioned above are taken as the source point and sink point, respectively. The algorithm calculates the capacity of each edge between the source and sink, and then uses the edge with the minimum total capacity to separate the source and sink. Typically, the algorithm uses the difference between the two sides of each edge as the capacity of that edge. In this embodiment, the candidate boundary coefficient E of the adjacent edges of adjacent patches is... As the capacity of each adjacent edge in the algorithm; It is a very small positive number, such as 0.01.

[0044] Since the boundary coefficient is largest on the boundary line between different components, therefore... As the capacity in the algorithm, a boundary line with the maximum boundary coefficient and the minimum total capacity can be obtained; If there are several paths consisting of patches between the source and sink, the algorithm will select an adjacent edge flow between these paths. By using the minimum cut algorithm, the cutting surface with the smallest total capacity is found, which determines the segmentation boundary of the 3D mesh model. Finally, the globally optimal label assignment scheme is obtained, and then the segmentation is performed to obtain the component segmentation result.

[0045] It should be noted that, in other embodiments of the present invention, a local network graph can also be constructed for each pair of adjacent clusters in the three-dimensional mesh model, and each pair of adjacent clusters can be used as the source and sink respectively. The local minimum cut algorithm is then performed in combination with the capacity of adjacent edges, and the segmentation boundary and component segmentation results of the three-dimensional mesh model are obtained by integrating all local cutting surfaces.

[0046] Furthermore, based on the refined semantic segmentation mesh obtained above, the system will automatically execute the following application process: First, cluster the set of connected facets with the same semantic label, and generate key geometric parameters such as the minimum bounding cube, center point, and normal vector for each component (such as window and balcony), and extract the spatial coordinate sequence of its main vertices to define the precise boundary contour, forming a structured component instance database. Subsequently, based on the component category and parameters, the system calls the corresponding parametric BIM family model (such as window family and door family that meet the dimensions) from the standard building component library, and drives it to automatically match and adapt, generating a preliminary facade BIM (Building Information Modeling) model.

[0047] Based on this model, designers can directly select replacement materials, adjust component styles, or implant energy-saving modules (such as replacing with energy-saving window types or adding sunshade louvers) on the integrated platform. The system will calculate and output the corresponding bill of quantities in real time. Ultimately, all component information, renovation plans, and related data are packaged together and can be directly imported into BIM design software or urban information modeling platforms for visual review of plans, refinement of construction drawings, and digital construction management.

[0048] To automatically extract components from a grid model for streetscape facade renovation, this invention uses 3D point cloud technology to collect 3D point cloud data and 2D images of building facades. A 3D mesh model is constructed using the collected 3D point cloud data. The faces (mesh regions) of the 3D mesh model are projected onto the corresponding 2D images, and semantic probability analysis of the faces is performed by analyzing the components in the 2D images, thus obtaining the 3D semantic probability vector of the faces. Furthermore, the boundary coefficients of adjacent edges are obtained by combining the geometric changes between faces with the overlap of adjacent edges with edge pixels in the 2D images and the 3D probability vector of the faces, thereby obtaining the segmentation boundary and completing the semantic segmentation result of the 3D mesh model. Compared to traditional algorithms that only automatically identify components based on the 3D mesh model, this invention obtains the 3D semantic probability vector of the 3D mesh model by projecting the 3D mesh model onto the 2D image, and reflects the semantic features of components in the 2D image into the 3D mesh model for correlation analysis. This makes the segmentation result more semantically meaningful, more accurately locating the boundaries of each component in the 3D mesh model from at least two dimensions: semantic features and morphological change features, thus achieving a more ideal segmentation effect.

[0049] Example 2: This invention also proposes an automatic extraction device for gridded model components used in streetscape facade renovation. The device can be a computer, server, or other data analysis and computing equipment, or a combination of multiple devices.

[0050] like Figure 6 As shown, Figure 6 This is a schematic diagram of the hardware operating environment of the automatic extraction device for gridded model components used in streetscape facade renovation, which is involved in the embodiments of the present invention.

[0051] like Figure 6 As shown, the automatic extraction device for gridded model components used in streetscape facade renovation may include: a processor 1001, such as a CPU; a network interface 1004; a user interface 1003; a memory 1005; and a communication bus 1002. The communication bus 1002 is used to establish communication between these components. The user interface 1003 may include a display or an input unit such as a control panel; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed RAM or a stable, non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage system independent of the aforementioned processor 1001. The memory 1005, as a computer storage medium, may include an automatic extraction program for gridded model components used in streetscape facade renovation (hereinafter referred to as the "automatic extraction program for gridded model components").

[0052] Those skilled in the art will understand that Figure 6 The hardware structure shown does not constitute a limitation on the device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0053] Continue to refer to Figure 6 , Figure 6 The memory 1005, which is a computer-readable storage medium, may include an operating system, a user interface module, a network communication module, and an automatic extraction program for gridded model components for streetscape facade renovation.

[0054] exist Figure 6 In this embodiment, the network communication module is mainly used to connect to the server and can communicate with the server for data; while the processor 1001 can call the automatic extraction program for the gridded model components for street scene facade renovation stored in the memory 1005 and execute the steps in the above embodiments.

[0055] Based on the hardware structure of the above-mentioned automatic extraction device for gridded model components for streetscape facade renovation, various embodiments of the automatic extraction method for gridded model components for streetscape facade renovation of the present invention are implemented.

[0056] In addition, this invention also provides an automatic extraction system for gridded model components for streetscape facade renovation (hereinafter referred to as the "automatic extraction system for gridded model components"). Please refer to [link / reference]. Figure 7 The automatic extraction system for gridded model components used in streetscape facade renovation includes: The mesh construction module A10 is used to acquire the 3D point cloud and 2D image of the street scene facade, and to construct a 3D mesh model composed of multiple mesh regions by meshing the 3D point cloud. The correlation analysis module A20 is used to determine the projection region of the grid region in the two-dimensional image and obtain the semantic probability vector of the grid region based on the projection region; determine the adjacent edges between adjacent grid regions in the three-dimensional grid model and the geometric discontinuities on both sides of the adjacent edges; determine the component image edges in the two-dimensional image; and determine the overlapping pixel information of the adjacent edge projection and the component image edge pixels in the corresponding two-dimensional image. The boundary localization module A30 is used to combine the semantic probability vectors on both sides of the adjacent edge, the geometric discontinuity and the overlapping pixel information to obtain the boundary coefficient of the adjacent edge, and to obtain the segmentation boundary and component segmentation result of the three-dimensional mesh model based on the boundary coefficient of each adjacent edge.

[0057] Furthermore, the mesh building module A10 is also used for: The triangulation algorithm is used to establish connections between adjacent 3D point clouds to obtain multiple grid regions; The average curvature of the grid region in the 3D mesh model is determined by using the neighborhood curvature parameters of each vertex of the grid region.

[0058] Furthermore, the correlation analysis module A20 is also used for: Projecting the vertices of the grid region onto the pixel coordinate system of the corresponding two-dimensional image yields the projected region in the two-dimensional image. Determine the probability of each pixel in the projection area corresponding to the semantic category of each component and obtain the semantic probability vector of the pixel; The semantic probability vector of the grid region is obtained based on the semantic probability vector of each pixel in the projection region.

[0059] Furthermore, the correlation analysis module A20 is also used for: The initial semantic probability vector of the grid region is obtained by averaging the semantic probability vectors of each pixel in the projection region. By utilizing the shooting perspective relationship between the grid region and the corresponding two-dimensional image, the quality weight of the grid region in the corresponding two-dimensional image representing perspective distortion is determined; By utilizing the quality weights of the grid regions in different corresponding two-dimensional images and the initial semantic probability vectors, the target semantic probability vector of the grid regions is obtained.

[0060] Furthermore, the correlation analysis module A20 is also used for: Determine the projected area of ​​the grid region in the corresponding two-dimensional image, and determine the vector angle between the normal vector of the grid region and the shooting direction vector of the corresponding two-dimensional image; Using the projected area and the vector angle, the quality weight of the grid region in the corresponding two-dimensional image representing perspective distortion is calculated.

[0061] Furthermore, the correlation analysis module A20 is also used for: Determine the unit normal difference and the average curvature difference of the grid regions on both sides of an adjacent edge; The geometric discontinuity on both sides of the adjacent edge is obtained by combining the unit normal vector difference and the average curvature difference.

[0062] Furthermore, the correlation analysis module A20 is also used for: Determine the projected edges of adjacent edges in the corresponding 2D image, and determine the number of overlapping pixels between the projected edges and the edges of the component image; The edge pixel density of adjacent edges on the corresponding two-dimensional image is obtained by using the number of overlapping pixels and the length of the projected edge.

[0063] Furthermore, the boundary positioning module A30 is also used for: Using the geometric discontinuities and the overlapping pixel information, the degree of semantic difference between the components on both sides of the adjacent edge is obtained; The boundary coefficient of the adjacent edge is obtained by combining the difference in semantic probability vectors on both sides of the adjacent edge and the degree of semantic difference of the components.

[0064] Furthermore, the boundary positioning module A30 is also used for: Each grid region is clustered according to its semantic probability vector to obtain multiple clusters, and adjacent clusters are used as source points and sink points respectively. The capacity of adjacent edges is obtained by using the boundary coefficient of adjacent edges. The obtained source points, sink points and capacity are then used to obtain the segmentation boundary and component segmentation results of the 3D mesh model using the minimum cut algorithm.

[0065] Furthermore, the present invention also provides a computer-readable storage medium. The computer-readable storage medium stores an automatic extraction program for gridded model components used in streetscape facade renovation, wherein, when executed by a processor, the automatic extraction program for gridded model components used in streetscape facade renovation implements the steps of the above-described automatic extraction method for gridded model components used in streetscape facade renovation.

[0066] The method implemented when the automatic extraction program for gridded model components for streetscape facade renovation is executed can be referred to in various embodiments of the automatic extraction method for gridded model components for streetscape facade renovation of the present invention, and will not be repeated here.

[0067] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0068] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0069] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0070] The above description is only a preferred embodiment of the present invention and does not limit the scope of protection of the present invention. All equivalent structural / method transformations made under the inventive concept of the present invention using the contents of the present invention specification and drawings, or direct / indirect applications in other related technical fields, are included within the scope of protection of the present invention.

Claims

1. A method for automatically extracting gridded model components for streetscape facade renovation, characterized in that, The method includes: Obtain 3D point cloud and 2D images of street scene facades, and construct a 3D mesh model composed of multiple mesh regions by meshing the 3D point cloud; Determine the projection region of the grid region in the 2D image and obtain the semantic probability vector of the grid region based on the projection region. Determine the adjacent edges between adjacent grid regions in the 3D grid model and the geometric discontinuities on both sides of the adjacent edges. Determine the edge of the component image in the two-dimensional image, and determine the overlapping pixel information of the adjacent edge projection with the edge pixel of the component image in the corresponding two-dimensional image; The boundary coefficients of adjacent edges are obtained by combining the semantic probability vectors on both sides of the adjacent edges, the geometric discontinuities, and the overlapping pixel information. Based on the boundary coefficients of each adjacent edge, the segmentation boundary and component segmentation results of the three-dimensional mesh model are obtained.

2. The method for automatic extraction of gridded model components for streetscape facade renovation according to claim 1, characterized in that, The process of constructing a 3D mesh model composed of multiple mesh regions by meshing a 3D point cloud includes: The triangulation algorithm is used to establish connections between adjacent 3D point clouds to obtain multiple grid regions; The average curvature of the grid region in the 3D mesh model is determined by using the neighborhood curvature parameters of each vertex of the grid region.

3. The method for automatically extracting gridded model components for streetscape facade renovation according to claim 1, characterized in that, The process of determining the projection region of the grid region in the two-dimensional image and obtaining the semantic probability vector of the grid region based on the projection region includes: Projecting the vertices of the grid region onto the pixel coordinate system of the corresponding two-dimensional image yields the projected region in the two-dimensional image. Determine the probability of each pixel in the projection area corresponding to the semantic category of each component and obtain the semantic probability vector of the pixel; The semantic probability vector of the grid region is obtained based on the semantic probability vector of each pixel in the projection region.

4. The method for automatic extraction of gridded model components for streetscape facade renovation according to claim 3, characterized in that, The process of obtaining the semantic probability vector of the grid region based on the semantic probability vector of each pixel in the projection region includes: The initial semantic probability vector of the grid region is obtained by averaging the semantic probability vectors of each pixel in the projection region. By utilizing the shooting perspective relationship between the grid region and the corresponding two-dimensional image, the quality weight of the grid region in the corresponding two-dimensional image representing perspective distortion is determined; By utilizing the quality weights of the grid regions in different corresponding two-dimensional images and the initial semantic probability vectors, the target semantic probability vector of the grid regions is obtained.

5. The method for automatically extracting gridded model components for streetscape facade renovation according to claim 4, characterized in that, The method of determining the quality weight of the grid region in the corresponding two-dimensional image, based on the shooting perspective relationship between the grid region and the corresponding two-dimensional image, includes: Determine the projected area of ​​the grid region in the corresponding two-dimensional image, and determine the vector angle between the normal vector of the grid region and the shooting direction vector of the corresponding two-dimensional image; Using the projected area and the vector angle, the quality weight of the grid region in the corresponding two-dimensional image representing perspective distortion is calculated.

6. The method for automatically extracting gridded model components for streetscape facade renovation according to claim 1, characterized in that, Determining the geometric discontinuities on both sides of an adjacent edge includes: Determine the unit normal difference and the average curvature difference of the grid regions on both sides of an adjacent edge; The geometric discontinuity on both sides of the adjacent edge is obtained by combining the unit normal vector difference and the average curvature difference.

7. The method for automatically extracting gridded model components for streetscape facade renovation according to claim 1, characterized in that, The determination of the overlapping pixel information between the adjacent edge projection and the edge pixels of the component image in the corresponding two-dimensional image includes: Determine the projected edges of adjacent edges in the corresponding 2D image, and determine the number of overlapping pixels between the projected edges and the edges of the component image; The edge pixel density of adjacent edges on the corresponding two-dimensional image is obtained by using the number of overlapping pixels and the length of the projected edge.

8. The method for automatically extracting gridded model components for streetscape facade renovation according to claim 1, characterized in that, The step of combining the semantic probability vectors on both sides of the adjacent edge, the geometric discontinuity, and the overlapping pixel information to obtain the boundary coefficient of the adjacent edge includes: Using the geometric discontinuities and the overlapping pixel information, the degree of semantic difference between the components on both sides of the adjacent edge is obtained; The boundary coefficient of the adjacent edge is obtained by combining the difference in semantic probability vectors on both sides of the adjacent edge and the degree of semantic difference of the components.

9. The method for automatically extracting gridded model components for streetscape facade renovation according to claim 1, characterized in that, The process of obtaining the segmentation boundary and component segmentation results of the 3D mesh model based on the boundary coefficients of each adjacent edge includes: Each grid region is clustered according to its semantic probability vector to obtain multiple clusters, and adjacent clusters are used as source points and sink points respectively. The capacity of adjacent edges is obtained by using the boundary coefficient of adjacent edges. The obtained source points, sink points and capacity are then used to obtain the segmentation boundary and component segmentation results of the 3D mesh model using the minimum cut algorithm.

10. An automatic extraction system for gridded model components used in streetscape facade renovation, characterized in that, The system is used to implement the automatic extraction method for gridded model components for streetscape facade renovation as described in any one of claims 1 to 9; the system includes: The mesh construction module is used to acquire the 3D point cloud and 2D image of the street scene facade, and to construct a 3D mesh model composed of multiple mesh regions by meshing the 3D point cloud. The correlation analysis module is used to determine the projection region of the grid region in the 2D image and obtain the semantic probability vector of the grid region based on the projection region; determine the adjacent edges between adjacent grid regions in the 3D grid model and the geometric discontinuities on both sides of the adjacent edges; determine the component image edges in the 2D image; and determine the overlapping pixel information of the adjacent edge projection and the component image edge pixels in the corresponding 2D image. The boundary localization module is used to combine the semantic probability vectors on both sides of the adjacent edge, the geometric discontinuity, and the overlapping pixel information to obtain the boundary coefficient of the adjacent edge, and to obtain the segmentation boundary and component segmentation result of the three-dimensional mesh model based on the boundary coefficient of each adjacent edge.

Images