Chessboard folding convolution-based point cloud to mesh real estate building cluster 3D reconstruction method

Abstract

The present application belongs to the technical field of 3D modeling reconstruction, and provides a real estate building cluster 3D reconstruction method for converting point cloud to MESH based on chessboard folding convolution, which comprises: data collection for reconstruction to be performed and modeling reconstruction; and a training process of the model, which comprises: data collection, model construction, folding convolution, feature fusion, weight calculation, feature representation calculation and spatial mapping; through multi-layer chessboard folding convolution, the present application can effectively map two-dimensional image information to three-dimensional space, realize adaptive extraction of multi-scale features, and simultaneously capture details and large scene structures; through feature fusion of the multi-layer convolution features, the point cloud and the Mesh structure can promote each other, and gradually improve the reconstruction accuracy and the detail expression capability; through hierarchical perception weight, the present application can automatically adapt to the complexity and diversity of the input image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of 3D modeling reconstruction, in particular to a real estate building cluster 3D reconstruction method based on chessboard folding convolution for point cloud to MESH. BACKGROUND

[0002] The current technology is mainly based on point cloud modeling, but the calculation amount of point cloud is quite large, and insufficient sampling of point cloud will lead to model collapse. Influenced by the complexity of community building cluster model, these methods may lead to difficult rendering in community building cluster, and depend on hardware foundation. SUMMARY

[0003] In order to overcome the shortcomings of the prior art, the purpose of the present application is to provide a real estate building cluster 3D reconstruction method based on chessboard folding convolution for point cloud to MESH, which solves the problems of excessive calculation amount, insufficient sampling and high complexity existing in the prior art 3D reconstruction method.

[0004] To achieve the above purpose, the present application provides the following scheme:

[0005] A real estate building cluster 3D reconstruction method based on chessboard folding convolution for point cloud to MESH, comprising:

[0006] Collecting image data of the target building to obtain reconstruction data;

[0007] Inputting the reconstruction data into a pre-trained 3D reconstruction model for processing to obtain a three-dimensional structure diagram; the training process of the 3D reconstruction model comprises:

[0008] Collecting a plurality of groups of reconstruction data;

[0009] Constructing an original network model;

[0010] Using the FoldConv module of the original network model to perform multi-layer chessboard folding convolution on the reconstruction data to obtain multi-layer convolution features;

[0011] Using the FeatureFusion module of the original network model to perform feature fusion on the multi-layer convolution features to obtain a three-dimensional feature representation;

[0012] Using the multi-layer perceptron of the original network model to process the three-dimensional feature representation to obtain a hierarchical perception weight;

[0013] Multiplying and accumulating the hierarchical perception weight and the three-dimensional feature representation to obtain a final feature representation;

[0014] Using the decoding function of the original network model to map the final feature representation to a three-dimensional space to obtain an iterative structure diagram;

[0015] calculating the iterative structure graph by using the loss function to obtain a total loss value, and performing parameter optimization and iteration on the original network model by using the total loss value to obtain the 3D reconstruction model trained.

[0016] Preferably, the total loss value is obtained by calculating the iterative structure graph by using the loss function, and the 3D reconstruction model trained is obtained by performing parameter optimization and iteration on the original network model by using the total loss value, comprising:

[0017] defining a loss function; the expression of the loss function is: E total =E+λ1E c +λ2E d ; wherein, E total is the total loss value; E, E c , E d are generation graph loss, feature extraction loss, and dynamic hierarchical perception loss, respectively; λ1 and λ2 are first weight hyperparameters and second weight hyperparameters, respectively; Z is an iterative structure graph; Z * is an actual structure graph corresponding to the iterative structure graph;

[0018] The loss value in the training process of the original network model is calculated by using the loss function to obtain the total loss value.

[0019] According to the total loss value, the model parameters of the original network model are optimized by using an Adam optimizer, and the 3D reconstruction model is obtained after iteration.

[0020] Preferably, the data to be reconstructed is determined, comprising:

[0021] collecting a two-dimensional image of a target building;

[0022] determining a three-dimensional tensor of a chessboard layout of the model;

[0023] splicing the two-dimensional image and the three-dimensional tensor to obtain the data to be reconstructed.

[0024] Preferably, before the multi-layer convolution features are obtained by using the FoldConv module of the original network model to perform multi-layer chessboard folding convolution on the data to be reconstructed, comprising:

[0025] setting a normalization function;

[0026] normalizing the two-dimensional coordinates of the two-dimensional image by using the normalization function; the normalization function comprises: wherein, x and y are the horizontal coordinate and the vertical coordinate of the two-dimensional coordinates, respectively; W and H are the width and the height of the image, respectively; xnorm , y norm are normalized horizontal and vertical coordinates, respectively.

[0027] Preferably, the original network model is used to perform multi-layer fold convolution on the to-be-reconstructed data by using a FoldConv module, to obtain multi-layer convolution features, including:

[0028] determining the fold convolution kernel and the fold parameter of the first layer;

[0029] performing multi-layer fold convolution on the to-be-reconstructed data by using the FoldConv module according to the fold convolution kernel and the fold parameter of the first layer, to obtain initialization features; the expression of the initialization features is: Z (0) = FoldConv (I, W f(1) , a1); wherein Z (0) is the initialization features; I is the to-be-reconstructed data; W f(1) , a1 are the fold convolution kernel and the fold parameter of the first layer, respectively; FoldConv (·) represents multi-layer fold convolution.

[0030] Preferably, the original network model is used to perform feature fusion on the multi-layer convolution features by using a FeatureFusion module, to obtain three-dimensional feature representation, including:

[0031] determining the fold convolution kernel and the fold parameter of the lth layer;

[0032] obtaining the three-dimensional feature representation of the (l-1)th layer;

[0033] performing feature fusion on the three-dimensional feature representation of the (l-1)th layer by using the FeatureFusion module according to the fold convolution kernel and the fold parameter of the lth layer, to obtain the three-dimensional feature representation of the lth layer; the expression of the three-dimensional feature representation of the lth layer is:

[0034] Z (l) = FeatureFusion (Z (l-1) , W f(l) , a l );

[0035] wherein Z (l) is the three-dimensional feature representation of the lth layer; FeatureFusion (·) represents a feature fusion operation.

[0036] Preferably, the original network model is used to process the three-dimensional feature representation by using a multi-layer perceptron, to obtain hierarchical perception weights, including:

[0037] The three-dimensional feature representation of the lth layer is processed by using the multi-layer perceptron to obtain a perception feature of the lth layer. An expression of the perception feature of the lth layer is g (l) =MLP(Z (l) ); wherein g (l) is the perception feature of the lth layer, and MLP(·) represents a multi-layer perception processing.

[0038] The perception feature of the lth layer is calculated by using an activation function to obtain the hierarchical perception weight of the lth layer. An expression of the hierarchical perception weight of the lth layer is ω (l) =σ(g (l) ); wherein ω (l) is the hierarchical perception weight of the lth layer, and σ(·) is the activation function.

[0039] Preferably, a real estate building cluster 3D reconstruction system based on a point cloud to MESH conversion and a chessboard folding convolution comprises:

[0040] A model generation module is configured to construct and train a 3D reconstruction model.

[0041] A data acquisition module is configured to determine to-be-reconstructed data.

[0042] A multi-layer convolution module is configured to perform multi-layer chessboard folding convolution on the to-be-reconstructed data by using a FoldConv module of the original network model to obtain multi-layer convolution features.

[0043] A feature fusion module is configured to perform feature fusion on the multi-layer convolution features by using a FeatureFusion module of the original network model to obtain a three-dimensional feature representation.

[0044] A perception weight extraction module is configured to process the three-dimensional feature representation by using a multi-layer perceptron of the original network model to obtain a hierarchical perception weight.

[0045] A weighted fusion module is configured to multiply and accumulate the hierarchical perception weight and the three-dimensional feature representation to obtain a final feature representation.

[0046] A space mapping module is configured to map the final feature representation to a three-dimensional space by using a decoding function of the original network model to obtain a three-dimensional structure diagram.

[0047] Preferably, an electronic device comprises at least one processor and a memory connected in communication with the processor. The memory stores instructions executable by the processor. The instructions are executed by the processor to enable the processor to perform the foregoing method for reconstructing a 3D real estate building cluster based on a point cloud to MESH conversion and a chessboard folding convolution.

[0048] Preferably, a non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the aforementioned real estate building cluster 3D reconstruction method based on a point cloud to MESH of a chessboard folding convolution.

[0049] The following technical effects are disclosed in the present application:

[0050] The present application provides a real estate building cluster 3D reconstruction method based on a point cloud to MESH of a chessboard folding convolution, which solves the dimensional limitations of traditional convolution in 3D space by performing multi-layer chessboard folding convolution on the data to be reconstructed, realizes adaptive extraction of multi-scale features, solves the defects of poor correlation between point cloud and Mesh structure in the prior art by performing feature fusion on the multi-layer convolution features, and realizes the improvement of reconstruction accuracy and detail expression ability, and solves the limitations of fixed architecture on specific scenes through hierarchical perception weight, and realizes the automatic adaptation of input image complexity and diversity. BRIEF DESCRIPTION OF DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0052] Figure 1 The present application provides a real estate building cluster 3D reconstruction method based on a point cloud to MESH of a chessboard folding convolution, which solves the dimensional limitations of traditional convolution in 3D space by performing multi-layer chessboard folding convolution on the data to be reconstructed, realizes adaptive extraction of multi-scale features, solves the defects of poor correlation between point cloud and Mesh structure in the prior art by performing feature fusion on the multi-layer convolution features, and realizes the improvement of reconstruction accuracy and detail expression ability, and solves the limitations of fixed architecture on specific scenes through hierarchical perception weight, and realizes the automatic adaptation of input image complexity and diversity.

[0053] Figure 2 The present application provides a real estate building cluster 3D reconstruction method based on a point cloud to MESH of a chessboard folding convolution, which solves the dimensional limitations of traditional convolution in 3D space by performing multi-layer chessboard folding convolution on the data to be reconstructed, realizes adaptive extraction of multi-scale features, solves the defects of poor correlation between point cloud and Mesh structure in the prior art by performing feature fusion on the multi-layer convolution features, and realizes the improvement of reconstruction accuracy and detail expression ability, and solves the limitations of fixed architecture on specific scenes through hierarchical perception weight, and realizes the automatic adaptation of input image complexity and diversity.

[0054] Figure 3 The present application provides a real estate building cluster 3D reconstruction method based on a point cloud to MESH of a chessboard folding convolution, which solves the dimensional limitations of traditional convolution in 3D space by performing multi-layer chessboard folding convolution on the data to be reconstructed, realizes adaptive extraction of multi-scale features, solves the defects of poor correlation between point cloud and Mesh structure in the prior art by performing feature fusion on the multi-layer convolution features, and realizes the improvement of reconstruction accuracy and detail expression ability, and solves the limitations of fixed architecture on specific scenes through hierarchical perception weight, and realizes the automatic adaptation of input image complexity and diversity. DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the present application will be described in detail below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0056] The present application provides a real estate building cluster 3D reconstruction method based on a point cloud to MESH of a chessboard folding convolution, which solves the dimensional limitations of traditional convolution in 3D space by performing multi-layer chessboard folding convolution on the data to be reconstructed, realizes adaptive extraction of multi-scale features, solves the defects of poor correlation between point cloud and Mesh structure in the prior art by performing feature fusion on the multi-layer convolution features, and realizes the improvement of reconstruction accuracy and detail expression ability, and solves the limitations of fixed architecture on specific scenes through hierarchical perception weight, and realizes the automatic adaptation of input image complexity and diversity.

[0057] In order to make the above objectives, characteristics and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0058] Figure 1 A point cloud to MESH conversion real estate building cluster 3D reconstruction process schematic diagram based on chessboard folding convolution is provided for the embodiments of the present application, Figure 2 A technical framework schematic diagram is provided for the embodiments of the present application, as shown in Figure 1 and Figure 2 The present application provides a point cloud to MESH conversion real estate building cluster 3D reconstruction method based on chessboard folding convolution, which comprises:

[0059] Step 100: collecting image data of a target building to obtain to-be-reconstructed data;

[0060] Step 200: inputting the to-be-reconstructed data into a pre-trained 3D reconstruction model for processing to obtain a three-dimensional structure diagram; the training process of the 3D reconstruction model comprises:

[0061] Step 201: collecting a plurality of groups of to-be-reconstructed data;

[0062] Step 202: constructing an original network model;

[0063] Step 203: using the FoldConv module of the original network model to perform multi-layer chessboard folding convolution on the to-be-reconstructed data to obtain multi-layer convolution features;

[0064] Step 204: using the FeatureFusion module of the original network model to perform feature fusion on the multi-layer convolution features to obtain a three-dimensional feature representation;

[0065] Step 205: using the multi-layer perceptron of the original network model to process the three-dimensional feature representation to obtain hierarchical perception weights;

[0066] Step 206: multiplying and accumulating the hierarchical perception weights and the three-dimensional feature representation to obtain a final feature representation;

[0067] Step 207: using the decoding function of the original network model to map the final feature representation to a three-dimensional space to obtain an iterative structure diagram;

[0068] Step 208: using a loss function to calculate the iterative structure diagram to obtain a total loss value, and using the total loss value to perform parameter optimization and iteration on the original network model to obtain the trained 3D reconstruction model.

[0069] Further, the loss function is used to calculate the iterative structure diagram to obtain a total loss value, and the total loss value is used to optimize and iterate the original network model to obtain the trained 3D reconstruction model, comprising:

[0070] define a loss function; the expression of the loss function is: E total =E+λ1E c +λ2E d ; wherein, E total is the total loss value; E, E c , E d are generation graph loss, feature extraction loss, and dynamic hierarchical perception loss, respectively; λ1, λ2 are first weight hyperparameters and second weight hyperparameters, respectively; Z is an iterative structure diagram; Z * is an actual structure diagram corresponding to the iterative structure diagram;

[0071] The loss function is used to calculate the loss value in the training process of the original network model to obtain the total loss value.

[0072] According to the total loss value, the model parameters of the original network model are optimized by using the Adam optimizer, and the 3D reconstruction model is obtained after iteration.

[0073] Specifically, the data to be reconstructed is determined, comprising:

[0074] Collecting a two-dimensional image of a target building;

[0075] Determining a three-dimensional tensor of the chessboard layout of the model;

[0076] Splicing the two-dimensional image and the three-dimensional tensor to obtain the data to be reconstructed.

[0077] Preferably, the FoldConv module of the original network model is used to perform multi-layer chessboard folding convolution on the data to be reconstructed, and a plurality of convolution features are obtained, comprising:

[0078] Setting a normalization function;

[0079] The two-dimensional coordinates of the two-dimensional image are normalized by using the normalization function; the normalization function comprises: wherein, x, y are the horizontal coordinate and the vertical coordinate of the two-dimensional coordinate, respectively; W, H are the width and the height of the image, respectively; x norm , y norm are the normalized horizontal coordinate and the normalized vertical coordinate, respectively.

[0080] Further, the FoldConv module of the original network model is used to perform multi-layer chessboard folding convolution on the to-be-reconstructed data, to obtain multi-layer convolution features, including:

[0081] determining the folding convolution kernel and the folding parameter of the first layer;

[0082] performing multi-layer chessboard folding convolution on the to-be-reconstructed data by using the FoldConv module according to the folding convolution kernel and the folding parameter of the first layer, to obtain initialization features; the expression of the initialization features is: Z (0) = FoldConv (I, W f(1) , α1); wherein Z (0) is the initialization features; I is the to-be-reconstructed data; W f(1) , α1 are the folding convolution kernel and the folding parameter of the first layer respectively; FoldConv (·) represents multi-layer chessboard folding convolution.

[0083] Specifically, the FeatureFusion module of the original network model is used to perform feature fusion on the multi-layer convolution features, to obtain three-dimensional feature representation, including:

[0084] determining the folding convolution kernel and the folding parameter of the lth layer;

[0085] obtaining the three-dimensional feature representation of the (l-1) th layer;

[0086] performing feature fusion on the three-dimensional feature representation of the (l-1) th layer by using the FeatureFusion module according to the folding convolution kernel and the folding parameter of the lth layer, to obtain the three-dimensional feature representation of the lth layer; the expression of the three-dimensional feature representation of the lth layer is:

[0087] Z (l) = FeatureFusion (Z (l-1) , W f(l) , α l );

[0088] wherein Z (l) is the three-dimensional feature representation of the lth layer; FeatureFusion (·) represents feature fusion operation.

[0089] Further, the multi-layer perceptron of the original network model is used to process the three-dimensional feature representation, to obtain hierarchical perception weight, including:

[0090] processing the three-dimensional feature representation of the lth layer by using the multi-layer perceptron, to obtain the perception feature of the lth layer; the expression of the perception feature of the lth layer is: g (l) = MLP (Z(l) ) ; wherein g (l) is the perception feature of the l-th layer; MLP(·) represents a multi-layer perception processing;

[0091] The perception feature of the l-th layer is calculated using an activation function to obtain the hierarchical perception weight of the l-th layer. The expression of the hierarchical perception weight of the l-th layer is: ω (l) = σ(g (l) ) ; wherein ω (l) is the hierarchical perception weight of the l-th layer; σ(·) is the activation function.

[0092] Specifically, the overall theory of the embodiment is as follows:

[0093] 1) Input data

[0094] Suppose there are image data from different views, denoted as X = x1, x2, …, wherein x e represents the image matrix of the e-th view. At the same time, the layout information of the chessboard needs to be input, denoted as C, representing the coordinate information and labels (such as color, edge, etc.) of each cell on the chessboard. The input data X and C will be used as the input of the network.

[0095] 2) Network architecture initialization

[0096] A two-encoder chessboard folding convolutional network is designed for source encoding and target encoding. The network architecture is as follows:

[0097] Z = g θ (X, C)

[0098] wherein Z represents the output feature matrix of the network, θ represents the network parameters, and g θ represents the network forward propagation function. The network structure mainly includes a source encoder and a target encoder, which respectively encode the source image and the chessboard structure.

[0099] 3) Feature extraction

[0100] First, the features of the multi-view image are extracted by the source encoder:

[0101]

[0102] wherein Z s represents the feature matrix output by the source encoder, is the feature extraction function of the source encoder, mainly extracting two-dimensional coordinates and rgb features.

[0103] Then, the features of the chessboard layout are extracted by the target encoder:

[0104]

[0105] where Z t represents the point cloud feature matrix output by the target encoder, is the feature extraction function of the target encoder, which here corresponds to the RGB color and two-dimensional coordinate tensor.

[0106] Further, a network architecture based on chessboard folding convolution is designed and implemented. The selection and design of the convolution kernel: determine how the convolution kernel slides in the chessboard layout to extract key local features. The depth of the network: considering the complexity of the chessboard layout, multiple layers of convolution are needed to extract higher-level abstract features. Assume K is the convolution kernel, which performs convolution operations on the chessboard layout C. The purpose of convolution is to extract local features by applying the convolution kernel K to different regions of the chessboard, resulting in the final feature tensor Z. The convolution kernel K is a structured filter that can extract higher-order features in the chessboard data. Specifically, the convolution kernel may focus on some specific patterns or positional relationships when processing the chessboard, which may not be global but local structural features such as the relative position of the chess pieces on the chessboard. Convolution operation: in each step of convolution operation, the convolution kernel K scans the input feature map C (chessboard layout) and calculates the convolution result. By adjusting the size and stride of the convolution kernel, the receptive field and the granularity of feature extraction can be controlled.

[0107] Preferably, the input data representation. Assume there is a three-dimensional tensor X∈R H×W×D where H and W represent the height and width respectively, and D represents the depth. The depth value corresponding to each pixel position (h, w) is D h,w In addition, the chessboard image I∈R H×W×C2 where C2 represents the number of channels. The input data consists of a chessboard image and a depth map, that is:

[0108] D = [I, X]

[0109] Further, the definition of chessboard folding convolution. Chessboard folding convolution is a special convolution operation used to map two-dimensional image information to three-dimensional space. The basic idea is to perform folding operations on the input data through multiple convolution kernels, thereby generating multiple layers of three-dimensional feature maps. Specifically, define a folding convolution kernel W f ∈R K×K×D where K is the size of the convolution kernel. The folding convolution operation can be represented as:

[0110] Normalized two-dimensional coordinates convert two-dimensional coordinates (x, y) to normalized coordinates in the range [-1, 1]:

[0111]

[0112] Determine the level of the checkerboard according to the level of the checkerboard l (counting from 0), determine the checkerboard where each point is located:

[0113] l = min{i | x norm <-α l ·2 i+1 or y norm <-α l ·2 i+1}

[0114] where i is the level of the checkerboard starting from 0, α l is the folding parameter of the l-th layer, based on 0 to 1.

[0115] Calculate the Z-axis depth according to the level of the checkerboard l and the maximum number of layers L:

[0116]

[0117] where L is the maximum checkerboard level (adjustable to control the folding depth), calculate the three-dimensional coordinates according to the normalized two-dimensional coordinates and the depth value:

[0118] X = x + x norm

[0119] Y = y + y norm

[0120] Z as described above

[0121] Keep the RGB value unchanged: the RGB value is directly mapped to the corresponding point in three-dimensional space:

[0122] R, G, B = Color(x, y)

[0123] The above process is represented by FoldConv, R, G, B represent the coordinate pixel color value, and Color(x, y) represents the color value corresponding to the coordinates (x, y).

[0124] Further, the multi-layer checkerboard folding convolution considers the complexity of the checkerboard layout, which preliminarily requires multi-layer convolution to extract higher-level abstract features, and also considers the three-dimensional folding, which is responsible for two-dimensional to 3D. In order to enhance the expression ability of the network, multi-layer checkerboard folding convolution is introduced. The folding convolution kernel of each layer has different parameters for extracting features at different levels. The folding convolution operation of the l-th layer is:

[0125] Z (l) = FoldConv(Z (l-1) , W f(l) , α l )

[0126] Preferably, dynamic hierarchical perception. To capture the dynamic nature of the board layout, a dynamic hierarchical perception module is introduced after the folding convolution of each layer. This module processes the features of each layer through a multi-layer perceptron (MLP) and generates hierarchical perception weights. Specifically, the hierarchical perception module of the l-th layer is:

[0127] g (l) = MLP(Z (l) )

[0128] ω (l) = σ(g (l) )

[0129] In particular, loss function definition. To optimize the network, a loss function needs to be defined. Assuming there is a target three-dimensional structure graph Z, the loss function can be defined as:

[0130]

[0131] To further improve the performance of the network, dynamic hierarchical perception loss and optimization loss are introduced:

[0132] E total = E + λ1E c + λ2E d

[0133] Further, optimization process. The Adam optimizer is used to minimize the total loss function:

[0134]

[0135] where α is the learning rate, is the gradient of the total loss function with respect to the parameter θ, and θ' is the optimized parameter.

[0136] In particular, initial feature extraction: assuming there is an input two-dimensional image I ∈ R H×W×C , where H and W represent height and width, respectively, and C represents the number of channels. Through the initialization of the folding convolution operation, the low-level features of the image are extracted:

[0137] Z (0) = FoldConv(I, W f(1) , α1)

[0138] Feature fusion:

[0139] Through multiple folding convolution operations, the features are gradually fused to generate higher-level three-dimensional feature representations. The feature fusion process of the l-th layer is:

[0140] Z (l) = FeatureFusion(Z (l-1) , W f(l) , αl )

[0141] Dynamic hierarchical perception: The dynamic hierarchical perception module of the first layer is:

[0142] g (l) = MLP(Z (l) )

[0143] ω (l) = σ(g (l) )

[0144] Combination of multi-layer folding convolution: Through the process of chessboard folding convolution with multiple iterations, the point cloud and Mesh structure can be gradually reconstructed. Specifically, all hierarchical perception weights are weighted and summed with the corresponding feature maps to obtain the final feature representation:

[0145]

[0146] where L is the number of layers of the chessboard folding convolution.

[0147] 3D reconstruction output:

[0148] Through the decoding function g, the final feature representation Z (L) is mapped to the three-dimensional space to generate point cloud and Mesh structure:

[0149]

[0150] where Z is the generated three-dimensional structure graph, and g is the decoding function.

[0151] In order to optimize the network, a loss function needs to be defined. Assuming there is a target three-dimensional structure graph Z, the loss function can be defined as:

[0152]

[0153] In order to further improve the performance of the network, dynamic hierarchical perception loss and optimization loss are introduced:

[0154] E total = E + λ1E c + λ2E d

[0155] where E c is the feature extraction loss, E d is the dynamic hierarchical perception loss, and λ1, λ2 are weight hyperparameters.

[0156] Optimization process:

[0157] The Adam optimizer is used to minimize the total loss function:

[0158]

[0159] The network architecture based on the chessboard folding convolution effectively maps two-dimensional image information to three-dimensional space through multi-layer folding convolution and dynamic hierarchical perception modules, and can capture multi-scale features of the chessboard layout. Through the chessboard folding convolution process of multiple iterations, the point cloud and Mesh structure can be gradually reconstructed, and the accuracy and detail expression ability of 3D reconstruction are improved. Finally, the 3D reconstruction task is completed, and the generated Mesh model is compared and optimized with the original 3D data to ensure the accuracy of the reconstruction result. Through multiple iterations of the chessboard folding convolution and the dynamic hierarchical perception module, the complete point cloud and Mesh structure are constructed. At this time, the generated Mesh model needs to be compared and optimized with the original 3D data to further improve the accuracy and detail expression ability of the reconstruction. Through the above steps, a high-quality 3D reconstruction result can be obtained, further improving the performance and practicality of the model. The final 3D effect diagram is referred to Figure 3 .

[0160] The beneficial effects of the present application are as follows:

[0161] The present application can effectively map two-dimensional image information to three-dimensional space by performing multi-layer chessboard folding convolution on the data to be reconstructed, realize adaptive extraction of multi-scale features, capture details and large scene structures at the same time, and avoid the dimensional limitations of traditional convolution in 3D space; through feature fusion of the multi-layer convolution features, the point cloud and Mesh structure can promote each other, gradually improving the accuracy and detail expression ability of the reconstruction; through the hierarchical perception weight, the complexity and diversity of the input image can be automatically adapted, avoiding the limitations of fixed architecture on specific scenes.

[0162] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0163] The principles and implementation modes of the present application are described by applying specific examples in this paper. The above embodiment description is only used to help understand the method and core idea of the present application; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed. In view of the above, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A method for 3D reconstruction of real estate building clusters based on checkerboard folding convolution of point clouds to MESH, characterized in that, include: Collect image data of the target building to obtain the data to be reconstructed; The data to be reconstructed is input into a pre-trained 3D reconstruction model for processing to obtain a three-dimensional structure diagram; The training process of the 3D reconstruction model includes: Collect several sets of the data to be reconstructed; Construct the original network model; Using the original network model The module performs multi-layer chessboard folding convolution on the data to be reconstructed to obtain multi-layer convolutional features; Using the original network model The module performs feature fusion on the multi-layer convolutional features to obtain a three-dimensional feature representation; The three-dimensional feature representation is processed using the multilayer perceptron of the original network model to obtain the hierarchical perceptron weights; The hierarchical perception weights and the three-dimensional feature representations are multiplied and accumulated to obtain the final feature representation; The final feature representation is mapped to a three-dimensional space using the decoding function of the original network model to obtain an iterative structure diagram; The total loss value is obtained by calculating the iterative structure graph using the loss function, and the parameters of the original network model are optimized and iterated using the total loss value to obtain the trained 3D reconstruction model. The data to be reconstructed includes: Acquire two-dimensional images of the target building; Determine the three-dimensional tensor of the chessboard layout of the model; The two-dimensional image and the three-dimensional tensor are stitched together to obtain the data to be reconstructed. Using the original network model The module performs multi-layer chessboard folding convolution on the data to be reconstructed to obtain multi-layer convolutional features, including: Determine the folded convolution kernel and folding parameters for the first layer; Based on the folded convolution kernel and folding parameters of the first layer, using the... The module performs multi-layer chessboard folding convolution on the data to be reconstructed to obtain initial features; the expression for the initial features is: ;in, For the initialization feature; The data to be reconstructed; , These are the folded convolution kernel and the folding parameters of the first layer, respectively. This represents a multi-layered chessboard folded convolution; Using the original network model The module performs feature fusion on the multi-layer convolutional features to obtain a three-dimensional feature representation, including: Determine the folded convolution kernel and the folding parameters of the l-th layer; Obtain the three-dimensional feature representation of the (l-1)th layer; Based on the folded convolution kernel and folding parameters of the l-th layer, using the... The module performs feature fusion on the three-dimensional feature representation of the (l-1)th layer to obtain the three-dimensional feature representation of the lth layer; the expression of the three-dimensional feature representation of the lth layer is: ； in, This represents the three-dimensional features of the l-th layer; This indicates a feature fusion operation.

2. The method for 3D reconstruction of real estate building clusters based on checkerboard folding convolution for point cloud to MESH conversion according to claim 1, characterized in that, The iterative structure graph is calculated using a loss function to obtain a total loss value. This total loss value is then used to optimize and iterate the parameters of the original network model, resulting in the trained 3D reconstruction model, including: Define a loss function; the expression for the loss function is: ;in, ; The total loss value; , , These are graph generation loss, feature extraction loss, and dynamic hierarchy perception loss, respectively. , These are the first weight hyperparameter and the second weight hyperparameter, respectively. This is an iterative structure diagram; This is the actual structure diagram corresponding to the iterative structure diagram; The loss value during the training process of the original network model is calculated using the loss function to obtain the total loss value; Based on the total loss value, the Adam optimizer is used to optimize the model parameters of the original network model, and the 3D reconstructed model is obtained after iteration.

3. The method for 3D reconstruction of real estate building clusters based on checkerboard folding convolution for point cloud to MESH conversion according to claim 1, characterized in that, Using the original network model Before the module performs multi-layer chessboard folding convolution on the data to be reconstructed to obtain multi-layer convolutional features, it includes: Define a normalization function; The two-dimensional coordinates of the two-dimensional image are normalized using the normalization function; the normalization function includes: , ;in, , These are the x-coordinate and y-coordinate of the two-dimensional coordinate system, respectively. , These are the width and height of the image, respectively; , These are the normalized x-axis and normalized y-axis, respectively.

4. The method for 3D reconstruction of real estate building clusters based on checkerboard folding convolution for point cloud to MESH conversion according to claim 1, characterized in that, The three-dimensional feature representation is processed using a multilayer perceptron of the original network model to obtain hierarchical perceptron weights, including: The three-dimensional feature representation of the l-th layer is processed using the multilayer perceptron to obtain the perceptual features of the l-th layer; the expression of the perceptual features of the l-th layer is: ;in, The perceptual features of the l-th layer; This indicates multi-layer sensing processing; The hierarchical perceptual weights of the l-th layer are obtained by calculating the perceptual features of the l-th layer using an activation function; the expression for the hierarchical perceptual weights of the l-th layer is: ;in, The hierarchical perception weights of the l-th layer; is the activation function.

5. A 3D reconstruction system for real estate building clusters based on checkerboard folding convolution for point cloud to MESH conversion, characterized in that, The system applied to the 3D reconstruction method for real estate building clusters based on checkerboard folding convolution for point cloud to MESH as described in claim 1, comprises: The model generation module is used to build and train 3D reconstruction models; The data acquisition module is used to determine the data to be reconstructed; Multi-layer convolutional modules are used to utilize the original network model. The module performs multi-layer chessboard folding convolution on the data to be reconstructed to obtain multi-layer convolutional features; The feature fusion module is used to utilize the original network model. The module performs feature fusion on the multi-layer convolutional features to obtain a three-dimensional feature representation; The perception weight extraction module is used to process the three-dimensional feature representation using the multilayer perceptron of the original network model to obtain the hierarchical perception weights. The weighted fusion module is used to multiply and accumulate the hierarchical perception weights and the three-dimensional feature representations to obtain the final feature representation. The spatial mapping module is used to map the final feature representation to a three-dimensional space using the decoding function of the original network model to obtain a three-dimensional structure diagram.

6. An electronic device, characterized in that, include: At least one processor, and a memory communicatively connected to the processor; wherein the memory stores instructions executable by the processor, the instructions being executed by the processor to enable the processor to perform a 3D reconstruction method for real estate building clusters based on checkerboard folding convolution of any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to execute any one of the following methods for 3D reconstruction of real estate building clusters based on checkerboard folding convolution of point clouds to MESH: a method according to any one of claims 1 to 4.

Images