Three-dimensional entity reconstruction method and system based on end-to-end three-dimensional entity reconstruction network

Abstract

The invention discloses a three-dimensional entity reconstruction method and system based on an end-to-end three-dimensional entity reconstruction network, the end-to-end three-dimensional entity reconstruction network comprises a depth estimation module, a pose estimation module, an instance segmentation module, an increment aerial triangulation module, a point cloud fusion module, an entity generation module, a surface optimization module and a texture mapping module; according to the method, the tasks are organically integrated together in a cascading or parallel connection mode between modules, interact with each other and supplement each other, training is carried out in combination with a proposed multi-task loss function, the task precision is improved at the same time, a high-precision three-dimensional entity model is reconstructed in an end-to-end mode, and the real-time performance of the three-dimensional entity model is improved. The method has a wide application prospect in the aspects of live-action three-dimensional construction, digital twinning, universe, smart city and the like.

Description

Technical Field

[0001] The present invention belongs to the technical field of digital photogrammetry and computer vision three-dimensional reconstruction, and relates to a three-dimensional entity reconstruction method and system, and specifically to a three-dimensional entity reconstruction method and system based on an end-to-end three-dimensional entity reconstruction network. Background Art

[0002] Three-dimensional entity reconstruction is to establish an object model (three-dimensional entity model) for three-dimensional objects that contains attribute information, geometric models, behavioral models, and corresponding calling interfaces. It is the basis for processing, operating, and analyzing its properties in a computer environment. In digital photogrammetry and computer vision, it is the process of reconstructing three-dimensional entities based on single-view or multi-view images. Three-dimensional entity reconstruction has been widely used in real-life three-dimensional construction, digital twins, metaverse, smart cities, historical and cultural relics protection, and other fields. Since the Ministry of Natural Resources issued the "Notice on Comprehensively Promoting the Construction of Real-life Three-dimensional China", the construction of urban real-life three-dimensional is still in full swing. Realizing efficient, high-precision, and highly robust end-to-end three-dimensional entity reconstruction is of great significance for the production of basic urban geographic entity data.

[0003] Traditional 3D entity reconstruction based on visual geometry has been widely studied and has been maturely used in various 3D reconstruction professional software such as Smart3D, COLMAP, PhotoScan, etc., but it performs poorly in application scenarios such as weak textures, non-Lambertian surfaces, and transparent reflections. With the development of deep learning in 3D entity reconstruction, not only the above problems have been overcome, but also the reconstruction accuracy and efficiency have been greatly improved. For example, multi-view geometry networks such as PatchMatchNet, CT-MVSNet, and RayMVSNet++ have improved the robustness and accuracy of multi-view depth estimation under the condition of known camera poses, but in more general scenarios, no camera poses or intrinsic parameters are given; pose estimation networks such as PoseNet and DiffPoseNet combine SIFT features or depth information to improve the accuracy of camera pose estimation, but do not perform more precise global optimization of camera poses; OneFormer3D, DC3D, etc. directly perform instance segmentation on 3D point clouds to achieve higher accuracy; Vis2Mesh, DHS3DMesh, etc. have achieved better results in the task of generating 3D point cloud surfaces.

[0004] According to the current research status, some shortcomings of the current research are summarized:

[0005] (1) There is a lack of an end-to-end 3D entity reconstruction network that covers the entire process from multi-view image input to 3D modeling. Most 3D reconstruction networks based on deep learning only generate depth maps or point clouds without performing surface optimization and texture mapping. Currently, there is still no end-to-end multi-task integrated network that can simultaneously complete depth estimation, camera pose estimation, 2D instance segmentation, point cloud fusion, point cloud instance segmentation, incremental aerial triangulation, cyclic reconstruction, 3D surface model generation, and texture mapping;

[0006] (2) Current 3D entity reconstruction networks often ignore the monomerization problem, which leads to serious adhesion of the reconstructed 3D mesh model, which brings great inconvenience to the management, query and spatial analysis of the 3D model;

[0007] (3) Current 3D entity reconstruction networks often ignore semantic issues, which results in the lack of semantic information in the reconstructed 3D models and the inability to effectively identify 3D entities and obtain their attribute information;

[0008] (4) The depth maps generated by the 3D entity reconstruction network are often normalized, which leads to the problem of scale inconsistency between the depth maps of multiple views, making it difficult to generate multiple views by projecting the camera pose into a unified object space. Figure 1 Consistent 3D point cloud.

[0009] (5) Current 3D entity reconstruction networks often fail to implement texture mapping. 3D models without texture information lack realism, which is not conducive to the intuitive expression and display of 3D models. Summary of the invention

[0010] In view of the technical problems in the prior art such as the lack of a network that simultaneously meets the requirements of end-to-end, multi-tasking and high reliability, the purpose of the present invention is to provide an end-to-end, high-reliability, instantiation-supported 3D entity reconstruction method and system based on an end-to-end 3D entity reconstruction network, which can not only achieve high-reliability incremental cyclic reconstruction, but also reconstruct a monomeric and semantic 3D entity model.

[0011] The technical solution adopted by the method of the present invention is: a 3D entity reconstruction method based on an end-to-end 3D entity reconstruction network, wherein the end-to-end 3D entity reconstruction network includes a depth estimation module, a pose estimation module, an instance segmentation module, an incremental spatial triangulation module, a point cloud fusion module and an entity generation module;

[0012] Multi-view image Combine the images in pairs in the order of 1 and 2, 2 and 3, ..., n-1 and n to obtain n-1 pairs of sequence images, which are sequentially input into the end-to-end 3D entity reconstruction network, and the multi-view images are obtained through the depth estimation module. Depth map Through the pose estimation module, multi-view images are obtained Camera parameters and pose Through the instance segmentation module, a multi-view image is obtained. Instance Segmentation and The incremental triangulation module is used to obtain the same-name points between the multi-view images. Generate optimized camera parameters and poses With the sparse three-dimensional points, the depth value scale of the multi-view images is further unified through the point cloud fusion module; Through the entity generation module, the point cloud fusion module is synchronized in parallel to fuse the instance information to generate a complete three-dimensional entity point cloud DSM n .

[0013] Preferably, the depth estimation module, based on the Lite-Mono lightweight network, adds a [7×7,1] convolution layer in the initial encoding stage, adds a skip connection in the decoding stage, introduces a depth separable convolution module and a multi-scale linear attention module in the third and fourth stages, and adds a self-attention mechanism in the decoding part;

[0014] The depth estimation module, in the third and fourth stages of the encoding process, first uses depth separable convolution DSConv to extract channel-by-channel local features DSConv(X) from the feature map obtained in the previous stage:

[0015] DSConv(X)=Dropout(PwConv(SE(DwConv(PwConv(X)))));

[0016] Among them, Dropout is a discard layer, PwConv is a 1×1 point-by-point convolution, SE is composed of a global average pooling and two fully connected layers, DwConv is a 5×5 channel-by-channel convolution, and X is an image feature map;

[0017] The depth estimation module uses multi-scale linear attention MS Extract multi-scale global features:

[0018]

[0019] Among them, Attention RG It is a ReLU-based global attention, and uses channel-by-channel convolutions of different sizes to extract multi-scale global features. Q, K, and V represent the query, key, and value matrices obtained by the feature map through the linear layer, respectively;

[0020] The depth estimation module, during the decoding process, adds a self-attention mechanism after the jump connection of the upsampling block fuses the higher-resolution image features. The upsampling block consists of a [7×7, 1] convolution layer, an upsampling layer, a [7×7, 1] convolution layer, a jump connection, and a self-attention mechanism; the multi-scale feature map output by the encoder is output as a normalized inverse depth map through the upsampling block and the depth prediction head with resolutions of 1:1, 1:2, and 1:4, respectively.

[0021] Preferably, the pose estimation module comprises a pose encoder having 17 convolutional layers, 5 pooling layers, a pose decoder having 4 convolutional layers, an average pooling layer and a fully connected layer;

[0022] The paired sequence images are first extracted through 17 convolutional layers of the pose encoder, and then downsampled through 5 pooling layers to obtain multi-scale feature maps. The minimum feature map is then input into the pose decoder for decoding through 4 convolutional layers, and finally output through an average pooling layer and a fully connected layer. The pose estimation module estimates the camera intrinsic parameters {K i-1 ,K i} and relative pose Among them, K i and They represent the camera intrinsic parameters of the i-th image and its camera pose relative to the i-1 image, respectively. are the identity matrix and the zero matrix respectively;

[0023] Among them, the camera intrinsic parameter matrix f x With f y They represent the product of the focal length and the number of pixels per millimeter in the u-axis and v-axis directions of the image coordinate system, respectively. x With c y They represent the positions of the optical center in the u-axis and v-axis directions of the image coordinate system respectively, and s is the tilt factor. Therefore, the camera internal parameter matrix K has a total of 5 degrees of freedom; the rotation matrix R includes three Euler angles: pitch angle, yaw angle, and roll angle; the translation matrix t includes the translation amounts in the x-, y-, and z-directions, with a total of 6 degrees of freedom.

[0024] As a preference, a single depth map scale unification strategy is implemented for the output of the depth estimation module to unify the depth value of a single image. in represents the depth map of image i generated by image i-1 and image i;

[0025] First, perform perspective transformation (homography transformation) on the two images to obtain the overlapping parts between the two images. Then find the intersection of the overlapping parts of the i-th image and its previous and next images Then perform linear regression on the corresponding pixel depth values ​​in the front and back depth maps corresponding to the intersection part During regression, the RANSAC algorithm is used to remove noise points and abnormal depth values; a and b represent the slope and intercept of the regression parameters respectively; then the depth value of the overlapping part with the next image is linearly transformed. in, Represents the estimated values ​​of a and b during regression; finally, the depth values ​​of the overlapping parts before and after are taken as the union, that is, the depth value of the single image is unified

[0026] Preferably, the instance segmentation module is composed of a SAMI pre-trained lightweight encoder as an image encoder and a SAM default mask decoder as a decoder;

[0027] The pre-training of the SAMI encoder uses the image features extracted by the SAM encoder as a guide. First, a mask image is randomly generated from the image, and then the mask image generates an unmasked embedding layer through a lightweight encoder. Then, through cross-attention decoding, the mask unit features are used as queries, and the unmasked features and mask features of the encoder are used as keys and values ​​to construct cross-attention. The output features of the mask unit from the cross-attention decoder are merged with the output features of the unmasked unit from the encoder. The combined features will be reordered to the original position of the input image; then the output features are aligned with the feature dimensions from the SAM image encoder using a linear projection head, and then the features are combined with the image features extracted by the SAM encoder to construct a reconstruction loss for training. The obtained SAMI pre-trained encoder is used as the image encoder of the instance segmentation network; the image features extracted by the SAMI pre-trained encoder are decoded with the SAM default mask decoder to generate a two-dimensional instance segmentation result, and the depth information generated by the joint depth estimation network is retrained.

[0028] Preferably, the incremental aerial triangulation module first extracts key points from the two images using the Fostner operator, projects the key points of one image into a three-dimensional space through a depth map and camera intrinsic parameters and posture to generate three-dimensional points, then uses the camera intrinsic parameters and posture of another image to back-project into the image to generate projection points, searches for the key point closest to the projection point within a preset radius around the projection point as a matching point, and generates matching points of the two images, i.e., points with the same name;

[0029] Then jointly optimize the camera intrinsic parameters and pose parameters The 3D scene point coordinates corresponding to the depth map projection of the key point Minimize the reprojection error:

[0030]

[0031] Among them, the ρ function projects the three-dimensional point set P into the image space, q i,j =[x i,j ,y i,j ,1] T is the normalized homogeneous pixel coordinate of the image, Contains all 3D points and camera parameters; the optimized camera intrinsic parameters, pose and 3D points are obtained through the Levenberg-Marquardt (LM) algorithm m represents the number of 3D scene points.

[0032] As a preference, the point cloud fusion module first determines the key points of the multi-view images as points with the same name, and then converts each of them from the depth map to the point cloud. is the three-dimensional same-name point; if the coordinate system is not unified, that is, p ij ≠p i+1j , then calculate the rigid registration relationship between the two, use the RANSAC algorithm to remove noise points and abnormal depth values, and then use the depth map to convert all remaining non-key points into three-dimensional points and perform the above-calculated rigid relationship transformation, convert them into a unified three-dimensional space coordinate system, and generate a fused three-dimensional point cloud;

[0033] The calculation of rigid registration is as follows: The following registration relationship is calculated:

[0034]

[0035] Among them, R and T are the rotation matrix and translation matrix of the rigid registration relationship, respectively, and m and k represent the number of three-dimensional points before and after the RANSAC algorithm is used.

[0036] Preferably, the entity generation module, for any two sets of fused point clouds and First establish the mapping relationship M, that is, if the point and There is a rigid registration relationship between them, then (i,j)∈M; then use the mapping relationship M to X 1 The point set Q for each 2D instance label x in 1 Query in X 2 The corresponding point set Q 2 And its corresponding instance y; then count Q separately 1 With Q 2 The points are recorded as and And count X 2 The number of points of instance y in like Then it means X 1 Instances of x and X 2The instance y is a highly overlapping instance label, that is, when the point cloud is fused, the point set Q 1 The instance label of X is changed to instance y to merge the instances; in order to make all instances in the point cloud be traversed, 2 Each instance in the is symmetrically operated in both directions until all instances are merged to generate a 3D solid point cloud;

[0037] For any local point cloud generated by n images For example, n is the total number of points in the image; first, the adjacent point cloud and Merge into 3D solid point cloud Then and Merge into a new 3D solid point cloud Repeat n-2 times until all point clouds are merged into a 3D solid point cloud of the entire scene. The above two-way merging strategy is used to merge the two groups of point clouds. is represented as a two-way merge operation, while the bottom-up merge rule is expressed as:

[0038]

[0039] Among them, the point cloud that has been fused in the i-th fusion And the point cloud that has not been fused A new fused point cloud is merged Until the 2D instance information corresponding to all point clouds is merged to generate a 3D solid point cloud DSM n .

[0040] Preferably, the three-dimensional entity point cloud DSM n , further through the surface optimization module and the texture mapping module, the surface optimized three-dimensional solid model M0 is obtained, and the final textured three-dimensional solid model M1 is generated;

[0041] The surface optimization module first performs point-by-point feature encoding, extracts point-by-point local features V0 from a three-dimensional solid point cloud with a number of n according to n×m feature vectors and n×3 xyz coordinate information through a point cloud network based on a Transformer architecture; the point cloud network of the Transformer architecture, based on the Point Transformer network, uses cross attention to encode the position feature code P after position encoding of the point cloud. i It is embedded into the point cloud features as a query; three consecutive point cloud self-attention modules are used to extract point-by-point local features of the point cloud:

[0042] P i =MLP(p i -pj )=ReLU(Linear(Linear(p i -p j )));

[0043]

[0044]

[0045] X k+1 =Linear(Attention self (Linear(X k )))+X k ;

[0046] Among them, the position-encoded multilayer perceptron MLP consists of two linear layers and a ReLU activation layer. i With p j is the three-dimensional coordinates of point i and point j, point j is the local area χ of point i i Point inside; F i is the point cloud feature, Q=P i W q , K=V=F i W k , Q is the query generated by the position encoding, K and V represent the key and value generated by the point cloud features; Y i represents the output feature of point cloud self-attention, X i With X j Represents the features of input point i and point j, ψ and α represent different linear layers, γ represents that the multilayer perceptron consists of two linear layers and one ReLU activation layer, and the Softmax function is used for standardization, and ⊙ represents the multiplication of corresponding elements of the matrix; X k represents the input features of the k-th point cloud self-attention module, k = 1, 2, 3, X k+1 Represents the features of the k-th output and the k+1-th input, and performs residual connection on each module. After passing through three point cloud self-attention modules, the final feature X4 is output as the extracted point-by-point local feature V0; then all point features belonging to the same voxel unit are integrated through the average pooling layer to obtain the shallow local volume feature H×W×D×C0 of the point cloud, where H, W, D, and C0 represent the height, width, depth, and number of channels of the volume feature, respectively;

[0047] Then, the volume feature encoding is performed, and a three-dimensional U-shaped network with a hybrid architecture of CNN and Transformer is used to extract the local and global volume features V of the point cloud from the shallow local volume features of the point cloud. In the encoding part, the first and second layers are [3×3×3,1] convolutional layers, retaining the original resolution features H×W×D×C1; the third layer is a [3×3×3,2] convolutional layer, which is averagely pooled with the original volume features and then connected to complete downsampling. The fourth, fifth, and sixth layers are 3D dilated convolutional layers with expansion rates r=1,2,3, and the seventh layer is 3D deformable attention. Deform :

[0048]

[0049] The input volume feature X is The query matrix Q is obtained by passing the volume features through a linear layer, x Q represents the three-dimensional coordinates of the query matrix Q, M represents the number of attention heads, K represents the fixed number of key samples, and W m ′、W m Represents the value matrix before and after 3D deformable attention, A mqk , Δx mpk represents the attention weight and sampling bias of the qth query and the kth sampling point of the mth attention head;

[0050] Then repeat the two stages for the third to seventh layers to obtain multi-scale volume features In the decoding stage, after three repeated stages of [3×3×3,1] convolutional layer, upsampling layer, skip connection, and [3×3×3,1] convolutional layer, the local and global volume features V of the point cloud are obtained;

[0051] Then, the occupancy field is decoded, and the volume feature V is interpolated by trilinear interpolation to determine the feature F corresponding to the position of any point p in the three-dimensional space. p , then p and F p Input to a lightweight multi-layer perceptron MLP, predicting the probability that p is classified as an occupied field within a 3D surface as P(p)∈R 3 :

[0052]

[0053] Finally, multi-resolution isosurface extraction and marching cubes are implemented to extract the surface mesh and generate a three-dimensional surface, i.e., a three-dimensional solid model M0;

[0054] The surface optimization module is a trained module. During the training, supervised training is performed to first calculate the ground real occupancy field. Then the binary cross entropy function is used to calculate the loss function, which penalizes the difference between the predicted value and the true occupancy value;

[0055]

[0056] Where Q represents the set of points sampled from the watertight grid boundary;

[0057] Then perform the sign uncertainty optimization, that is, use the unsigned binary cross entropy function to calculate the loss function and obtain the consistency constraint between the occupancy field and the unsigned input:

[0058]

[0059] The occupied area Using unsigned decoding:

[0060]

[0061]

[0062] in, is the ground truth surface The point set obtained, From the non-curved body The sampled point set; due to It is unknown in the test phase, and the observed surface S is taken as The approximation of , and identify random sampling points in three-dimensional space as non-surface point sets

[0063] The texture mapping module combines the camera's internal parameters and posture, adopts a texture mapping algorithm, selects images with smaller deformation from multiple views to first perform texture mapping on the three-dimensional surface, and then selects images from other perspectives to repeatedly paste textures to deal with occlusion problems, merges these textures together, completes seamless texture mapping, and generates the final textured three-dimensional solid model M1.

[0064] The technical solution adopted by the system of the present invention is: a 3D entity reconstruction system based on an end-to-end 3D entity reconstruction network, comprising:

[0065] one or more processors;

[0066] A storage device is used to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the three-dimensional solid reconstruction method based on the end-to-end three-dimensional solid reconstruction network.

[0067] The beneficial effects of the technical solution proposed by the present invention are:

[0068] (1) The present invention adopts an end-to-end 3D entity reconstruction network. Compared with other 3D reconstruction networks (MVSNet), it is no longer limited to a specific task (depth estimation) or condition (known pose or point cloud), and can achieve high-efficiency end-to-end reconstruction directly from multi-view images to 3D entity models. Its advantage is that the network framework only inputs multi-view images and can output 3D entity reconstruction results of multi-view image scale-consistent depth maps, camera internal parameters and poses, instance segmentation maps, 3D entity point clouds, and 3D entity models.

[0069] (2) The present invention proposes a multi-task integrated network. In the depth estimation and instance segmentation modules, unlike using known depth information or instance information as a supervisory signal to guide the instance segmentation or depth estimation task of a monocular image, the depth and instance information of multi-view images are effectively integrated to mutually improve the prediction accuracy. In the point cloud fusion module and the entity generation module, unlike the general mode of first generating a three-dimensional model or point cloud and then performing individualization and semanticization, the present invention simultaneously generates a three-dimensional entity point cloud by fusion, and then generates a three-dimensional entity model through a surface optimization network. The advantage of the former is that the depth map and the instance segmentation map have a stronger consistency, which is convenient for the subsequent generation of entity point clouds or models. The advantage of the latter is that there is no need to perform additional individualization and semanticization steps on the three-dimensional model, and a semantic three-dimensional entity model can be directly generated.

[0070] (3) The present invention proposes a key point object space projection search strategy. Different from the feature matching operator (SIFT) that generates matching points, the Fostner operator is used to extract key points. Three-dimensional points are generated in the object space according to the depth map and the camera posture, and the matching is completed by reprojection search. The advantage of the present invention is that it generates more reliable points of the same name between multiple views, which can be used in subsequent self-checking and bundle adjustment.

[0071] (4) The present invention proposes a depth scale unification strategy to overcome the scale inconsistency problem between depth maps generated from multi-view images. That is, for a single image, in order to make its depth information more comprehensive, the depth map is estimated twice for the single image based on the previous and next images. The normalized depth maps of the two estimates have certain differences in image range and depth value, and need to be fused to obtain a larger depth map range and consistent depth values. However, for multi-view images, it is impossible to normalize these depth values ​​to [0,1] and lose the scale information of the depth map, and then project it into a unified object space to generate a point cloud. The depth scale unification strategy not only unifies the depth map of a single image using the important depth information of the overlapping parts of the multi-view and the previous and next images, but also unifies the depth maps of different images using the registration relationship of the same-name points in the multi-view images. Its advantage is that the depth map after scale unification can be projected into a multi-view point cloud. Figure 1 Consistent 3D point cloud.

[0072] (5) The present invention proposes a two-way merging strategy to overcome the problem of inconsistent instance information between multiple views, that is, the instance labels generated by two-dimensional instance segmentation of different images often have differences in order, and cannot be effectively matched according to the default order. This will cause the point clouds generated from different perspectives of a three-dimensional entity to have different instance labels, which is unreasonable. The two-way merging strategy can use the mapping relationship between the three-dimensional points generated between multiple views to merge the point clouds with the same instance information generated from different views. Its advantage is that it effectively unifies the instance information between multiple views, so that the same object has consistent instance labels under multiple perspectives.

[0073] (6) The present invention proposes an incremental cyclic reconstruction strategy, which gradually adds paired images to cyclically reconstruct three-dimensional entity point clouds and three-dimensional entity models, and continuously optimizes the camera pose and three-dimensional entity point cloud in the process. The advantage is that it avoids the simultaneous processing and storage of a large number of multi-view images and three-dimensional entity data, significantly reducing the instantaneous high load requirements on memory resources. The system can complete the construction and optimization of three-dimensional point clouds and models with a smaller memory overhead, thereby improving computing efficiency and resource utilization; it allows new images to be flexibly added to generate new three-dimensional entity point clouds based on the already generated three-dimensional entity point clouds and integrate them into the original point cloud to continue to complete the reconstruction; the cyclic optimization mechanism can reduce the cumulative error generated when multiple views are reconstructed two by two, thereby improving the reliability of the overall reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] The technical solution of the present invention is further described below using embodiments and specific implementation methods. In addition, in the process of describing the technical solution, some drawings are also used. For those skilled in the art, other drawings and the intention of the present invention can also be obtained based on these drawings without paying creative work.

[0075] Figure 1 A schematic diagram of a method according to an embodiment of the present invention;

[0076] Figure 2 Schematic diagram of a depth estimation module and a pose estimation module according to an embodiment of the present invention;

[0077] Figure 3 It is a schematic diagram of an instance segmentation module, a point cloud fusion module and an entity generation module according to an embodiment of the present invention;

[0078] Figure 4 Schematic diagram of a surface optimization module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0079] In order to facilitate ordinary technicians in the field to understand and implement the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the implementation examples described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0080] Please see Figure 1 The present embodiment provides a 3D entity reconstruction method based on an end-to-end 3D entity reconstruction network, wherein the end-to-end 3D entity reconstruction network includes a depth estimation module, a pose estimation module, an instance segmentation module, an incremental aerial triangulation module, a point cloud fusion module, an entity generation module, a surface optimization module, and a texture mapping module.

[0081] In this embodiment, the multi-view image Combine the images in pairs in the order of 1 and 2, 2 and 3, ..., n-1 and n to obtain n-1 pairs of sequence images, which are sequentially input into the end-to-end 3D entity reconstruction network, and the multi-view images are obtained through the depth estimation module. Depth map Through the pose estimation module, multi-view images are obtained Camera parameters and pose Through the instance segmentation module, a multi-view image is obtained. Instance Segmentation and The incremental triangulation module is used to obtain the same-name points between the multi-view images. Generate optimized camera parameters and poses With the sparse three-dimensional points, the depth value scale of the multi-view images is further unified through the point cloud fusion module; Through the entity generation module, the point cloud fusion module is synchronized in parallel to fuse the instance information to generate a complete three-dimensional entity point cloud DSM n The three-dimensional entity point cloud DSM n , further through the surface optimization module and the texture mapping module, the surface optimized three-dimensional solid model M0 is obtained, and the final textured three-dimensional solid model M1 is generated.

[0082] Please see Figure 2 This embodiment specifically includes the following steps:

[0083] Step S1: First, multi-view images The images are input into the pose estimation module and the depth estimation module in the order of 1 and 2, 2 and 3, ..., n-1 and n. The depth maps of the two images and the camera internal parameters and pose are randomly initialized. Then, the loss is constructed using the photometric consistency constraint between views, and self-supervised training is performed. Finally, the normalized depth maps of the two images are generated bidirectionally. Camera intrinsic parameters and pose (in represents the depth map of image i generated by image i-1 and image i, K i and They represent the camera intrinsic parameters of the i-th image and its camera pose relative to the i-1 image, respectively. are the identity matrix and the zero matrix respectively).

[0084] Please see Figure 2 , the pose estimation module and the depth estimation module:

[0085] The pose estimation module inputs the paired sequence images into the same pose estimation network to estimate the camera internal parameters and camera pose. The pose estimation network includes a pose encoder with 17 convolutional layers and a pose decoder with 4 convolutional layers.

[0086] The paired sequence images are first extracted through 17 convolutional layers of the pose encoder, and then downsampled through 5 pooling layers to obtain multi-scale feature maps. The minimum feature map is then input into the pose decoder for decoding through 4 convolutional layers, and finally output through an average pooling layer and a fully connected layer. The camera intrinsic parameters and pose are different from the existing pose estimation network that only estimates the camera pose with 6 degrees of freedom. This network estimates the camera intrinsic parameters {K i-1 ,K i} and relative pose

[0087] The camera intrinsic parameter matrix K is expressed as:

[0088]

[0089] where f x With f y They represent the product of the focal length and the number of pixels per millimeter in the u-axis and v-axis directions of the image coordinate system, respectively. x With c y They represent the positions of the optical center in the u-axis and v-axis directions of the image coordinate system, respectively, and s is the tilt factor, so the camera internal parameter matrix K has a total of 5 degrees of freedom. The rotation matrix R includes three Euler angles: pitch angle, yaw angle, and roll angle, and the translation matrix t includes the translation amount in the xyz direction, with a total of 6 degrees of freedom.

[0090] The depth estimation module inputs the paired sequence images into the same depth estimation network to estimate the image depth map. Based on the Lite-Mono lightweight network, the depth estimation network has been improved as follows: in order to retain the high-resolution features of the original image, especially the important edge information, for subsequent skip connections, a [7×7,1] convolution layer is added in the initial encoding stage, and a skip connection is added in the decoding stage to avoid the loss of high-frequency features (edge ​​information) caused by multiple downsampling encodings; in order to extract channel-by-channel local features and multi-scale global features, and to fuse multi-channel and multi-scale local and global information, a deep separable convolution module and a multi-scale linear attention module are introduced in the third and fourth stages to complement the continuous dilated convolution for extracting multi-scale local features and the cross-covariance attention for extracting channel-by-channel global features; in order to balance the top-heavy encoding and decoding architecture of the entire network, a self-attention mechanism is added in the decoding part, so that the depth information of the three-dimensional entity in the depth map during decoding receives more attention, which is beneficial to the subsequent reconstruction of the three-dimensional entity.

[0091] In the third and fourth stages of the encoding process, the feature map obtained in the previous stage is firstly subjected to the deep separable convolution DSConv to extract the local features of each channel:

[0092] DSConv(X)=Dropout(PwConv(SE(DwConv(PwConv(X))))) (2)

[0093] Among them, Dropout is the discard layer, PwConv is a 1×1 point-by-point convolution, SE consists of a global average pooling and two fully connected layers, and DwConv is a 5×5 channel-by-channel convolution.

[0094] Then use multi-scale linear attention MS Extract multi-scale global features:

[0095]

[0096] Among them, Attention RG It is a global attention based on ReLU instead of the ordinary attention Softmax function. It can effectively convert the complex quadratic computation into linear computation with only a small amount of information loss, and uses channel-by-channel convolutions of different sizes to extract multi-scale global features. Q, K, and V represent the query, key, and value matrices obtained by the feature map through the linear layer, respectively.

[0097] Finally, in the decoding process, in order to enable self-attention to act on the high-frequency edge depth information of the image, after the jump connection of the upsampling block fuses the higher-resolution image features, a self-attention mechanism is added to focus on both the 3D entity structure information with similar depth features and the 3D entity edge information with large depth changes. Therefore, the upsampling block consists of a [7×7,1] convolution layer, an upsampling layer, a [7×7,1] convolution layer, a jump connection, and a self-attention mechanism. The multi-scale feature map output by the encoder is output as a normalized inverse depth map through the upsampling block and the depth prediction head with a resolution of 1:1, 1:2, and 1:4, respectively.

[0098] The pose estimation module and the depth estimation module of this embodiment are trained pose estimation modules and depth estimation modules. During the training, self-supervised training is adopted. Different from the training using the ground truth label, this embodiment uses the photometric consistency constraint between views to construct the loss, that is, for a pixel p in the i-th image i , using the predicted depth map D i , calculate its corresponding 3D point by backprojection. Then the 3D point is transformed to the 3D position of the i-1th image by the predicted camera pose. Its position in the i-1th image is p' i-1 It can be obtained by reprojection, and finally the reprojected i-th image is synthesized from the i-1-th image by bilinear interpolation (<>)

[0099]

[0100] During the training process, the synthetic images With the i-th image I i The photometric loss between is minimized, and the photometric loss can be expressed as a weighted sum of the structural similarity SSIM and the L1 loss:

[0101]

[0102] In addition, in order to smooth the generated inverse depth map, the edge-aware smoothing loss is calculated for the i-th image:

[0103]

[0104] in is the average normalized inverse depth of the i-th image to prevent the gradient from disappearing.

[0105] The final total loss of self-supervised training is expressed as:

[0106]

[0107] Where s is the depth map image with different resolutions output by the depth decoder, n is the total number of images, and λ is set to 10-3 .

[0108] Step S2: Implement a single depth map scale unification strategy to unify the depth value of a single image

[0109] First, perform perspective transformation (homography transformation) on the two images to obtain the overlapping parts between the two images. Then find the intersection of the overlapping parts of the i-th image and its previous and next images Then perform linear regression on the corresponding pixel depth values ​​in the front and back depth maps corresponding to the intersection part During regression, the RANSAC algorithm is used to remove noise points and abnormal depth values; a and b represent the slope and intercept of the regression parameters respectively; then the depth value of the overlapping part with the next image is linearly transformed. in, Represents the estimated values ​​of a and b during regression; finally, the depth values ​​of the overlapping parts before and after are taken as the union, that is, the depth value of a single image is unified

[0110] Step S3: Implement instance segmentation; This embodiment proposes a method of constructing a joint loss with a depth estimation module. First, an image encoder is generated by a lightweight pre-trained SAMI, and then the multi-view image is input into the image encoder and the SAM mask decoder for two-dimensional instance segmentation training. Then, a joint loss function is constructed based on the correspondence between the depth information and the instance information to generate a depth map and two-dimensional instance segmentation result with stronger consistency.

[0111] Please see Figure 3 ,The instance segmentation module of this embodiment inputs all images into the same ,instance segmentation module for instance segmentation. The instance segmentation module is composed of the ,SAMI pre-trained lightweight encoder as the image encoder and the ,SAM default mask decoder as the decoder.

[0112] The pre-training of the SAMI encoder uses the image features extracted by the SAM encoder as a guide. First, a mask image is randomly generated from the image. Then, the mask image generates an unmasked embedding layer through a lightweight encoder. Then, through cross-attention decoding, the mask unit features are used as queries, and the unmasked features and mask features of the encoder are used as keys and values ​​to construct cross-attention. Then, the output features of the mask unit from the cross-attention decoder are merged with the output features of the unmasked unit from the encoder. The combined features will be reordered to the original position of the input image. Then, the linear projection head is used to align the output features with the feature dimensions from the SAM image encoder, and then the features are trained by constructing the reconstruction loss with the image features extracted by the SAM encoder. The obtained SAMI pre-trained encoder is used as the image encoder of the instance segmentation network.

[0113] Finally, the image features extracted by the SAMI pre-trained encoder are decoded by the SAM default mask decoder to generate a two-dimensional instance segmentation result, and the depth information generated by the joint depth estimation network is retrained.

[0114] The instance segmentation module of this embodiment is a trained module. During the training, a joint loss is constructed with the depth estimation module. The specific steps are as follows:

[0115] Step 3.1, construct a joint smoothness loss that first enforces similar depth values ​​for neighboring pixels inside the instance, and then achieves stronger alignment between depth edges and instance contours:

[0116]

[0117] Where P i is the instance label, is the instance contour, is the average normalized inverse depth. For two adjacent pixels (p0, p1), Defined as:

[0118]

[0119] Step 3.2, construct the joint edge discontinuity loss. Adjacent pixels at the edge of an instance often have large depth discontinuities:

[0120]

[0121] Step 3.3, construct a joint triplet loss, that is, perform a large depth difference on the pixels of different instances, so that the L2 distance between the deep feature of one instance and the deep feature of different instances, that is, the negative feature As large as possible, and the L2 distance between the deep features of the same instance, that is, the positive features As small as possible, that is, the triple loss with a margin of m is:

[0122]

[0123] Step S4: Using the incremental triangulation module, a key point object space projection search strategy is proposed to obtain the same-name points between the multi-view images. Then combine the camera internal parameters and pose Use the RANSAC algorithm to remove noise points and abnormal depth values, implement self-calibration bundle adjustment, and generate optimized camera internal parameters and poses with sparse 3D points.

[0124] The key point object space projection search strategy and self-calibration bundle adjustment are specifically as follows:

[0125] Step 4.1, implement the key point object space projection search strategy: first use the Fostner operator to extract the key points of the two images, project the key points of one image into the three-dimensional space through the depth map and the camera's intrinsic parameters and posture to generate three-dimensional points, then use the camera's intrinsic parameters and posture of the other image to back-project it into this image to generate projection points, and search for the key point closest to the projection point within a certain radius around the projection point as the matching point, that is, the matching points (points with the same name) of the two images are generated.

[0126] Step 4.2: Implement self-calibration bundle adjustment: Jointly optimize camera intrinsic parameters and pose parameters The 3D scene point coordinates corresponding to the depth map projection of the key points Minimize the reprojection error:

[0127]

[0128] The ρ function projects the three-dimensional point set P into the image space, q i,j =[x i,j ,y i,j ,1] T is the normalized homogeneous pixel coordinate of the image, Contains all 3D points and camera parameters. The optimized camera intrinsic parameters, pose and 3D points are obtained through the Levenberg-Marquardt (LM) algorithm. m represents the number of 3D scene points.

[0129] Step S5: See Figure 3 This embodiment adopts a point cloud fusion module and proposes a multi-view depth map scale unification strategy to unify the depth value scale of multi-view images and complete point cloud fusion. First, the multi-view images are determined as key points of the same name, and each is converted from the depth map to the point cloud. If the coordinate system is not unified, that is, p ij ≠p i+1j , the rigid registration relationship between the two is calculated, and the noise points and abnormal depth values ​​are eliminated by the RANSAC algorithm. Then, all the remaining non-key points are converted into three-dimensional points using the depth map and the above-calculated rigid relationship transformation is performed to convert them into a unified three-dimensional space coordinate system to generate a fused three-dimensional point cloud.

[0130] The rigid registration is calculated in step S5 as follows: The following registration relationship is calculated:

[0131]

[0132] Where R and T are the rotation matrix and translation matrix of the rigid registration relationship, respectively, and m and k represent the number of three-dimensional points before and after the RANSAC algorithm is used.

[0133] Step S6: See Figure 3 This embodiment adopts an entity generation module, proposes a two-way merging strategy, and fuses instance information in parallel and synchronously with the point cloud fusion module. While performing point cloud fusion, the two-dimensional instance information corresponding to the point cloud is bidirectionally merged, and the instance segmentation of the three-dimensional point cloud is synchronously completed to generate a three-dimensional entity point cloud. For the above steps S1 to S6, an incremental cyclic reconstruction strategy is proposed, which can be expanded to first generate an initial three-dimensional entity point cloud from the first and second images, and then gradually add the second and third,..., n-1 and n images, perform cyclic reconstruction, bundle adjustment optimization, and generate a complete three-dimensional entity point cloud DSM. n .

[0134] Please see Figure 3 , the specific steps of this embodiment using the entity generation module and the two-way merging strategy are:

[0135] For any two sets of fused point clouds and For example, we first establish the mapping relationship M, that is, if the point and There is a rigid registration relationship in step S5 between them, then (i,j)∈M. Then use the mapping relationship M to X 1 The point set Q for each 2D instance label x in 1 Query in X 2 The corresponding point set Q 2 And its corresponding instance y. Then count Q separately 1 With Q 2 The points are recorded as and And count X 2 The number of points of instance y in like Then it means X 1 Instances of x and X 2 The instance y is a highly overlapping instance label, that is, when the point cloud is fused, the point set Q 1 The instance label of X is changed to instance y to merge the instances. In order to traverse all instances in the point cloud, 2 Symmetrical bidirectional operations are performed on each instance in the ensemble until all instances are merged to generate a three-dimensional solid point cloud.

[0136] For any local point cloud generated by n images First, the adjacent point cloud and Merge into 3D solid point cloud Then and Merge into a new 3D solid point cloud Repeat n-2 times until all point clouds are merged into a 3D solid point cloud of the entire scene. The above two-way merging strategy is used to merge the two groups of point clouds. It is represented as a two-way merge operation, and the bottom-up merge rule can be expressed as:

[0137]

[0138] Among them, the point cloud that has been fused in the i-th fusion And the point cloud that has not been fused A new fused point cloud is merged Until the 2D instance information corresponding to all point clouds is merged to generate a 3D solid point cloud DSM n .

[0139] Step S7: Using the surface optimization module, the point cloud network is improved and a new three-dimensional U-shaped network is proposed to extract the point-by-point features and volume features of the three-dimensional entity point cloud, and then the features are input into a multi-layer perceptron (MLP) to generate occupancy field probabilities, generate and optimize the three-dimensional surface, and obtain the three-dimensional entity model M0.

[0140] Please see Figure 4 , this embodiment adopts the specific steps of the surface optimization module:

[0141] Step 7.1, implement point-by-point feature encoding. First, extract point-by-point local features V0 from the n-number of 3D entity point clouds according to the n×m feature vectors and n×3 xyz coordinate information through a point cloud network based on the Transformer architecture. Based on the Point Transformer network, the point cloud network has made the following improvements. In order to more effectively use the spatial structure features of the point cloud to optimize the surface of the 3D entity, after the point cloud is position-encoded, the position feature encoding P is encoded using cross attention. i It is embedded into the point cloud features as a query. To avoid the point cloud features being severely compressed during the downsampling process, which results in the loss of local subtle surface concave-convex information, three consecutive point cloud self-attention modules are used to extract point-by-point local features of the point cloud:

[0142] P i =MLP(p i -p j )=ReLU(Linear(Linear(p i -p j ))) (15)

[0143]

[0144]

[0145] Xk+1 =Linear(Attention self (Linear(X k )))+X k (k=1,2,3) (18)

[0146] The position-encoded multilayer perceptron MLP consists of two linear layers and a ReLU activation layer. i With p j is the three-dimensional coordinates of point i and point j, point j is the local area χ of point i i Points within the area calculated using the K nearest neighbor algorithm. i is the point cloud feature, Q=P i W q , K=V=F i W k , Q is the query generated by the position encoding, K and V represent the key and value generated by the point cloud features. i represents the output feature of point cloud self-attention, X i With X j Represents the features of input point i and point j, ψ and α represent different linear layers, γ represents the multilayer perceptron consisting of two linear layers and one ReLU activation layer, and the Softmax function is used for standardization, and ⊙ represents the multiplication of corresponding elements of the matrix. k represents the input features of the k-th point cloud self-attention module, X k+1 Represents the features of the k-th output and the k+1-th input, and performs residual connection on each module. After passing through three point cloud self-attention modules, the final feature X4 is output as the extracted point-by-point local feature V0. Then, all point features belonging to the same voxel unit are integrated through the average pooling layer to obtain the shallow local volume feature H×W×D×C0 of the point cloud, where H, W, D, and C0 represent the height, width, depth, and number of channels of the volume feature respectively;

[0147] Step 7.2, implement volume feature encoding, and propose a new 3D U-shaped network with a hybrid architecture of CNN and Transformer to extract local and global volume features V of point clouds from shallow local volume features of point clouds. The network mainly uses 3D dilated convolution and 3D deformable attention. The former extracts local volume features with a larger receptive field and captures a wider range of spatial information. The latter fuses the local and global shape information of the point cloud to represent complex geometric structures. Compared with other convolutions or attention, both have smaller computational complexity and better shape perception effects. In the encoding part, the first and second layers are [3×3×3,1] convolution layers, which retain the original resolution features H×W×D×C1. The third layer is a [3×3×3,2] convolution layer, which is connected with the original volume features after average pooling to complete downsampling. The fourth, fifth, and sixth layers are 3D dilated convolution layers with expansion rates r=1,2,3 (similar to step 1.2), and the seventh layer is 3D deformable attention Attention. Deform :

[0148]

[0149] The input volume feature X is The query matrix Q is obtained by passing the volume features through a linear layer, x Q represents the three-dimensional coordinates of the query matrix Q, M represents the number of attention heads, K represents the fixed number of key samples (much smaller than H×W×D / 8), and W m ′、W m Represents the value matrix before and after 3D deformable attention, A mqk , Δx mpk represents the attention weight and sampling bias of the qth query and the kth sampling point of the mth attention head.

[0150] Then repeat the two stages for the third to seventh layers to obtain multi-scale volume features In the decoding stage, after three repeated stages of [3×3×3,1] convolutional layer, upsampling layer, skip connection, and [3×3×3,1] convolutional layer, the local and global volume features V of the point cloud are obtained.

[0151] Step 7.3, implement occupancy field decoding, and use trilinear interpolation to determine the feature F corresponding to the position of any point p in the three-dimensional space. p , then p and F p Input to a lightweight multi-layer perceptron MLP, predicting the probability that p is classified as an occupied field within a 3D surface as P(p)∈R 3 :

[0152]

[0153] The surface optimization module implemented in this paper is a trained surface optimization module. During the training, supervised training is implemented to first calculate the real occupancy field on the ground. The binary cross entropy function is then used to calculate the loss function, which penalizes the difference between the predicted value and the true occupancy value.

[0154]

[0155] Where Q represents the set of points sampled from within the watertight mesh boundary volume.

[0156] Then we perform a signed uncertainty optimization, that is, we use the unsigned binary cross entropy function to calculate the loss function, and obtain the consistency constraint between the occupancy field and the unsigned input without using the surface normal:

[0157]

[0158] The occupied area Using unsigned decoding:

[0159]

[0160]

[0161] in, is the ground truth surface The point set obtained, From the non-curved body The sampled point set. It is unknown in the test phase, and the observed surface S is taken as The approximation of , and identify random sampling points in three-dimensional space as non-surface point sets

[0162] Finally, multi-resolution isosurface extraction and marching cubes are implemented to extract the surface mesh and generate a three-dimensional surface, namely, a three-dimensional solid model M0.

[0163] Step S8: This embodiment adopts a texture mapping module, combines the camera internal parameters and posture in step S4, and adopts a texture mapping algorithm to select images with smaller deformation from multiple views to first perform texture mapping on the three-dimensional surface, and then select images from other perspectives to repeatedly paste textures to deal with occlusion problems, merge these textures together, complete seamless texture mapping, and generate the final textured three-dimensional solid model M1.

[0164] In one implementation, the end-to-end 3D entity reconstruction network is a trained end-to-end 3D entity reconstruction network; during the training, the total loss function used is:

[0165]

[0166] where γ s , γ e , γ t , γ b , γ u They are the coefficients of smoothness loss, edge discontinuity loss, triplet loss, binary cross entropy loss, and unsigned binary cross entropy loss, respectively.

[0167] This embodiment further provides a 3D entity reconstruction system based on an end-to-end 3D entity reconstruction network, including:

[0168] one or more processors;

[0169] A storage device is used to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the three-dimensional solid reconstruction method based on the end-to-end three-dimensional solid reconstruction network.

[0170] The end-to-end 3D entity reconstruction network provided by the present invention includes a depth estimation module, a pose estimation module, an instance segmentation module, an incremental aerial triangulation module, a point cloud fusion module, an entity generation module, a surface optimization module and a texture mapping module; these tasks are organically integrated together by cascading or paralleling modules, interacting and complementing each other, and trained in combination with the proposed multi-task loss function to simultaneously improve the accuracy of the above tasks, and reconstruct a high-precision 3D entity model end-to-end, which has broad application prospects in real-life 3D construction, digital twins, metaverse, smart cities, etc.

[0171] It should be understood that the embodiments described above are part of the embodiments of the present invention, rather than all of the embodiments. In addition, the technical features in the various embodiments or single embodiments provided by the present invention can be combined with each other arbitrarily to form a feasible technical solution. Such combination is not restricted by the sequence of steps and / or the structural composition mode, but must be based on the ability of ordinary technicians in the field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the protection scope required by the present invention.

[0172] It should be understood that the above description of the preferred embodiment is relatively detailed and cannot be regarded as limiting the scope of patent protection of the present invention. Under the enlightenment of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested for the present invention shall be based on the attached claims.

Claims

1. A 3D entity reconstruction method based on an end-to-end 3D entity reconstruction network, characterized in that: The end-to-end 3D entity reconstruction network includes a depth estimation module, a pose estimation module, an instance segmentation module, an incremental spatial triangulation module, a point cloud fusion module and an entity generation module; Multi-view image Combine the images in pairs in the order of 1 and 2, 2 and 3, ..., n-1 and n to obtain n-1 pairs of sequence images, which are sequentially input into the end-to-end 3D entity reconstruction network, and multi-view images are obtained through the depth estimation module. Depth map Through the pose estimation module, multi-view images are obtained Camera parameters and pose Through the instance segmentation module, a multi-view image is obtained. Instance Segmentation and The incremental triangulation module is used to obtain the same-name points between the multi-view images. Generate optimized camera parameters and poses With the sparse three-dimensional points, the depth value scale of the multi-view images is further unified through the point cloud fusion module; Through the entity generation module, the point cloud fusion module is synchronized in parallel to fuse the instance information to generate a complete three-dimensional entity point cloud DSM n .

2. The 3D entity reconstruction method based on an end-to-end 3D entity reconstruction network according to claim 1, characterized in that: The depth estimation module, based on the Lite-Mono lightweight network, adds a [7×7,1] convolution layer in the initial encoding stage, adds a skip connection in the decoding stage, introduces a depth-separable convolution module and a multi-scale linear attention module in the third and fourth stages, and adds a self-attention mechanism in the decoding part; The depth estimation module, in the third and fourth stages of the encoding process, first uses depth separable convolution DSConv to extract channel-by-channel local features DSConv(X) from the feature map obtained in the previous stage: DSConv(X)=Dropout(PwConv(SE(DwConv(PwConv(X))))); Among them, Dropout is a discard layer, PwConv is a 1×1 point-by-point convolution, SE is composed of a global average pooling and two fully connected layers, DwConv is a 5×5 channel-by-channel convolution, and X is an image feature map; The depth estimation module uses multi-scale linear attention MS Extract multi-scale global features: Among them, Attention RG It is a ReLU-based global attention, and uses channel-by-channel convolutions of different sizes to extract multi-scale global features. Q, K, and V represent the query, key, and value matrices obtained by the feature map through the linear layer, respectively; The depth estimation module, during the decoding process, adds a self-attention mechanism after the jump connection of the upsampling block fuses the higher-resolution image features. The upsampling block consists of a [7×7, 1] convolution layer, an upsampling layer, a [7×7, 1] convolution layer, a jump connection, and a self-attention mechanism; the multi-scale feature map output by the encoder is output as a normalized inverse depth map through the upsampling block and the depth prediction head with resolutions of 1:1, 1:2, and 1:4, respectively.

3. The 3D entity reconstruction method based on an end-to-end 3D entity reconstruction network according to claim 1, characterized in that: The pose estimation module includes a pose encoder with 17 convolutional layers, 5 pooling layers, a pose decoder with 4 convolutional layers, an average pooling layer and a fully connected layer; The paired sequence images are first extracted through 17 convolutional layers of the pose encoder, and then downsampled through 5 pooling layers to obtain multi-scale feature maps. The minimum feature map is then input into the pose decoder for decoding through 4 convolutional layers, and finally output through an average pooling layer and a fully connected layer. The pose estimation module estimates the camera intrinsic parameters {K i-1 ,K i } and relative pose Among them, K i and They represent the camera intrinsic parameters of the i-th image and its camera pose relative to the i-1 image, respectively. are the identity matrix and the zero matrix respectively; Among them, the camera intrinsic parameter matrix f x With f y They represent the product of the focal length and the number of pixels per millimeter in the u-axis and v-axis directions of the image coordinate system, respectively. x With c y They represent the positions of the optical center in the u-axis and v-axis directions of the image coordinate system respectively, and s is the tilt factor. Therefore, the camera internal parameter matrix K has a total of 5 degrees of freedom; the rotation matrix R includes three Euler angles: pitch angle, yaw angle, and roll angle; the translation matrix t includes the translation amounts in the x-, y-, and z-directions, with a total of 6 degrees of freedom.

4. The 3D entity reconstruction method based on an end-to-end 3D entity reconstruction network according to claim 1, characterized in that: For the output of the depth estimation module, a single depth map scale unification strategy is implemented to unify the depth value of a single image. in represents the depth map of image i generated by image i-1 and image i; First, perform perspective transformation registration on the two images to obtain the overlapping parts between the two images. Then find the intersection of the overlapping parts of the i-th image and its previous and next images Then perform linear regression on the corresponding pixel depth values ​​in the front and back depth maps corresponding to the intersection part During regression, the RANSAC algorithm is used to remove noise points and abnormal depth values; a and b represent the slope and intercept of the regression parameters respectively; then the depth value of the overlapping part with the next image is linearly transformed. in, Represents the estimated values ​​of a and b during regression; finally, the depth values ​​of the overlapping parts before and after are taken as the union, that is, the depth value of the single image is unified 5. The 3D entity reconstruction method based on an end-to-end 3D entity reconstruction network according to claim 1, characterized in that: The instance segmentation module is composed of a SAMI pre-trained lightweight encoder as an image encoder and a SAM default mask decoder as a decoder; The pre-training of the SAMI encoder uses the image features extracted by the SAM encoder as a guide. First, a mask image is randomly generated from the image, and then the mask image generates an unmasked embedding layer through a lightweight encoder. Then, through cross-attention decoding, the mask unit features are used as queries, and the unmasked features and mask features of the encoder are used as keys and values ​​to construct cross-attention. The output features of the mask unit from the cross-attention decoder are merged with the output features of the unmasked unit from the encoder. The combined features will be reordered to the original position of the input image; then the output features are aligned with the feature dimensions from the SAM image encoder using a linear projection head, and then the features are combined with the image features extracted by the SAM encoder to construct a reconstruction loss for training. The obtained SAMI pre-trained encoder is used as the image encoder of the instance segmentation network; the image features extracted by the SAMI pre-trained encoder are decoded with the SAM default mask decoder to generate a two-dimensional instance segmentation result, and the depth information generated by the joint depth estimation network is retrained.

6. The 3D entity reconstruction method based on an end-to-end 3D entity reconstruction network according to claim 1, characterized in that: The incremental aerial triangulation module first extracts key points from the two images using the Fostner operator, projects the key points of one image into the three-dimensional space through the depth map and the camera's intrinsic parameters and posture to generate three-dimensional points, then uses the camera's intrinsic parameters and posture of the other image to back-project into the image to generate projection points, searches for the key point closest to the projection point within a preset radius around the projection point as a matching point, and generates matching points of the two images, i.e., points with the same name; Then jointly optimize the camera intrinsic parameters and pose parameters The 3D scene point coordinates corresponding to the depth map projection of the key point Minimize the reprojection error: Among them, the ρ function projects the three-dimensional point set P into the image space, q i,j =[x i,j ,y i,j ,1] T is the normalized homogeneous pixel coordinate of the image, Contains all 3D points and camera parameters; the optimized camera intrinsic parameters, pose and 3D points are obtained through the Levenberg-Marquardt algorithm m represents the number of 3D scene points.

7. The 3D entity reconstruction method based on an end-to-end 3D entity reconstruction network according to claim 1, characterized in that: The point cloud fusion module first determines the key points of the multi-view images as points with the same name, and converts each of them from the depth map to the point cloud. is the three-dimensional same-name point; if the coordinate system is not unified, that is, p ij ≠p i+1j , then calculate the rigid registration relationship between the two, use the RANSAC algorithm to remove noise points and abnormal depth values, and then use the depth map to convert all remaining non-key points into three-dimensional points and perform the above-calculated rigid relationship transformation, convert them into a unified three-dimensional space coordinate system, and generate a fused three-dimensional point cloud; The calculation of rigid registration is as follows: The following registration relationship is calculated: Among them, R and T are the rotation matrix and translation matrix of the rigid registration relationship, respectively, and m and k represent the number of three-dimensional points before and after the RANSAC algorithm is used.

8. The 3D entity reconstruction method based on an end-to-end 3D entity reconstruction network according to claim 1, characterized in that: The entity generation module, for any two sets of fused point clouds and First establish the mapping relationship M, that is, if the point and There is a rigid registration relationship between them, then (i,j)∈M; then use the mapping relationship M to X 1 The point set Q for each 2D instance label x in 1 Query in X 2 The corresponding point set Q 2 And its corresponding instance y; then count Q separately 1 With Q 2 The points are recorded as and And count X 2 The number of points of instance y in like Then it means X 1 Instances of x and X 2 The instance y is a highly overlapping instance label, that is, when the point cloud is fused, the point set Q 1 The instance label of X is changed to instance y to merge the instances; in order to make all instances in the point cloud be traversed, 2 Each instance in the is symmetrically operated in both directions until all instances are merged to generate a 3D solid point cloud; For the local point cloud generated by any n images For example, n is the total number of points in the image; first, the adjacent point cloud and Merge into 3D solid point cloud Then and Merge into a new 3D solid point cloud Repeat n-2 times until all point clouds are merged into a 3D solid point cloud of the entire scene. The above-mentioned bidirectional merging strategy is used to merge the two sets of point clouds, and ⊕ is represented as a bidirectional merging operation, and the bottom-up merging rule is expressed as: Among them, the point cloud that has been fused in the i-th fusion And the point cloud that has not been fused A new fused point cloud is merged Until the 2D instance information corresponding to all point clouds is merged to generate a 3D solid point cloud DSM n .

9. The 3D entity reconstruction method based on an end-to-end 3D entity reconstruction network according to any one of claims 1 to 8, characterized in that: The 3D entity point cloud DSM n , further through the surface optimization module and the texture mapping module, the surface optimized three-dimensional solid model M0 is obtained, and the final textured three-dimensional solid model M1 is generated; The surface optimization module first performs point-by-point feature encoding, extracts point-by-point local features V0 from a three-dimensional solid point cloud with a number of n according to n×m feature vectors and n×3 xyz coordinate information through a point cloud network based on a Transformer architecture; the point cloud network of the Transformer architecture, based on the Point Transformer network, uses cross attention to encode the position feature code P after position encoding of the point cloud. i Embedded into point cloud features as queries; Three consecutive point cloud self-attention modules are used to extract point-by-point local features of the point cloud: P i =MLP(p i -p j )=ReLU(Linear(Linear(p i -p j ))); X k+1 =Linear(Attention self (Linear(X k )))+X k ; Among them, the position-encoded multilayer perceptron MLP consists of two linear layers and a ReLU activation layer. i With p j is the three-dimensional coordinates of point i and point j, point j is the local area χ of point i i Point inside; F i is the point cloud feature, Q=P i W q , K=V=F i W k , Q is the query generated by the position encoding, K and V represent the key and value generated by the point cloud features; Y i represents the output feature of point cloud self-attention, X i With X j Represents the features of input point i and point j, ψ and α represent different linear layers, γ represents that the multilayer perceptron consists of two linear layers and one ReLU activation layer, and the Softmax function is used for standardization, and ⊙ represents the multiplication of corresponding elements of the matrix; X k represents the input features of the k-th point cloud self-attention module, k = 1, 2, 3, X k+1 Represents the features of the k-th output and the k+1-th input, and performs residual connection on each module. After passing through three point cloud self-attention modules, the final feature X4 is output as the extracted point-by-point local feature V0; then all point features belonging to the same voxel unit are integrated through the average pooling layer to obtain the shallow local volume feature H×W×D×C0 of the point cloud, where H, W, D, and C0 represent the height, width, depth, and number of channels of the volume feature, respectively; Then, the volume feature encoding is performed, and a three-dimensional U-shaped network with a hybrid architecture of CNN and Transformer is used to extract the local and global volume features V of the point cloud from the shallow local volume features of the point cloud. In the encoding part, the first and second layers are [3×3×3,1] convolutional layers, retaining the original resolution features H×W×D×C1; the third layer is a [3×3×3,2] convolutional layer, which is averagely pooled with the original volume features and then connected to complete downsampling. The fourth, fifth, and sixth layers are 3D dilated convolutional layers with expansion rates r=1,2,3, and the seventh layer is 3D deformable attention. Deform : The input volume feature X is The query matrix Q is obtained by passing the volume features through a linear layer, x Q represents the three-dimensional coordinates of the query matrix Q, M represents the number of attention heads, K represents the fixed number of key samples, and W m ′、W m Represents the value matrix before and after 3D deformable attention, A mqk , Δx mpk represents the attention weight and sampling bias of the qth query and the kth sampling point of the mth attention head; Then repeat the two stages for the third to seventh layers to obtain multi-scale volume features In the decoding stage, after three repeated stages of [3×3×3,1] convolutional layer, upsampling layer, skip connection, and [3×3×3,1] convolutional layer, the local and global volume features V of the point cloud are obtained; Then, the occupancy field is decoded, and the volume feature V is interpolated by trilinear interpolation to determine the feature F corresponding to the position of any point p in the three-dimensional space. p , then p and F p Input to a lightweight multi-layer perceptron MLP, predicting the probability that p is classified as an occupied field within a 3D surface as P(p)∈R 3 : Finally, multi-resolution isosurface extraction and marching cubes are implemented to extract the surface mesh and generate a three-dimensional surface, i.e., a three-dimensional solid model M0; The texture mapping module combines the camera's internal parameters and posture, adopts a texture mapping algorithm, selects images with smaller deformation from multiple views to first perform texture mapping on the three-dimensional surface, and then selects images from other perspectives to repeatedly paste textures to deal with occlusion problems, merges these textures together, completes seamless texture mapping, and generates the final textured three-dimensional solid model M1.

10. A 3D entity reconstruction system based on an end-to-end 3D entity reconstruction network, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the three-dimensional solid reconstruction method based on an end-to-end three-dimensional solid reconstruction network as described in any one of claims 1 to 9.

Images