Scene completion methods, devices, electronic devices and storage media

Abstract

This application discloses a scene completion method, apparatus, electronic device, and storage medium. This application can acquire a 3D point cloud and multiple 2D images of a scene to be modeled. The 3D point cloud consists of multiple points, and the 2D images are images of the scene to be modeled taken from any viewpoint. Semantic segmentation is performed on the 2D images to obtain a semantic image, where each pixel is labeled with a semantic tag. The semantic image is then aligned with the coordinates of each point in the 3D point cloud to obtain a semantic point cloud, where each point is associated with a semantic tag. Fusion features of the 3D point cloud, 2D images, and semantic point cloud are acquired. Based on the semantic point cloud, missing scene regions in the scene to be modeled are determined. Based on the fusion features, the missing scene regions are completed to obtain a complete scene. This application can accurately complete the missing content in a scene to be modeled.

Description

Technical Field

[0001] This application relates to the field of computers, specifically to a scene completion method, apparatus, electronic device, and storage medium. Background Technology

[0002] A point cloud is a collection of millions of unordered, scattered spatial points in a 3D data set. Each point in a point cloud typically contains its position coordinates, and may also contain attribute data such as color and normals. These data are usually derived from samples taken from a laser scanner or depth camera. The core goal of point cloud reconstruction is to recover a complete and continuous 3D object or scene surface model from these incomplete raw point cloud data, thereby achieving scene completion.

[0003] Traditional scene completion methods rely on manual calibration, which is inefficient and has poor completion accuracy, often resulting in structural misalignment and other problems in the reconstructed model. Summary of the Invention

[0004] This application provides a scene completion method, apparatus, electronic device, and storage medium, which can quickly and accurately complete the missing content in the scene to be modeled, thereby improving the efficiency of scene completion.

[0005] This application provides a scene completion method, including:

[0006] The process involves acquiring a 3D point cloud and multiple 2D images of the scene to be modeled. The 3D point cloud consists of multiple points, and the 2D images are images of the scene to be modeled taken from any viewpoint.

[0007] Semantic segmentation is performed on a two-dimensional image to obtain a semantic image, which is a two-dimensional image in which each pixel is labeled with a semantic tag.

[0008] The semantic image is aligned with the coordinates of each point in the 3D point cloud to obtain the semantic point cloud, which is a 3D point cloud associated with a semantic label for each point.

[0009] Obtain the fusion features of 3D point cloud, 2D image, and semantic point cloud;

[0010] Based on semantic point clouds, identify the missing scene regions in the scene to be modeled;

[0011] Based on the fusion features, the missing areas of the scene to be modeled are filled in to obtain a complete scene.

[0012] This application embodiment also provides a scene completion device, including:

[0013] The acquisition unit is used to acquire the 3D point cloud and multiple 2D images of the scene to be modeled. The 3D point cloud is composed of multiple points, and the 2D images are images of the scene to be modeled taken from any perspective.

[0014] Semantic units are used to perform semantic segmentation on two-dimensional images to obtain semantic images, which are two-dimensional images with semantic labels labeled for each pixel.

[0015] Alignment units are used to align the semantic image with each point in the 3D point cloud to obtain a semantic point cloud, which is a 3D point cloud with semantic labels associated with each point.

[0016] The fusion unit is used to acquire the fusion features of 3D point cloud, 2D image, and semantic point cloud;

[0017] Missing units are used to determine the missing areas in the scene to be modeled based on semantic point clouds.

[0018] The completion unit is used to complete the missing areas of the scene to be modeled based on the fusion features, so as to obtain the complete scene.

[0019] In some embodiments, obtaining the fusion features of 3D point cloud, 2D image, and semantic point cloud includes:

[0020] Extracting geometric features from 3D point clouds;

[0021] Extracting texture features from two-dimensional images;

[0022] Extract semantic spatial relationship features from semantic point clouds;

[0023] The geometric features of 3D point clouds, the texture features of 2D images, and the semantic space relationship graph are fused to obtain fused features;

[0024] In some embodiments, extracting the geometric features of a 3D point cloud includes:

[0025] Calculate the angle between the neighborhood normal vectors of each point in the 3D point cloud to obtain the curvature distribution map of the 3D point cloud;

[0026] Based on the curvature distribution map, high curvature regions and low curvature regions are determined in the 3D point cloud;

[0027] Feature extraction is performed on high curvature regions to obtain dense point features;

[0028] Feature extraction is performed on the low curvature region to obtain sparse point features;

[0029] The dense point features and sparse point features are fused to obtain the geometric feature vector.

[0030] In some embodiments, feature extraction is performed on the high curvature region to obtain dense point features, including:

[0031] Determine the FPFH descriptor for each point in the high curvature region to obtain the first feature;

[0032] The first feature is input into the feature extraction network to obtain dense point features, which represent the geometric features of the first feature at multiple scales.

[0033] In some embodiments, feature extraction is performed on the low curvature region to obtain sparse point features, including:

[0034] Determine the FPFH descriptor for each point in the low curvature region to obtain the second feature;

[0035] A three-layer MLP network is used to extract sparse point features, which represent the global contour features of the second feature.

[0036] In some embodiments, extracting semantic spatial relationship features from the semantic point cloud includes:

[0037] Construct a spatial relationship graph of semantic point cloud. The spatial relationship graph includes nodes and edges. Nodes represent objects in the scene to be modeled, and edges represent the spatial relationships between objects.

[0038] A graph convolutional network is used to extract semantic spatial relationship features from the spatial relationship graph.

[0039] In some embodiments, based on fusion features, missing areas of the scene to be modeled are filled in to obtain a complete scene, including:

[0040] Based on pre-defined building code data, determine geometric constraints;

[0041] Generate a binary semantic mask for the missing regions of the scene to be modeled;

[0042] The fused features, geometric constraints, and binary semantic masks are input into the 3D GAN generator, and the output is a completed point cloud.

[0043] Obtain the texture map corresponding to the missing area of ​​the scene in the 2D image;

[0044] By combining 3D point clouds, complete point clouds, and texture maps, a complete scene is obtained.

[0045] This application also provides an electronic device, including a memory storing multiple instructions; the processor loads instructions from the memory to execute steps in any of the scenario completion methods provided in this application.

[0046] This application also provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute steps in any of the scenario completion methods provided in this application.

[0047] This application embodiment can acquire a 3D point cloud and multiple 2D images of a scene to be modeled. The 3D point cloud is composed of multiple points, and the 2D images are images of the scene to be modeled taken from any viewpoint. Semantic segmentation is performed on the 2D images to obtain semantic images, which are 2D images with semantic labels labeled for each pixel. The semantic images are then aligned with the coordinates of each point in the 3D point cloud to obtain a semantic point cloud, which is a 3D point cloud with semantic labels associated with each point. The fusion features of the 3D point cloud, 2D images, and semantic point cloud are acquired. Based on the semantic point cloud, the missing scene regions in the scene to be modeled are determined. Based on the fusion features, the missing scene regions in the scene to be modeled are filled in to obtain a complete scene.

[0048] This application uses semantic segmentation to assign pixel-level labels to 2D images and aligns them precisely with point clouds, seamlessly transferring rich semantic information from the images to 3D point clouds to construct semantic point clouds. These semantic point clouds can then be used to distinguish different objects within the point cloud, overcoming the lack of semantic information in traditional point clouds. Next, features from the 3D point cloud, 2D image, and semantic point cloud are fused to form a complementary joint feature representation: the geometric information of the 3D point cloud ensures spatial accuracy, the texture of the 2D image supplements surface details, and the semantic labels of the semantic point cloud endow the scene with structured understanding capabilities, collectively forming a more robust foundation for scene representation. Finally, the semantic point cloud can accurately locate semantically incoherent or geometrically broken regions, such as occluded object parts. Compared to traditional methods that rely solely on geometric hole detection, the semantic consistency proposed in this scheme becomes a more reliable basis for identifying missing regions. The completion process utilizes fused features including geometry, texture, and semantics to ensure that the generated scene structure conforms to both geometric rationality and semantic logic.

[0049] Despite the challenges of complex occlusion, this application can generate structurally complete and semantically accurate 3D scene models, with semantically guided completion significantly reducing unreasonable completion content. Therefore, embodiments of this application can quickly and accurately complete missing scene content in the scene to be modeled. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a flowchart illustrating the scene completion method provided in the embodiments of this application;

[0052] Figure 2 This is a schematic diagram of the scene completion device provided in the embodiments of this application;

[0053] Figure 3 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0054] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0055] This application provides a scene completion method, apparatus, electronic device, and storage medium.

[0056] Specifically, the scene completion device can be integrated into an electronic device, such as a terminal or server. The terminal can be a mobile phone, tablet, smart Bluetooth device, laptop, or personal computer (PC); the server can be a single server or a server cluster composed of multiple servers.

[0057] In some embodiments, the scene completion device can also be integrated into multiple electronic devices. For example, the scene completion device can be integrated into multiple servers, and the scene completion method of this application can be implemented by multiple servers.

[0058] In some embodiments, the terminal may also be used as a server to perform some or all of the functions of a server.

[0059] For example, the electronic device can be a terminal that can acquire a 3D point cloud and multiple 2D images of the scene to be modeled. The 3D point cloud consists of multiple points, and the 2D images are images of the scene to be modeled taken from any viewpoint. The 2D images are semantically segmented to obtain a semantic image, which is a 2D image in which each pixel is labeled with a semantic tag. The semantic image is then aligned with the coordinates of each point in the 3D point cloud to obtain a semantic point cloud, which is a 3D point cloud in which each point is associated with a semantic tag. The fusion features of the 3D point cloud, the 2D images, and the semantic point cloud are acquired. Based on the semantic point cloud, the missing areas of the scene to be modeled are determined. Based on the fusion features, the missing areas of the scene to be modeled are filled in to obtain a complete scene.

[0060] The following sections provide detailed descriptions of each example. It should be noted that the sequence numbers of the following embodiments are not intended to limit the preferred order of the embodiments.

[0061] In this embodiment, a scene completion method is provided, referring to... Figure 1 The specific process of this scenario completion method can be summarized as follows:

[0062] 110. Obtain the 3D point cloud and multiple 2D images of the scene to be modeled. The 3D point cloud consists of multiple points, and the 2D images are images of the scene to be modeled taken from any perspective.

[0063] The scene to be modeled refers to the virtual 3D scene to be completed, which may contain one or more missing areas. A virtual 3D scene is a virtual three-dimensional spatial environment completely simulated using digital information in a computer, and it can be composed of one or more virtual objects. For example, a courtyard scene can be composed of doors, windows, walls, roofs, etc.

[0064] Virtual objects and virtual 3D scenes can both contain data such as virtual models and material maps. Material maps define the visual attributes and details of the model's surface, such as color, texture, smoothness, and reflectivity.

[0065] A 3D point cloud is a collection of discrete 3D points collected by sensors such as LiDAR. Each point can contain location information, such as coordinates.

[0066] 120. Perform semantic segmentation on a two-dimensional image to obtain a semantic image, which is a two-dimensional image in which each pixel is labeled with a semantic tag.

[0067] Semantic segmentation is a process that classifies each pixel in an image into a predefined object category, such as walls, windows, and roofs. In some embodiments, semantic segmentation can be achieved based on a neural network model, where a two-dimensional image is input into the neural network model, and the pixel-level semantic annotation results output by the neural network model are used as the semantic image.

[0068] Each pixel in a semantic image can be labeled with a semantic tag, and in some embodiments, the tag value can correspond to an object category.

[0069] 130. Align the semantic image with the coordinates of each point in the 3D point cloud to obtain the semantic point cloud, which is a 3D point cloud with semantic labels associated with each point.

[0070] In some embodiments, a transformation matrix can be established between the coordinate system of the two-dimensional image and the coordinate system of the three-dimensional point cloud. The transformation matrix is ​​used to achieve coordinate alignment, thereby projecting the semantic image into the spatial system of the three-dimensional point cloud, so that the pixels in the semantic image are aligned one-to-one with each point in the three-dimensional point cloud.

[0071] 140. Obtain the fusion features of 3D point cloud, 2D image, and semantic point cloud.

[0072] In some embodiments, obtaining the fusion features of 3D point cloud, 2D image, and semantic point cloud includes:

[0073] Extracting geometric features from 3D point clouds;

[0074] Extracting texture features from two-dimensional images;

[0075] Extract semantic spatial relationship features from semantic point clouds;

[0076] The geometric features of 3D point clouds, the texture features of 2D images, and the semantic space relationship graph are fused to obtain fused features.

[0077] Among them, geometric features are feature vectors that characterize the spatial structure of an object; texture features are feature vectors that characterize the visual attributes of an object's surface; and semantic spatial relationship features are structured feature vectors that characterize the topological relationships between different objects.

[0078] This embodiment achieves coupled geometric-texture-semantic representation by fusing geometric features, texture features, and semantic space relationship graphs to obtain cross-modal fused features.

[0079] In some embodiments, extracting the geometric features of a 3D point cloud includes:

[0080] Calculate the angle between the neighborhood normal vectors of each point in the 3D point cloud to obtain the curvature distribution map of the 3D point cloud;

[0081] Based on the curvature distribution map, high curvature regions and low curvature regions are determined in the 3D point cloud;

[0082] Feature extraction is performed on high curvature regions to obtain dense point features;

[0083] Feature extraction is performed on the low curvature region to obtain sparse point features;

[0084] The dense point features and sparse point features are fused to obtain the geometric feature vector.

[0085] Dense point features are local detailed geometric features extracted from high curvature regions. In some embodiments, dense point features can be extracted using multi-scale convolutional networks, which can better represent the microstructure of high curvature regions.

[0086] Sparse point features are global contour features extracted from low curvature regions. In some embodiments, sparse point features can be extracted using fully connected networks, which can better represent the macroscopic structure of low curvature regions.

[0087] Among them, the curvature distribution map K p The curvature distribution map K represents the degree of surface curvature at point p, which can be obtained from the angle between the neighborhood normal vectors. p The calculation formula is:

[0088]

[0089] Where, n p Let n be the normal vector of point p. pi Let be the normal vector of the i-th neighbor of point p, and k be the total number of neighboring points of point p; a neighboring point refers to a point adjacent to point p, that is, there is an edge between point p and the i-th neighbor of point p.

[0090] In some embodiments, points with K greater than a preset threshold can be assigned to high curvature regions, and points with K less than the preset threshold can be assigned to low curvature regions.

[0091] In some embodiments, voxels in high curvature regions can be set as the first size, and voxels in low curvature regions can be set as the second size, thereby ensuring a balance between performance and accuracy.

[0092] For example, in some embodiments, the preset threshold can be set to 0.3, the first dimension is 0.05 meters (m), and the second dimension is 0.3 meters, thus obtaining:

[0093]

[0094] That is, there are 8,000 points per cubic meter in the high curvature region and 125 points per cubic meter in the low curvature region.

[0095] In some embodiments, feature extraction is performed on the high curvature region to obtain dense point features, including:

[0096] Determine the FPFH descriptor for each point in the high curvature region to obtain the first feature;

[0097] The first feature is input into the feature extraction network to obtain dense point features, which represent the geometric features of the first feature at multiple scales.

[0098] In some embodiments, the high-curvature region can be downsampled before determining the FPFH descriptor for each point in the high-curvature region to reduce the data size of the high-curvature region and improve the efficiency of subsequent processing. Voxel mesh downsampling is one such downsampling method; for example, the number of points in a region can be reduced by dividing the point cloud into a voxel mesh and averaging the points.

[0099] The Fast Point Feature Histogram (FPFH) is a feature descriptor used for 3D point cloud data processing. It is mainly used to describe the local geometric features of each point in the point cloud. FPFH generates a feature vector with discriminative power by statistically analyzing the spatial geometric relationships between a point in the point cloud and its neighboring points, quantizing these relationships into a histogram. The spatial geometric relationships can include normal differences, distances, angles, etc.

[0100] The feature extraction network can employ a multi-scale grouping (MSG) module to extract features from high-curvature regions. This feature extraction network includes a sampling module, a grouping module, and a feature extraction module, wherein:

[0101] 1. Sampling module:

[0102] The sampling module can achieve uniform coverage sampling by iteratively selecting the point farthest from the existing center point using the farthest point sampling formula; the farthest point sampling formula is as follows:

[0103]

[0104] Where S is the center point of the point cloud set, which is the set of points in a region of high curvature; S k+1 The (k+1)th sampling center point is given by K, where K is the number of sampling points; ||ps|| is the Euclidean distance between point p and center point S.

[0105] 2. Grouping module:

[0106] The grouping module can aggregate neighborhood point sets across multiple radius scales, that is, it collects neighborhood points within a radius r with the center point as the sphere center, thus constructing a local geometric context; the formula is as follows:

[0107]

[0108] Where r is the radius scale, G k Let be the set of neighborhood points of the k-th sampling center point.

[0109] The radius scale r can take multiple values ​​to capture features from micro-texture to macro-contour, achieving collaborative extraction of details and global features.

[0110] 3. Feature Extraction Module:

[0111] The feature extraction module extracts local features for each group and outputs multi-scale geometric features through shared MLP and max pooling; the formula is as follows:

[0112] f k =MaxPool(MLP(Concat(p j ,FPFH(p j )))),p j ∈G k

[0113] f k r1 = MLP(G k r1 ), f k r2 = MLP(G k r2 )

[0114] f k out =W res ·f k l-1 +Conv(Concat(f k r1 f k r2 ))

[0115] F global =max(f k 3 ) ∈R 256

[0116] in, f k out : for the first k The output feature vector of each center point; f k l-1 These are the input features for the previous layer; fk ( r 1), fk ( r 2) Local features at different radius scales; W res is a learnable residual weight matrix used to adjust the influence of input features; Conv is a 1×1 convolutional layer used for multi-scale feature fusion; MaxPool is a max pooling layer; Concat is a feature fusion operation.

[0117] In some embodiments, feature extraction is performed on the low curvature region to obtain sparse point features, including:

[0118] Determine the FPFH descriptor for each point in the downsampled low curvature region to obtain the second feature;

[0119] A three-layer MLP network is used to extract sparse point features, which represent the global contour features of the second feature.

[0120] In some embodiments, the low curvature region can be downsampled before determining the FPFH descriptor for each point in the low curvature region to reduce the data size of the low curvature region and improve the efficiency of subsequent processing.

[0121] One approach is to use a three-layer MLP (Multi-Layer Perceptron) network to extract sparse point features. This MLP network consists of three fully connected layers. The input layer projects the second feature to 128 dimensions; the hidden layer learns a nonlinear transformation through the ReLU activation function; and the output layer compresses the output of the hidden layer to a 64-dimensional feature vector and introduces a residual connection to preserve the original FPFH information.

[0122] In some embodiments, extracting semantic spatial relationship features from the semantic point cloud includes:

[0123] Construct a spatial relationship graph G=(V,R) for the semantic point cloud. The spatial relationship graph includes nodes V and edges R. Nodes represent objects in the scene to be modeled, and edges represent the spatial relationships between objects.

[0124] A graph convolutional network is used to extract semantic spatial relationship features from the spatial relationship graph.

[0125] In some embodiments, edges may have weights that characterize the strength of spatial relationships. Therefore, the spatial relationship graph G may also include a weight matrix Wr(l) corresponding to the spatial relationships. Thus, the formula for extracting the semantic spatial relationship features of the spatial relationship graph using a graph convolutional network is as follows:

[0126]

[0127] Among them, H i l+1 This represents the output feature of node i at layer (l+1). σ Let r be the activation function, r be a discrete variable, and N be the variable. r (i) c is the set of neighboring nodes that have a spatial relationship r with node i; i,r is the normalization coefficient, and j is the neighboring node of node i.

[0128] In some embodiments, extracting texture features from a two-dimensional image includes:

[0129] Generate multiple key points of a two-dimensional image and their corresponding first descriptors;

[0130] Based on key points and a first descriptor, calculate the confidence of a matching pair between a 2D image and at least one other 2D image from a different viewpoint;

[0131] Filter matching pairs with a confidence level greater than a preset threshold to obtain the first matching set;

[0132] Identify low-texture regions in a 2D image and generate a second descriptor for the low-texture regions.

[0133] The nearest neighbor ratio test is performed on the first descriptor and the second descriptor, and complementary matching pairs that meet the ratio threshold are selected to obtain the second matching set;

[0134] By fusing the first matching set and the second matching set, texture features containing sparse correspondences are obtained.

[0135] Texture features refer to the pixel-level correspondence and associated descriptors established through keypoint matching, which are used to transfer texture information in cross-modal alignment; keypoints are pixel locations in an image with significant geometric characteristics, such as window corners, door frame edges, etc.; confidence score is a quantitative value of matching reliability; low-texture regions are image regions that lack significant gradients, such as solid-color walls.

[0136] In some embodiments, the confidence level M of the matching pair can be calculated based on a graph neural network with an attention mechanism. ij The formula is as follows:

[0137] M ij =Sim(d i A ,d j B )+λ·Attn ij cross

[0138] Where Sim is the cosine similarity; d i A d is the first descriptor of the i-th keypoint in image A; j B Let λ be the first descriptor of the j-th keypoint in image B; λ is the attention weight coefficient; Attn ij cross This refers to the cross-image attention weights of a graph neural network.

[0139] In some embodiments, the nearest neighbor ratio test can be performed using the following formula:

[0140]

[0141] Where, d i d is the first descriptor of the pixel to be matched. j (1) The second descriptor is the nearest neighbor pixel; d j (2) It is the second descriptor of the next nearest pixel.

[0142] In some embodiments, the geometric features of a 3D point cloud, the texture features of a 2D image, and the semantic space relation graph are fused to obtain fused features. Furthermore, cross-modal attention mechanisms and multi-head self-attention can be added. Therefore:

[0143] Embedded vectors can be processed using a three-layer Transformer decoder, with each layer performing the following:

[0144] 1. Multi-head self-attention computation: The input features are split into 8 heads, and attention weights are calculated for each head in a 64-dimensional subspace;

[0145] 2. Cross-modal attention computation: Perform attention interactions in both directions from point cloud to image and from image to point cloud, and add 3D relative position offsets to point cloud features;

[0146] 3. Feedforward network transformation: Nonlinear mapping is performed through a fully connected layer with a hidden layer dimension of 1024.

[0147] Then:

[0148] Point clouds and image features are concatenated to generate a 64-channel hybrid feature map;

[0149] The contribution of each modality is dynamically adjusted through learnable weights;

[0150] Based on validation set learning, a classification confidence weighted voting method is used to output unified fusion features.

[0151] The formula for calculating the relative position offset is:

[0152]

[0153] Where, Δp ij This refers to the relative coordinate offset of the point cloud, such as the physical distance between a window frame corner and the window texture; W Q p With W K p For the location projection weights, Q i K is the point cloud feature query vector. j Let d be the image feature key vector. k This is the scaling factor.

[0154] The weighting formula is as follows:

[0155] α=Softmax(W•[Fp||Fi||Fs]+b);

[0156] Where Fp represents the point cloud geometric features; Fi represents the image texture features; Fs represents the semantic relation features; W represents the learnable weight matrix; b represents the bias term; and α represents the modal weight.

[0157] 150. Based on semantic point cloud, determine the missing scene regions in the scene to be modeled.

[0158] For example, a semantic segmentation module used in a convolutional neural network can be used to classify the scene to be modeled, thereby identifying the missing areas in the scene to be modeled.

[0159] In some embodiments, a semantic mask for the missing area of ​​the scene can be generated, and the texture of the corresponding area can be extracted from the image so that the texture map of the missing area of ​​the scene can be considered in subsequent steps when completing the scene.

[0160] 160. Based on the fusion features, the missing areas of the scene to be modeled are filled in to obtain a complete scene.

[0161] In some embodiments, based on fusion features, missing areas of the scene to be modeled are filled in to obtain a complete scene, including:

[0162] Based on pre-defined building code data, determine geometric constraints;

[0163] Generate a binary semantic mask for the missing regions of the scene to be modeled;

[0164] The fused features, geometric constraints, and binary semantic masks are input into the 3D GAN generator, and the output is a completed point cloud.

[0165] Obtain the texture map corresponding to the missing area of ​​the scene in the 2D image;

[0166] By combining 3D point clouds, complete point clouds, and texture maps, a complete scene is obtained.

[0167] The preset building code data refers to a set of structured rules stored in an external building code database, such as the requirement that the height of a Gothic arch should be 1.5 times its span. This data is based on standards and constraints in the building field and is accessed and transmitted through an extended MCP protocol.

[0168] In some embodiments, a large language model can be used as an intelligent intermediary to parse and extract relevant information from user instructions to form a knowledge database, thereby providing these building code data as an input source.

[0169] Geometric constraints refer to specific geometric limit parameters derived by the system from preset building code data, such as component size ratios, location requirements, etc. In some embodiments, the code data can be encoded into operable constraint templates through the MCP protocol standardization interface and associated with semantic tags, ultimately being encoded into a 128-dimensional condition vector to ensure that the generated geometry strictly follows the rules of the building domain.

[0170] Binary semantic mask is a binary image representation based on semantic segmentation technology.

[0171] As described above, this embodiment of the application can acquire a 3D point cloud and multiple 2D images of the scene to be modeled. The 3D point cloud consists of multiple points, and the 2D images are images of the scene to be modeled taken from any viewpoint. Semantic segmentation is performed on the 2D images to obtain semantic images, where each pixel is labeled with a semantic tag. The semantic images are then aligned with the coordinates of each point in the 3D point cloud to obtain a semantic point cloud, where each point is associated with a semantic tag. The fusion features of the 3D point cloud, 2D images, and semantic point cloud are acquired. Based on the semantic point cloud, missing scene regions in the scene to be modeled are determined. Based on the fusion features, the missing scene regions are filled in to obtain a complete scene. Therefore, this solution can quickly and accurately fill in the missing content of the scene to be modeled.

[0172] To better implement the above methods, this application also provides a scene completion device, which can be integrated into an electronic device, such as a terminal or server. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, or personal computer; the server can be a single server or a server cluster composed of multiple servers.

[0173] For example, in this embodiment, the method of this application embodiment will be described in detail by taking the scene completion device specifically integrated into the terminal as an example.

[0174] For example, such as Figure 2 As shown, the scene completion device may include an acquisition unit 301, a semantic unit 302, an alignment unit 303, a fusion unit 304, a missing unit 305, and a completion unit 306, as follows:

[0175] (a) Acquisition unit 301. Acquisition unit 301 is used to acquire the three-dimensional point cloud and multiple two-dimensional images of the scene to be modeled. The three-dimensional point cloud is composed of multiple points, and the two-dimensional images are images of the scene to be modeled taken from any perspective.

[0176] (ii) Semantic unit 302.

[0177] Semantic unit 302 is used to perform semantic segmentation on a two-dimensional image to obtain a semantic image, which is a two-dimensional image in which each pixel is labeled with a semantic tag.

[0178] (iii) Alignment unit 303.

[0179] Alignment unit 303 is used to perform coordinate alignment processing between the semantic image and each point in the 3D point cloud to obtain a semantic point cloud, in which a semantic point cloud is a 3D point cloud with a semantic label associated with each point.

[0180] (iv) Fusion Unit 304.

[0181] The fusion unit 304 is used to acquire the fusion features of 3D point cloud, 2D image and semantic point cloud.

[0182] (v) Missing unit 305.

[0183] Missing unit 305 is used to determine the missing area of ​​the scene in the scene to be modeled based on semantic point cloud.

[0184] (vi) Complete Unit 306.

[0185] The completion unit 306 is used to complete the missing areas of the scene to be modeled based on the fusion features, so as to obtain a complete scene.

[0186] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0187] In this application embodiment, the terms "module" or "unit" refer to a computer program product or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0188] As described above, the scene completion device in this embodiment acquires a 3D point cloud and multiple 2D images of the scene to be modeled by an acquisition unit. The 3D point cloud is composed of multiple points, and the 2D images are images of the scene to be modeled taken from any perspective. The semantic unit performs semantic segmentation on the 2D images to obtain a semantic image, which is a 2D image with semantic labels labeled for each pixel. The alignment unit performs coordinate alignment processing on the semantic image and each point in the 3D point cloud to obtain a semantic point cloud, which is a 3D point cloud with semantic labels associated for each point. The fusion unit acquires the fusion features of the 3D point cloud, the 2D images, and the semantic point cloud. The missing area unit determines the missing scene area in the scene to be modeled based on the semantic point cloud. The completion unit completes the missing scene area in the scene to be modeled based on the fusion features to obtain a complete scene.

[0189] Therefore, the embodiments of this application can improve the speed and accuracy of completing the missing content in the scene to be modeled.

[0190] This application also provides an electronic device, which can be a terminal, a server, or other similar device. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers, etc.

[0191] In some embodiments, the scene completion device can also be integrated into multiple electronic devices. For example, the scene completion device can be integrated into multiple servers, and the scene completion method of this application can be implemented by multiple servers.

[0192] In this embodiment, the electronic device will be described in detail as a terminal, for example, such as... Figure 3 As shown, it illustrates a structural schematic diagram of the electronic device involved in the embodiments of this application, specifically:

[0193] The electronic device may include components such as a processor 410 with one or more processing cores, a memory 420 with one or more computer-readable storage media, a power supply 430, an input module 440, and a communication module 450. Those skilled in the art will understand that... Figure 3 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0194] The processor 410 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 420, and by calling data stored in the memory 420, it performs various functions and processes data, thereby performing overall detection of the electronic device. In some embodiments, the processor 410 may include one or more processing cores; in some embodiments, the processor 410 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 410.

[0195] The memory 420 can be used to store software programs and modules. The processor 410 executes various functional applications and data processing by running the software programs and modules stored in the memory 420. The memory 420 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 420 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 420 may also include a memory controller to provide the processor 410 with access to the memory 420.

[0196] The electronic device also includes a power supply 430 that supplies power to the various components. In some embodiments, the power supply 430 can be logically connected to the processor 410 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 430 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0197] The electronic device may also include an input module 440, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0198] The electronic device may also include a communication module 450. In some embodiments, the communication module 450 may include a wireless module, through which the electronic device can perform short-range wireless transmission, thereby providing users with wireless broadband internet access. For example, the communication module 450 can be used to help users send and receive emails, browse web pages, and access streaming media.

[0199] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 410 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 420 according to the following instructions, and the processor 410 runs the applications stored in the memory 420 to realize various functions, as follows:

[0200] The process involves acquiring a 3D point cloud and multiple 2D images of the scene to be modeled. The 3D point cloud consists of multiple points, and the 2D images are images of the scene to be modeled taken from any viewpoint.

[0201] Semantic segmentation is performed on a two-dimensional image to obtain a semantic image, which is a two-dimensional image in which each pixel is labeled with a semantic tag.

[0202] The semantic image is aligned with the coordinates of each point in the 3D point cloud to obtain the semantic point cloud, which is a 3D point cloud associated with a semantic label for each point.

[0203] Obtain the fusion features of 3D point cloud, 2D image, and semantic point cloud;

[0204] Based on semantic point clouds, identify the missing scene regions in the scene to be modeled;

[0205] Based on the fusion features, the missing areas of the scene to be modeled are filled in to obtain a complete scene.

[0206] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0207] As can be seen from the above, the embodiments of this application can quickly and accurately complete the missing content in the scene to be modeled.

[0208] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0209] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the scene completion methods provided in embodiments of this application. For example, the instructions can execute the following steps:

[0210] The process involves acquiring a 3D point cloud and multiple 2D images of the scene to be modeled. The 3D point cloud consists of multiple points, and the 2D images are images of the scene to be modeled taken from any viewpoint.

[0211] Semantic segmentation is performed on a two-dimensional image to obtain a semantic image, which is a two-dimensional image in which each pixel is labeled with a semantic tag.

[0212] The semantic image is aligned with the coordinates of each point in the 3D point cloud to obtain the semantic point cloud, which is a 3D point cloud associated with a semantic label for each point.

[0213] Obtain the fusion features of 3D point cloud, 2D image, and semantic point cloud;

[0214] Based on semantic point clouds, identify the missing scene regions in the scene to be modeled;

[0215] Based on the fusion features, the missing areas of the scene to be modeled are filled in to obtain a complete scene.

[0216] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0217] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the methods provided in various optional implementations of the scene reconstruction or scene completion aspects provided in the above embodiments.

[0218] Since the instructions stored in the storage medium can execute the steps in any of the scene completion methods provided in the embodiments of this application, the beneficial effects that any of the scene completion methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0219] The above provides a detailed description of a scene completion method, apparatus, electronic device, and computer-readable storage medium provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A scene completion method, characterized in that, include: Acquire a 3D point cloud and multiple 2D images of the scene to be modeled. The 3D point cloud is composed of multiple points, and the 2D images are images of the scene to be modeled taken from any viewpoint. The two-dimensional image is semantically segmented to obtain a semantic image, wherein each pixel of the two-dimensional image is labeled with a semantic tag. The semantic image is aligned with the coordinates of each point in the three-dimensional point cloud to obtain a semantic point cloud, wherein the semantic point cloud is associated with a semantic label for each point in the three-dimensional point cloud. Obtain the fusion features of the 3D point cloud, the 2D image, and the semantic point cloud; Based on the semantic point cloud, determine the scene missing regions in the scene to be modeled; Based on the fusion features, the missing areas in the scene to be modeled are filled in to obtain a complete scene; The process of acquiring the fusion features of the 3D point cloud, the 2D image, and the semantic point cloud includes: Extract the geometric features of the three-dimensional point cloud; Extract the texture features of the two-dimensional image; Extract the semantic spatial relationship features of the semantic point cloud; The geometric features of the 3D point cloud, the texture features of the 2D image, and the semantic spatial relationship features are fused to obtain the fused features; The extraction of geometric features from the 3D point cloud includes: Calculate the angle between the neighborhood normal vectors of each point in the three-dimensional point cloud to obtain the curvature distribution map of the three-dimensional point cloud; Based on the curvature distribution map, high curvature regions and low curvature regions are determined in the three-dimensional point cloud; Feature extraction is performed on the high curvature region to obtain dense point features; Feature extraction is performed on the low curvature region to obtain sparse point features; The dense point features and the sparse point features are fused to obtain a geometric feature vector. Based on the fusion features, the missing areas in the scene to be modeled are filled in to obtain a complete scene, including: Based on pre-defined building code data, determine geometric constraints; Generate a binary semantic mask for the missing scene region of the scene to be modeled; The fused features, geometric constraints, and binary semantic mask are input into the 3D GAN generator, and the output is a completed point cloud. Obtain the texture map corresponding to the missing area of ​​the scene in the two-dimensional image; The complete scene is obtained by combining the 3D point cloud, the completed point cloud, and the texture map.

2. The scene completion method as described in claim 1, characterized in that, The step of extracting features from the high curvature region to obtain dense point features includes: Determine the FPFH descriptor for each point in the high curvature region to obtain the first feature; The first feature is input into a feature extraction network to obtain dense point features, which represent the geometric features of the first feature at multiple scales.

3. The scene completion method as described in claim 1, characterized in that, The step of extracting features from the low curvature region to obtain sparse point features includes: Determine the FPFH descriptor for each point in the low curvature region to obtain the second feature; A three-layer MLP network is used to extract sparse point features, which represent the global contour features of the second feature.

4. The scene completion method as described in claim 1, characterized in that, The extraction of semantic spatial relationship features from the semantic point cloud includes: Construct a spatial relationship graph of the semantic point cloud, the spatial relationship graph including nodes and edges, the nodes representing objects in the scene to be modeled, the edges representing the spatial relationship between the objects; A graph convolutional network is used to extract the semantic spatial relationship features of the spatial relationship graph.

5. A scene completion device, characterized in that, include: The acquisition unit is used to acquire a 3D point cloud and multiple 2D images of the scene to be modeled. The 3D point cloud is composed of multiple points, and the 2D images are images of the scene to be modeled taken from any perspective. A semantic unit is used to perform semantic segmentation on the two-dimensional image to obtain a semantic image, wherein the semantic image is a two-dimensional image in which each pixel is labeled with a semantic tag. An alignment unit is used to perform coordinate alignment processing between the semantic image and each point in the three-dimensional point cloud to obtain a semantic point cloud, wherein the semantic point cloud is the three-dimensional point cloud with semantic labels associated with each point. A fusion unit is used to acquire the fusion features of the 3D point cloud, the 2D image, and the semantic point cloud; Missing units are used to determine the missing scene regions in the scene to be modeled based on the semantic point cloud; The completion unit is used to complete the missing scene regions in the scene to be modeled based on the fusion features, so as to obtain a complete scene; The process of acquiring the fusion features of the 3D point cloud, the 2D image, and the semantic point cloud includes: Extract the geometric features of the three-dimensional point cloud; Extract the texture features of the two-dimensional image; Extract the semantic spatial relationship features of the semantic point cloud; The geometric features of the 3D point cloud, the texture features of the 2D image, and the semantic spatial relationship features are fused to obtain the fused features; The extraction of geometric features from the 3D point cloud includes: Calculate the angle between the neighborhood normal vectors of each point in the three-dimensional point cloud to obtain the curvature distribution map of the three-dimensional point cloud; Based on the curvature distribution map, high curvature regions and low curvature regions are determined in the three-dimensional point cloud; Feature extraction is performed on the high curvature region to obtain dense point features; Feature extraction is performed on the low curvature region to obtain sparse point features; The dense point features and the sparse point features are fused to obtain a geometric feature vector. Based on the fusion features, the missing areas in the scene to be modeled are filled in to obtain a complete scene, including: Based on pre-defined building code data, determine geometric constraints; Generate a binary semantic mask for the missing scene region of the scene to be modeled; The fused features, geometric constraints, and binary semantic mask are input into the 3D GAN generator, and the output is a completed point cloud. Obtain the texture map corresponding to the missing area of ​​the scene in the two-dimensional image; The complete scene is obtained by combining the 3D point cloud, the completed point cloud, and the texture map.

6. An electronic device, characterized in that, The method includes a processor and a memory, the memory storing multiple instructions; the processor loads instructions from the memory to perform the steps in the scene completion method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the steps of the scene completion method according to any one of claims 1 to 4.

Images