3D modeling methods, devices, and storage media based on neural implicit surface reconstruction

Abstract

This application provides a 3D modeling method, device, and storage medium based on neural implicit surface reconstruction. The method includes: semantic segmentation of aerial images to determine pixel categories and generate category label masks; aerial triangulation of the aerial images to obtain the poses of each image; constructing an explicit mesh for the target region based on the aerial images and their poses; inputting the category label mask and the 3D coordinates of the explicit mesh into an improved neural implicit surface reconstruction model to obtain the SDF value, color, and semantic vector of the target region; performing volume rendering based on the SDF value and color of each sampled point in the target region to obtain a rendered image; extracting the scene surface and texture map of the target region from the rendered image based on the semantic vector; and performing surface reconstruction of the target region based on the scene surface and texture map. This solves the technical problem that the reconstruction accuracy of existing 3D reconstruction methods based on aerial images is highly dependent on the quality of the aerial images.

Description

Technical Field

[0001] This application belongs to the field of 3D reconstruction technology, and relates to a 3D modeling method, device and storage medium based on neural implicit surface reconstruction. Background Technology

[0002] With the rapid development of drone technology, drones have become an important auxiliary tool in many fields, such as topographic mapping, disaster assessment, crop sowing and spraying, emergency rescue, smart logistics, and image shooting.

[0003] In particular, in the field of 3D reconstruction technology, traditional 3D reconstruction methods include manual mapping reconstruction, laser scanning reconstruction, and aerial image reconstruction. When using drones to acquire aerial imagery, precise flight path planning is required to ensure that the drone can cover the entire area to be reconstructed. Therefore, factors such as terrain features, flight altitude, shooting angle, and overlap need to be considered during flight path planning. Furthermore, during drone aerial photography, the appearance of the scene may change due to long shooting intervals and lighting variations, resulting in inconsistent appearances. All of these factors can affect the accuracy of 3D modeling. Summary of the Invention

[0004] This application provides a 3D modeling method, device, and storage medium based on neural implicit surface reconstruction, which solves the technical problem that the reconstruction accuracy of existing 3D reconstruction methods based on aerial images is highly dependent on the quality of the aerial images.

[0005] The following technical solution is adopted in this application:

[0006] In a first aspect, embodiments of this application provide a 3D modeling method based on neural implicit surface reconstruction, comprising: performing semantic segmentation on aerial images of a target region to determine the pixel categories therein, and generating a category label mask for the aerial images; performing aerial triangulation on the aerial images of the target region to obtain the pose of each aerial image; constructing an explicit mesh for the target region based on the aerial images and their poses; inputting the category label mask and the corresponding 3D coordinate points in the explicit mesh into an improved neural implicit surface reconstruction model to obtain the SDF value, color, and semantic vector of the target region; wherein the backbone network of the improved neural implicit surface reconstruction model is a multilayer perceptron, and the branch networks include an SDF branch, a color branch, and a semantic branch; performing volume rendering based on the SDF value and color of each sampling point in the target region to obtain a rendered image; extracting the scene surface and texture map of the target region from the rendered image based on the semantic vector; and performing surface reconstruction of the target region based on the scene surface and texture map.

[0007] In conjunction with the first aspect, in one possible implementation, the construction of an explicit mesh of the target region based on aerial images and their poses includes: cleaning and registering the aerial images; generating corresponding point cloud data based on the poses of each pixel in the aerial images; and performing Poisson reconstruction on the point cloud data to construct an explicit mesh of the target region.

[0008] In conjunction with the first aspect, in one possible implementation, the step of inputting the category label mask and its corresponding 3D coordinate points in the explicit grid into an improved neural implicit surface reconstruction model to obtain the SDF value, color, and semantic vector of the target region includes: projecting the 3D coordinates of each sampling point in the explicit grid onto a three-plane grid to determine the planar features of the 3D coordinates of the sampling points on the three planes; using the planar features and the category label mask as input to the improved neural implicit surface reconstruction model to obtain the SDF value and semantic vector of each sampling point; and determining the color of each sampling point using the improved neural implicit surface reconstruction model based on the semantic vector, the latent encoding of the aerial image, and the normal vector of the sampling point in the explicit grid.

[0009] In conjunction with the first aspect, in one possible implementation, the step of using the planar features and the category label mask as inputs to the improved neural implicit surface reconstruction model to obtain the SDF value and semantic vector of each sampling point includes: fusing the planar features and positional information of the sampling points to obtain a first fused feature; and inputting the first fused feature into the improved neural implicit surface reconstruction model to obtain the SDF value and semantic vector of the sampling points.

[0010] In conjunction with the first aspect, in one possible implementation, after using the planar features and the category label mask as inputs to the improved neural implicit surface reconstruction model to obtain the SDF value and semantic vector of each sampling point, the method further includes: performing bilinear interpolation on the projected coordinates of each plane to obtain planar features; concatenating the planar features of the sampling points with their position information to obtain the concatenated features of the corresponding sampling points; defining a three-dimensional window centered on the sampling point and determining the surrounding neighboring points based on the three-dimensional window; performing bilinear interpolation and concatenating the position information of the neighboring points to obtain the concatenated features of the neighboring points; performing a linear transformation on the concatenated features of the sampling points to obtain their Q values; determining the K and V values ​​of the sampling points based on the concatenated features of the neighboring points; determining the similarity based on the Q and K values, and obtaining the attention weights after normalization; using the attention weights to perform a weighted summation of the V values ​​to obtain self-attention features; updating the concatenated features of the sampling points through the self-attention features, and inputting the updated concatenated features of the sampling points into the improved neural implicit surface reconstruction model to obtain the updated SDF value and semantic vector.

[0011] In conjunction with the first aspect, in one possible implementation, the step of determining the color of each sampling point using an improved neural implicit surface reconstruction model based on the semantic vector, the latent encoding of the aerial image, and the normal vector of the sampling point in the explicit grid includes: mapping the aerial image to the latent space through an encoder to obtain its latent encoding; enhancing the expressive power of the sampling point's position information and normal vector using position encoding to obtain position encoding and normal vector encoding; fusing the semantic vector, the latent encoding of the aerial image, the position encoding, and the normal vector encoding to obtain a second fused feature; and inputting the second fused feature into the improved neural implicit surface reconstruction model to obtain the color of the sampling point.

[0012] In conjunction with the first aspect, in one possible implementation, the step of performing volume rendering based on the SDF values ​​and colors of each sampling point in the target region to obtain a rendered image includes: determining the density value of the sampling points based on their SDF values, as follows:

[0013] ;

[0014] In the formula, This represents the density value of the sampling points. Let represent the derivative of the sigmoid function with a learnable scaling factor s. Indicates the direction of the ray. This represents the SDF value at the sampling point. This represents the derivative of the SDF value with respect to the location information of the sampling point. This indicates the location information of the sampling point;

[0015] The pixel color at the sampling point is determined based on the density value and color of the sampling point, as follows:

[0016] ;

[0017] In the formula, The pixel color used in volume rendering calculations. This represents the opacity of the i-th sampling point. , Indicates the interval between adjacent sampling points. This represents the density value at the sampling interval of the i-th sampling point. This represents the cumulative transparency of the i-th sampling point. , The color of the space at the i-th sampling point is represented by m, and the number of sampling points is represented by m. The rendered image is output based on the pixel color at each sampling point in the target region.

[0018] In conjunction with the first aspect, in one possible implementation, the step of extracting the scene surface and texture map of the target region from the rendered image based on semantic vectors includes: extracting multiple triangular facets of the target scene from the rendered image to obtain the scene surface; performing UV unwrapping on the scene surface and tiling the multiple triangular facets therein to obtain a tiled image; projecting the vertices of the triangular facets onto the pixel space of the corresponding aerial image and obtaining the color of the corresponding region through bilinear interpolation; filling the triangular facets corresponding to each pixel in the tiled image according to the color of the corresponding region; clustering based on the semantic vectors of the pixels in each triangular facet to obtain the semantic labels of the corresponding triangular facets; analyzing the semantic correlation between adjacent triangular facets based on the semantic labels of each triangular facet to determine the type of seam; wherein, the type of seam includes same-class seams and cross-class seams; determining a stitching method based on the type of seam to perform color smoothing on the seams between each triangular facet to obtain a texture map.

[0019] In a second aspect, this application also provides a computer device, the device comprising: a processor; a memory for storing processor-executable instructions; wherein, when the processor executes the executable instructions, it implements the method as described in the first aspect or any possible implementation of the first aspect.

[0020] Thirdly, embodiments of this application provide a non-volatile computer-readable storage medium, the non-volatile computer-readable storage medium including storage for storing a computer program or instructions that, when executed, cause the method described in the first aspect or any possible implementation of the first aspect to be implemented.

[0021] The beneficial effects of this application are:

[0022] This application proposes a 3D modeling method, device, and storage medium based on neural implicit surface reconstruction. By determining the pose of the target region through aerial triangulation of aerial images, the global consistency of the aerial images can be guaranteed, overcoming the effects of motion blur, lighting, or occlusion, and improving the scene adaptability of the aerial images. By establishing an explicit mesh, the global structure of the target region can be quickly constructed, preventing the subsequent neural implicit surface reconstruction model from getting trapped in local optima. Rendering the image overcomes the bias of a single viewpoint. Based on semantic vectors, the scene surface and texture map of the target region are extracted from the rendered image. The global consistency of the rendered image ensures the continuity and realism of the texture, making similar regions more coherent and the boundaries of different regions clearer. The final texture map conforms to both visual consistency and faithfulness to the semantic attributes of the scene. This effectively solves the technical problem that the reconstruction accuracy of existing 3D reconstruction methods based on aerial images is highly dependent on the quality of the aerial images. Thus, it achieves both the preservation of the global stability of the model reconstruction and the realization of high-precision, high-fidelity scene reconstruction. Attached Figure Description

[0023] Figure 1 A flowchart illustrating a three-dimensional modeling method based on neural implicit surface reconstruction provided in this application;

[0024] Figure 2 A schematic diagram of a network model employing an improved self-attention convolution mechanism is provided for this application. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0026] The following description of some technologies involved in the embodiments of this application is provided to aid understanding and should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, some descriptions of well-known functions and structures are omitted in the following description.

[0027] Figure 1 This is a flowchart of a three-dimensional modeling method based on neural implicit surface reconstruction provided in an embodiment of this application, including steps 101 to 107. Wherein, Figure 1 This is merely one execution order shown in the embodiments of this application and does not represent the only execution order for a 3D modeling method based on neural implicit surface reconstruction. Where the final result can be achieved, Figure 1 The steps shown can be performed in parallel or in reverse order.

[0028] Step 101: Perform semantic segmentation on the aerial image of the target area to determine the pixel category and generate a category label mask for the aerial image.

[0029] Specifically, semantic segmentation models from deep learning, such as U-Net and PSPNet, can be used here to classify aerial images pixel by pixel, determining the category of each pixel, such as buildings, roads, vegetation, etc., and generating corresponding category label masks. This step provides semantic information for subsequent 3D reconstruction, helping to more accurately reconstruct different objects in the scene.

[0030] Furthermore, in this embodiment, all acquired aerial images are encrypted before storage. For example, an encryption algorithm (such as AES-256) is used to prevent unauthorized third parties from reading or tampering with the aerial images. The encryption process is completed in real time on the drone's hardware, ensuring that it does not affect the normal execution of the flight mission. When the drone returns to its base or a safe area, its memory device is connected to a local server. The encrypted aerial images are decrypted using the same key as the encryption process on the local server, thus obtaining the captured aerial images.

[0031] Step 102: Perform aerial triangulation on the aerial images of the target area to obtain the pose of each aerial image.

[0032] Specifically, this application employs median filtering or Gaussian filtering to remove noise from aerial images. Feature points in the aerial images are matched, and then affine or projective transformations are applied to align multiple aerial images to the same coordinate system. Combining the poses of each aerial image with the matching points between them, a 3D point cloud (i.e., point cloud data) is generated through aerial triangulation. This application performs aerial triangulation on the aerial images to obtain the poses of each image. This method does not rely on the camera's precise initial intrinsic and extrinsic parameters; it automatically calculates the camera pose (position and attitude information) through matching points (such as feature points) between aerial images. It can handle large-scale, unstructured aerial images (such as irregular path images captured by UAVs), reducing reliance on external positioning equipment and lowering hardware costs.

[0033] Furthermore, this application can also ensure that the pose of all aerial images remains consistent in the global coordinate system through optimization methods such as bundle adjustment, avoiding the accumulation of pose errors in single aerial images and providing a reliable observation viewpoint benchmark for subsequent 3D reconstruction. Even if the aerial images have motion blur, occlusion, or illumination changes, as long as there are sufficient matching features, the pose can still be output stably, demonstrating strong robustness.

[0034] Step 103: Construct an explicit mesh for the target region based on aerial imagery and its pose. In this embodiment, the aerial imagery is cleaned and registered. Point cloud data is generated based on the pose of each pixel in the aerial imagery. Poisson reconstruction is performed on the point cloud data to construct an explicit mesh for the target region.

[0035] Specifically, the point cloud data is filtered to remove outliers. The point cloud data is divided into a voxel grid, with only one representative point retained within each voxel, thus reducing the number of points. The octree depth is adjusted according to the point cloud data density, and the point cloud data is stored in an octree structure. Poisson reconstruction relies on the normal vectors of the point cloud data (pointed from the camera's spatial physical coordinates to the corresponding pixels in the aerial image). A local plane is fitted using the K-nearest neighbor algorithm (e.g., taking 10-30 neighboring points from each point cloud), and the normal vectors are calculated. Global optimization is then used to unify the normal vector direction to avoid internal and external confusion. The normal vectors are substituted into the Poisson equation, and the indicator function is obtained through iterative solving using the Laplacian matrix. A threshold is selected to extract isosurfaces; here, the threshold is set to zero for the SDF value (representing the shortest distance from a point in space to the boundary of a certain geometric shape). Then, a moving cube algorithm is used to generate a triangular mesh model, i.e., the explicit mesh of the target region.

[0036] Before performing step 103, the improved neural implicit surface reconstruction model is pre-trained using real scene images. Specifically, the class label mask and rendered image of the real scene image are obtained, and the improved neural implicit surface reconstruction model is trained under supervision. After supervised training, the SDF value, color, and semantic vector of each sampling point are extracted from the improved neural implicit surface reconstruction model. In the reconstructed region, an SDF value of 0 indicates that the sampling point is on the surface of an object, an SDF value < 0 indicates that the sampling point is inside the object, and an SDF value > 0 indicates that the sampling point is outside the object.

[0037] Step 104: Input the category label mask and its corresponding 3D coordinate points in the explicit mesh into the improved neural implicit surface reconstruction model to obtain the SDF value, color, and semantic vector of the target region. The backbone network of the improved neural implicit surface reconstruction model is a multilayer perceptron, and the branch networks include SDF branches, color branches, and semantic branches.

[0038] In this embodiment, the 3D coordinates of each sampling point in the explicit mesh are projected onto a three-plane mesh to determine the planar features of the sampling points' 3D coordinates on the three planes. The planar features and the category label mask are used as inputs to an improved neural implicit surface reconstruction model to obtain the SDF value and semantic vector of each sampling point. Based on the semantic vector, the latent encoding of the aerial image, and the normal vector of the sampling point in the explicit mesh, the color of each sampling point is determined using the improved neural implicit surface reconstruction model.

[0039] Specifically, the three-dimensional coordinates of each sampling point in the explicit grid are projected onto the three planes XY, YZ, and XZ respectively (i.e., a three-plane grid), and the two-dimensional coordinates of each sampling point on the three planes are obtained. The regional features formed by these two-dimensional coordinates are the planar features.

[0040] Furthermore, the three-plane mesh consists of three mutually orthogonal planes. , Represents a three-plane mesh. Represents the XY plane, Represents the YZ plane. Let XZ plane be represented. Centered on the origin of the three-plane grid, each sampling point is mapped onto one of the three planes, with a resolution of [resolution missing]. The feature dimension is n, and the number of learnable features for each plane is set to n. .

[0041] Furthermore, the planar features and category label masks are used as inputs to the improved neural implicit surface reconstruction model to obtain the SDF value and semantic vector of each sampling point. This includes fusing the planar features and positional information of the sampling points to obtain a first fused feature. The first fused feature is then input into the improved neural implicit surface reconstruction model to obtain the SDF value and semantic vector of the sampling points.

[0042] Specifically, the obtained planar features are input into a pre-trained improved neural implicit surface reconstruction model. Through complex calculations and processing within the model, the SDF value and semantic vector corresponding to each sampling point are output. It should be noted that the SDF value of each sampling point describes the distance from that sampling point to the scene surface.

[0043] Furthermore, after using planar features and category label masks as inputs to the improved neural implicit surface reconstruction model to obtain the SDF value and semantic vector of each sampling point, the process includes: performing bilinear interpolation on the projected coordinates of each plane to obtain planar features; concatenating the planar features and positional information of the sampling points to obtain the concatenated features of the corresponding sampling points; defining a three-dimensional window centered on the sampling point and determining the surrounding neighboring points based on the three-dimensional window; performing bilinear interpolation and concatenating the positional information of the neighboring points to obtain the concatenated features of the neighboring points; performing a linear transformation on the concatenated features of the sampling points to obtain their Q values; determining the K and V values ​​of the sampling points based on the concatenated features of the neighboring points; determining the similarity based on the Q and K values, and obtaining attention weights after normalization; using the attention weights to perform a weighted summation of the V values ​​to obtain self-attention features; updating the concatenated features of the sampling points using the self-attention features, and inputting the updated concatenated features of the sampling points into the improved neural implicit surface reconstruction model to obtain updated SDF values ​​and semantic vectors.

[0044] Specifically, such as Figure 2As shown in the figure, sim represents similarity calculation, softmax represents the activation function used for normalization, multiply represents multiplication, sum represents summation, d represents the distance between sampling points (including interpolation points), Linear represents linear interpolation, query represents query, key represents key, and value represents value. Attention weights are used to reduce resource consumption. Features are queried from a three-plane grid using bilinear interpolation. During backpropagation, only nearby planar features can be updated, which cannot fully train the three-plane grid. Furthermore, learning surface locations generally involves changes in the SDF values ​​of all sampling points within a region; therefore, updating the SDF value of any sampling point requires changes in other spatial features. To achieve this, this application calculates the Key (K value), Query (Q value), and Value (V value) of the spliced ​​features of the interpolated sampling points using self-attention, and calculates the Key (K value) and Value (V value) using features interpolated from other neighboring points within the 3D window. This operation focuses all attention on the sampling point to be queried, reducing resource consumption. The distance between different sampling points (including interpolation points) within the 3D window (i.e., the distance between the sampling points) is calculated using bilinear interpolation. Figure 2 d) in the equation should be consistent with the width or height of the three-plane grid.

[0045] Specifically, the self-attention mechanism (QKV calculation) is used to capture the association between features by the similarity between the "query" (Q value) and the "key" (K value), and then the "value" (V value) is output by weighting the association weight.

[0046] The initial Q-value of a sampling point is used as the initial semantic vector at that point. The initial semantic vectors of other neighboring points within the 3D window are used as the K and V values. These K and V values ​​introduce neighborhood features, making the semantic vector of the sampling point more correlated with changes in its surrounding points, thus satisfying the spatial correlation of the SDF value. Similarity is calculated using the Q-value and each K-value of the sampling point, and then the similarity is normalized using Softmax (an activation function) to obtain attention weights. The V values ​​are then weighted and summed using these attention weights to obtain self-attention features. The concatenated features of the sampling points are updated using these self-attention features, and the updated concatenated features are input into the improved neural implicit surface reconstruction model to obtain the updated SDF value and semantic vector.

[0047] For example, the improved neural implicit surface reconstruction model includes SDF, color, and semantic branches. The SDF branch is responsible for predicting the distance from the sampling point to the scene surface, i.e., the SDF value, which is crucial for determining the geometry of objects. The SDF branch performs a non-linear transformation on the input features using a multilayer perceptron, progressively extracting high-level features and ultimately outputting the SDF value for each sampling point. These SDF values ​​are not only used to construct the object's surface but also serve as the basis for subsequent color and semantic predictions. The color branch predicts the color of the sampling point based on its location information, semantic vector, and the latent encoding of the aerial imagery. The color branch also uses a multilayer perceptron structure, but its design places greater emphasis on capturing and reconstructing color information. By combining semantic information and the latent encoding of the aerial imagery, the color branch can generate more realistic and detailed color maps, making the reconstructed 3D scene visually more lifelike. The semantic branch performs deep analysis of the input features using a multilayer perceptron, extracting features related to semantic categories and ultimately outputting the semantic vector for each sampling point. The semantic vector helps in the subsequent understanding and analysis of the reconstructed scene.

[0048] The improved neural implicit surface reconstruction model consists of multiple fully connected layers (e.g., 5-8 layers). Each layer introduces non-linearity using an activation function, progressively compressing / transforming the concatenated input features to obtain intermediate features (e.g., compressing from 3D+E dimensions to 256 or 512 dimensions). In the SDF branch, the intermediate features output the SDF values ​​of the sampling points through 1-2 fully connected layers. In the semantic branch, the intermediate features output semantic vectors through independent fully connected layers. Simultaneously, skip connections are added to the encoder of the improved neural implicit surface reconstruction model (e.g., concatenating shallow and deep features) to help the model retain low-dimensional spatial details and avoid gradient vanishing. Through the collaborative work of the SDF, color, and semantic branches, the improved neural implicit surface reconstruction model achieves high-precision, high-fidelity 3D reconstruction of the target region. This model not only maintains the global stability of the reconstruction but also improves the accuracy and efficiency of reconstruction by introducing optimization methods such as self-attention mechanisms.

[0049] Furthermore, based on semantic vectors, the latent encoding of aerial imagery, and the normal vectors of sampling points in the explicit grid, an improved neural implicit surface reconstruction model is used to determine the color of each sampling point. This includes: mapping the aerial imagery to the latent space using an encoder to obtain its latent encoding; enhancing the expressive power of the sampling point's positional information and normal vector using positional encoding to obtain positional and normal vector encodings; fusing the semantic vectors, the latent encoding of the aerial imagery, the positional encoding, and the normal vector encoding to obtain a second fused feature; and inputting the second fused feature into the improved neural implicit surface reconstruction model to obtain the color of the sampling point.

[0050] Specifically, the process of mapping aerial imagery to a latent space through an encoder essentially involves compressing and transforming high-dimensional image data into a low-dimensional latent representation. This representation captures the key features and structural information of the imagery. Simultaneously, the semantic vector, the latent encoding of the aerial imagery, the positional encoding, and the normal vector encoding are fused to obtain a second fused feature. This increases the feature's dimensionality and expressive power, helping the model learn more complex color change patterns. Through the computation of the improved neural implicit surface reconstruction model, the color of each sampling point is determined.

[0051] Step 105: Perform volume rendering based on the SDF values ​​and colors of each sampling point in the target area to obtain the rendered image. In this embodiment, the density value of the sampling point is determined based on its SDF value, as follows:

[0052] .

[0053] In the formula, This represents the density value of the sampling points. Let represent the derivative of the sigmoid function with a learnable scaling factor s. Indicates the direction of the ray. This represents the SDF value at the sampling point. This represents the derivative of the SDF value with respect to the location information of the sampling point. This indicates the location information of the sampling point.

[0054] The pixel color at a sampling point is determined based on the density value and color of the sampling point, as follows:

[0055] .

[0056] In the formula, The pixel color used in volume rendering calculations. This represents the opacity of the i-th sampling point. , Indicates the interval between adjacent sampling points. This represents the density value at the sampling interval of the i-th sampling point. This represents the cumulative transparency of the i-th sampling point. , Let m represent the color of the space at the i-th sampling point, and m represent the number of sampling points. Output the rendered image based on the pixel colors at each sampling point in the target region.

[0057] Specifically, the pixel colors at each sampling point are arranged and combined according to their spatial location, and volumetric rendering technology is used to integrate these discrete pixel color information into a complete image. In this process, volumetric rendering technology considers the propagation path of light in three-dimensional space and its interaction with each sampling point, thereby simulating realistic lighting effects and the reflective properties of object surfaces. Finally, by calculating and synthesizing the pixel colors of all sampling points, a rendered image of the target area is obtained. This image accurately reflects the three-dimensional structure and surface features of the target area, providing strong support for subsequent scene understanding and analysis.

[0058] Step 106: Extract the scene surface and texture map of the target area from the rendered image based on semantic vectors. In this embodiment, multiple triangular facets of the target scene are extracted from the rendered image to obtain the scene surface. The scene surface is UV unwrapped, and the multiple triangular facets are tiled to obtain a tiled image. The vertices of the triangular facets are projected onto the pixel space of the corresponding aerial image, and the color of the corresponding area is obtained through bilinear interpolation. The triangular facet corresponding to each pixel in the tiled image is filled according to the color of the corresponding area. Clustering is performed based on the semantic vectors of the pixels in each triangular facet to obtain the semantic labels of the corresponding triangular facets. The semantic correlation between adjacent triangular facets is analyzed based on the semantic labels of each triangular facet to determine the type of seam. The type of seam includes intra-class seams and cross-class seams. A stitching method is determined based on the type of seam to smooth the color of the seams between the triangular facets and obtain the texture map.

[0059] Specifically, the scene in the rendered image is extracted and represented in the form of triangular facets. A block-based algorithm is used to extract the triangular facets of the scene surface piece by piece, and then the texture map of the scene surface is further extracted.

[0060] Furthermore, the mapping relationship between triangular facets and pixels is determined by reverse lookup using UV coordinates, and the semantic vectors of pixels are clustered to obtain semantic labels for the triangular facets. All triangular facets are traversed, and edge pixels between adjacent facets are identified. Edge pixels shared by two facets in UV space, or edge pixels with a distance less than a threshold (e.g., 1-2 pixels), are considered seam regions (i.e., splicing seams). If the semantic labels of the facets on both sides of the splicing seam are the same, it is defined as a same-class splicing seam with a semantic relevance of 1; if the semantic labels of the facets on both sides of the splicing seam are different, it is defined as a cross-class splicing seam with a semantic relevance of 0. In addition, the cosine similarity of the semantic vectors of two adjacent facets can be calculated and used as the semantic relevance. When the semantic relevance is greater than 0.8, the splicing seam between two adjacent facets is considered a same-class splicing seam; otherwise, it is considered a cross-class splicing seam.

[0061] Specifically, for seams within the same category, a high weight (e.g., 0.8-1.0) is used to allow for a larger range of color fusion, extending the smoothing range to 3-5 pixels on both sides of the seam. Gaussian filtering (large standard deviation σ=2-3) or bilateral filtering (smoothing while preserving details) is used to enhance color continuity at the seam and eliminate color blocks. For seams across categories, a low weight (e.g., 0.1-0.3) is used for only slight transitions, limiting the smoothing range to 1-2 pixels on both sides of the seam. Mean filtering (small 3×3 window) or local correction is only applied to pixels with excessively large jumps (e.g., pixels exceeding the semantic category color gamut) to preserve semantic boundaries.

[0062] In one embodiment, since lighting conditions may change when acquiring aerial images, the appearance of the same scene surface may change. To adapt to variable lighting conditions and enable post-processing for photometric changes, each aerial image is assigned a length of [length missing]. The real-valued appearance embedding vector is used. That is, when training with pixel supervision from aerial imagery, sampling is performed along the ray of that pixel. When calculating the color of the sampled point, the latent encoding of the input aerial imagery corresponds to the appearance embedding vector of that aerial imagery. When generating texture maps of the target scene, the input to the latent appearance encoding of the neural implicit surface reconstruction model is a fixed value, thus ensuring the consistency of the scene appearance.

[0063] It should be noted that the derivative of the SDF value with respect to position at the scene surface can be regarded as the normal vector of the scene surface. When extracting texture maps, since the direction of the ray is unknown, the normal vector of the surface is used as the observation direction.

[0064] Step 107: Surface reconstruction of the target region based on the scene surface and texture maps. In this embodiment, an initial three-dimensional geometric framework of the target region is constructed based on the scene surface. A mesh is generated on the surface of the initial three-dimensional geometric framework to obtain the initial mesh. The pixel information in the texture map is accurately mapped onto the surface of the initial mesh to establish the correspondence between the texture and the three-dimensional coordinates (i.e., UV mapping). That is, the correspondence between the pixels of the texture map and the vertices of the three-dimensional mesh is calculated, and accurate mapping is performed according to the correspondence. If the target region is covered by multiple texture maps, the textures are stitched together by weighted fusion to eliminate color differences at the seams and obtain the reconstructed model of the target scene. For example, weighted fusion can use distance-based weights, where the closer a pixel is to the center of the image, the higher its weight.

[0065] This application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements a three-dimensional modeling method based on neural implicit surface reconstruction as described in any of the above method embodiments.

[0066] This application provides a terminal device including a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to achieve a corresponding method flow or function. The processor described in this application embodiment can be used for the operation of a three-dimensional modeling method based on neural implicit surface reconstruction.

[0067] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements any one of the three-dimensional modeling methods based on neural implicit surface reconstruction described in the above method embodiments.

[0068] In one embodiment, this application also provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a terminal device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and extended storage media supported by the terminal device. The computer-readable storage medium provides storage space that stores the terminal's operating devices. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the three-dimensional modeling method based on neural implicit surface reconstruction described in the above embodiment.

[0069] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0070] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0071] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0072] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0073] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and not to limit them. Although this application has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of this application. Any modifications or equivalent substitutions that do not depart from the spirit and scope of this application should be covered within the protection scope of the claims of this application.

Claims

1. A three-dimensional modeling method based on neural implicit surface reconstruction, characterized in that, include: Semantic segmentation is performed on aerial images of the target area to determine the pixel categories and generate a category label mask for the aerial image. Aerial triangulation is performed on aerial images of the target area to obtain the pose of each aerial image. An explicit mesh for the target region is constructed based on aerial imagery and its pose. The improved neural implicit surface reconstruction model inputs the category label mask and its corresponding 3D coordinates in the explicit grid to obtain the SDF value, color, and semantic vector of the target region. This includes: projecting the 3D coordinates of each sampling point in the explicit grid onto a three-plane grid to determine the planar features of the sampling points' 3D coordinates on the three planes; using the planar features and the category label mask as input to the improved neural implicit surface reconstruction model to obtain the SDF value and semantic vector of each sampling point; and determining the color of each sampling point based on the semantic vector, the latent encoding of the aerial image, and the normal vector of the sampling point in the explicit grid. The backbone network of the improved neural implicit surface reconstruction model is a multilayer perceptron, and the branch networks include an SDF branch, a color branch, and a semantic branch. Volume rendering is performed based on the SDF values ​​and colors of each sampling point in the target region to obtain a rendered image. This includes determining the density values ​​of the sampling points based on their SDF values, as follows: ； In the formula, This represents the density value of the sampling points. Let represent the derivative of the sigmoid function with a learnable scaling factor s. Indicates the direction of the ray. This represents the SDF value at the sampling point. This represents the derivative of the SDF value with respect to the location information of the sampling point. This indicates the location information of the sampling point; The pixel color at the sampling point is determined based on the density value and color of the sampling point, as follows: ； In the formula, The pixel color used in volume rendering calculations. This represents the opacity of the i-th sampling point. , Indicates the interval between adjacent sampling points. This represents the density value at the sampling interval of the i-th sampling point. This represents the cumulative transparency of the i-th sampling point. , Let m represent the color of the space at the i-th sampling point, and m represent the number of sampling points. The rendered image is output based on the pixel color at each sampling point in the target area; Extract the scene surface and texture map of the target region from the rendered image based on semantic vectors; Surface reconstruction of the target area is performed based on scene surface and texture map.

2. The three-dimensional modeling method based on neural implicit surface reconstruction as described in claim 1, characterized in that, The explicit mesh for constructing the target region based on aerial imagery and its pose includes: The aerial images were cleaned and registered. Based on the pose of each pixel in the aerial image, generate corresponding point cloud data; Poisson reconstruction is performed on the point cloud data to construct an explicit mesh for the target region.

3. The three-dimensional modeling method based on neural implicit surface reconstruction as described in claim 1, characterized in that, The step of using the planar features and the category label mask as input to the improved neural implicit surface reconstruction model to obtain the SDF value and semantic vector of each sampling point includes: The planar features of the sampling points are fused with the location information to obtain the first fused feature; The first fusion feature is input into the improved neural implicit surface reconstruction model to obtain the SDF value and semantic vector of the sampling point.

4. The three-dimensional modeling method based on neural implicit surface reconstruction as described in claim 1, characterized in that, The step of using the planar features and the category label mask as input to the improved neural implicit surface reconstruction model to obtain the SDF value and semantic vector of each sampling point further includes: Bilinear interpolation is performed on the projected coordinates of each plane to obtain the plane features; The planar features and location information of the sampling points are concatenated to obtain the concatenated features of the corresponding sampling points; A three-dimensional window is defined with the sampling point as the center, and the surrounding neighboring points are determined based on the three-dimensional window; Bilinear interpolation and positional information splicing are performed on the neighboring points to obtain the spliced ​​features of the neighboring points; A linear transformation is performed on the splicing features of the sampling points to obtain their Q values; The K and V values ​​of the sampling points are determined by the splicing characteristics of neighboring points; Similarity is determined based on Q and K values, and attention weights are obtained after normalization. The self-attention feature is obtained by weighting and summing the V values ​​using the attention weights. The splicing features of the sampling points are updated by the self-attention features, and the updated splicing features of the sampling points are input into the improved neural implicit surface reconstruction model to obtain the updated SDF value and semantic vector.

5. The three-dimensional modeling method based on neural implicit surface reconstruction as described in claim 1, characterized in that, The determination of the color of each sampling point based on the semantic vector, the latent encoding of the aerial image, and the normal vector of the sampling point in the explicit grid, using an improved neural implicit surface reconstruction model, includes: Aerial imagery is mapped to a latent space using an encoder to obtain its latent encoding. By utilizing positional encoding to enhance the expressive power of the sampling point's positional information and normal vector, positional encoding and normal vector encoding are obtained; The semantic vector, the latent encoding of the aerial image, the position encoding, and the normal vector encoding are fused to obtain the second fused feature; The second fusion feature is input into the improved neural implicit surface reconstruction model to obtain the color of the sampling point.

6. The three-dimensional modeling method based on neural implicit surface reconstruction as described in claim 1, characterized in that, The extraction of scene surface and texture map of the target region from the rendered image based on semantic vectors includes: Multiple triangular facets of the target scene are extracted from the rendered image to obtain the scene surface; The scene surface is UV unwrapped, and multiple triangular facets within it are tiled to obtain a tiled image. The vertices of the triangular facets are projected onto the pixel space of the corresponding aerial image, and the color of the corresponding area is obtained by bilinear interpolation. Fill the triangular facet corresponding to each pixel in the tiled image with the color of the corresponding region; Clustering is performed based on the semantic vectors of pixels in each triangular facet to obtain the semantic labels of the corresponding triangular facets; The semantic association between adjacent triangular facets is analyzed based on the semantic tags of each triangular facet to determine the type of splicing seam; wherein, the type of splicing seam includes same-class splicing seams and cross-class splicing seams; The splicing method is determined based on the type of splicing seam, and the seams between each triangular facet are smoothed with color to obtain a texture map.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the three-dimensional modeling method based on neural implicit surface reconstruction as described in any one of claims 1 to 6.

8. A non-volatile computer-readable storage medium, characterized in that, Includes storage for computer programs or instructions that, when executed, enable the three-dimensional modeling method based on neural implicit surface reconstruction as described in any one of claims 1 to 6 to be implemented.

Images