A three-dimensional reconstruction method, device, equipment and medium based on a unified semantic field

Abstract

This application discloses a method, apparatus, device, and medium for 3D reconstruction based on a unified semantic field. This application can preprocess multi-source heterogeneous data to obtain processed multi-source data with a unified format; extract and align feature points from the processed multi-source data, and combine this with joint regional network adjustment to determine the device pose and the sparse 3D point cloud corresponding to the target 3D region; use a pre-set geometric field construction model to fill and smooth the sparse 3D point cloud to obtain a continuous geometric field corresponding to the target 3D region; perform semantic segmentation on the full-scene images in the processed multi-source data, and map multiple semantic labels to the continuous geometric field through spatial backprojection based on the device pose to obtain a unified 3D semantic field corresponding to the target 3D region; perform block-based distributed modeling based on the unified 3D semantic field to generate a 3D spatial model corresponding to the target 3D region, improving the efficiency and effectiveness of urban 3D reconstruction.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a three-dimensional reconstruction method, apparatus, device, and medium based on a unified semantic field. Background Technology

[0002] Traditional city-level multi-scale 3D reconstruction often employs a single data source for each modeling task. For example, for urban aerial scenes, oblique photography equipment is used to acquire images and complete the reconstruction; for ground buildings, manual on-site surveying and modeling are used; and for interior spaces, specialized scanning equipment or CAD (Computer-Aided Design) drawings are used for BIM (Building Information Modeling) modeling. These processes operate independently, and the output models are typically stored independently and unrelated. This results in a high degree of reliance on manual intervention for stitching together models from different scenes, i.e., manual coordinate alignment by professionals. This approach is not only extremely inefficient but also struggles to guarantee the continuity of model geometry and topological consistency in large-scale, complex scenes, easily leading to misalignments and gaps, thus affecting the quality of urban 3D reconstruction. Summary of the Invention

[0003] This application provides a three-dimensional reconstruction method, apparatus, device, and medium based on a unified semantic field, which can improve the effect of urban three-dimensional reconstruction.

[0004] This application provides a three-dimensional reconstruction method based on a unified semantic field, including: Acquire multi-source heterogeneous data of a target three-dimensional region collected by multiple devices, and preprocess the multi-source heterogeneous data to obtain processed multi-source data with a unified format; Feature points are extracted and aligned from the processed multi-source data, and combined with joint regional network adjustment calculations to determine the device pose and the sparse 3D point cloud corresponding to the target 3D region. A pre-set geometric field construction model is used to fill and smooth the sparse three-dimensional point cloud to obtain the continuous geometric field corresponding to the target three-dimensional region. Semantic segmentation is performed on the full-scene image in the processed multi-source data to obtain multiple semantic labels. Based on the device pose, the multiple semantic labels are mapped to the continuous geometric field through spatial back projection to obtain a unified three-dimensional semantic field corresponding to the target three-dimensional region. Based on the unified three-dimensional semantic field, block-based distributed modeling is performed to generate a three-dimensional spatial model corresponding to the target three-dimensional region.

[0005] In some embodiments, the plurality of devices includes imaging devices and non-imaging scanning devices. The step of extracting and aligning feature points from the processed multi-source data, and combining this with joint regional network adjustment to determine the device pose and the sparse 3D point cloud corresponding to the target 3D region, includes: Based on a pre-set deep learning model, feature points are extracted from the processed multi-source data to obtain multi-source feature points to be matched. Identify and align matching feature points indicating the same object from a plurality of multi-source feature points to be matched, so as to obtain a plurality of aligned multi-source feature points; Based on the projection relationship of the aligned multi-source feature points in the multi-view image and their geometric constraints in the laser point cloud data, the initial pose of the imaging device is determined, and an initial three-dimensional point cloud covering the target three-dimensional region is generated. Using point cloud data collected by laser scanning equipment as geometric constraints, a joint regional network adjustment model is used to jointly optimize the initial pose and initial 3D point cloud of the imaging equipment and non-imaging equipment, so as to obtain the sparse 3D point cloud corresponding to the device pose and the target 3D region.

[0006] In some embodiments, based on the projection relationship of the aligned multi-source feature points in multi-view images and their geometric constraints in laser point cloud data, the initial pose of the imaging device is determined, and an initial 3D point cloud covering the target 3D region is generated, including: Based on the aligned multi-source feature points in multi-view images of different data types, the intrinsic parameters of the imaging device are determined. The pose of the imaging device is determined based on the projection position of the aligned multi-source feature points in the multi-view image, the geometric constraints of the aligned multi-source feature points in the laser point cloud data, and the intrinsic parameters. Using the pose of any of the imaging devices as the initial origin, and based on the geometric intersection relationship between the multi-view images, triangulation is performed on the aligned multi-source feature points to recover the coordinates of the aligned multi-source feature points in three-dimensional space, thereby generating an initial three-dimensional point cloud covering the target three-dimensional region.

[0007] In some embodiments, a preset geometric field construction model is used to fill and smooth the sparse three-dimensional point cloud to obtain a continuous geometric field corresponding to the target three-dimensional region, including: Construct an initial geometric field model and obtain sample 3D point clouds; Layered random sampling is performed on the surface of the sample 3D point cloud and within a preset range to generate several training sample points. For each training sample point, calculate the Euclidean distance from the training sample point to the nearest neighbor point in the sample 3D point cloud, and use the Euclidean distance with positive and negative signs as the ground truth label of the training sample point based on the spatial location information of the training sample point. The training sample points are input into the initial geometric field construction model to predict the symbolic distance value of the training sample points; The loss function value is determined based on the difference between the symbolic distance value and the truth label, and the model parameters of the initial geometric field construction model are iteratively optimized through the backpropagation algorithm until the model converges, thus obtaining the preset geometric field construction model. Random sampling is performed on the sparse three-dimensional point cloud surface and within a preset range to obtain query sampling points. The query sampling points are then input into the preset geometric field construction model. The symbolic distance value corresponding to each query sampling point is output, and the isosurface with a symbolic distance value of zero is extracted to obtain the continuous geometric field corresponding to the target three-dimensional region.

[0008] In some embodiments, semantic segmentation is performed on the full-scene image in the processed multi-source data to obtain multiple semantic labels, and the multiple semantic labels are mapped to the continuous geometric field through spatial back projection based on the device pose to obtain a unified three-dimensional semantic field corresponding to the target three-dimensional region, including: Based on a pre-built semantic segmentation network, semantic recognition is performed on the full-scene images in the processed multi-source data to obtain multiple semantic labels carrying category information. Based on the intrinsic and extrinsic parameters of the imaging device in the device pose, establish the projection correspondence between pixels and three-dimensional spatial rays in the full scene image; From the full scene image, target pixels whose semantic labels belong to the non-background category are selected, and dense sampling is performed along the spatial ray direction corresponding to the target pixel in the physical space represented by the continuous geometric field to generate candidate three-dimensional sampling points. Based on the surface constraints of the continuous geometric field, effective three-dimensional sampling points located on the scene geometric surface and near-surface region are selected from the candidate three-dimensional sampling points. By associating the effective 3D sampling points with their corresponding semantic labels, a unified 3D semantic field corresponding to the target 3D region is obtained.

[0009] In some embodiments, semantic recognition is performed on the full-scene images in the processed multi-source data based on a preset semantic segmentation network to obtain multiple semantic labels carrying category information, including: Based on a pre-built semantic segmentation network, semantic recognition is performed on full-scene images from different perspectives in the processed multi-source data to obtain the initial semantic labels and label confidence of each region in the full-scene image. Multiple initial semantic labels and label confidence scores belonging to the same target region are selected from the full-scene images from different perspectives; Based on the tag confidence, the confidence of the same semantic tags in the multiple initial semantic tags is weighted to obtain multiple aggregated semantic tags and their corresponding aggregated tag confidence; Based on the confidence level of the aggregated labels, the semantic label corresponding to the target region is determined from multiple aggregated semantic labels.

[0010] In some embodiments, block-based distributed modeling is performed based on the unified three-dimensional semantic field to generate a three-dimensional spatial model corresponding to the target three-dimensional region, including: Based on the spatial range and geometric semantic distribution characteristics of the unified three-dimensional semantic field, the target three-dimensional region is divided into multiple computational sub-blocks according to the scene complexity. Each of the computational sub-blocks is distributed to different computational units for distributed parallel training, generating a local 3D model corresponding to each computational sub-block; The local 3D model is clipped and registered to be stitched together to generate a globally consistent 3D spatial model.

[0011] This application also provides a three-dimensional reconstruction device based on a unified semantic field, comprising: The data processing module is used to acquire multi-source heterogeneous data of a target three-dimensional region collected by multiple devices, and to preprocess the multi-source heterogeneous data to obtain processed multi-source data with a unified format. The point cloud extraction module is used to extract and align feature points from the processed multi-source data, and combine it with joint regional network adjustment calculation to determine the device pose and the sparse three-dimensional point cloud corresponding to the target three-dimensional region. The geometry field construction module is used to fill and smooth the sparse three-dimensional point cloud using a preset geometry field construction model to obtain the continuous geometry field corresponding to the target three-dimensional region. The semantic embedding module is used to perform semantic segmentation on the full-scene image in the processed multi-source data to obtain multiple semantic labels, and to map the multiple semantic labels to the continuous geometric field through spatial back projection based on the device pose to obtain a unified three-dimensional semantic field corresponding to the target three-dimensional region. The distributed reconstruction module is used to perform block-based distributed modeling based on the unified three-dimensional semantic field to generate a three-dimensional spatial model corresponding to the target three-dimensional region.

[0012] This application also provides an electronic device, including a memory storing multiple instructions; the processor loads instructions from the memory to execute steps in any of the three-dimensional reconstruction methods based on a unified semantic field provided in this application.

[0013] This application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps in any of the three-dimensional reconstruction methods based on a unified semantic field provided in this application.

[0014] This application also provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute steps in any of the three-dimensional reconstruction methods based on a unified semantic field provided in this application.

[0015] This application embodiment can acquire multi-source heterogeneous data from multiple devices targeting a 3D region, and preprocess the heterogeneous data to obtain processed multi-source data with a unified format. Feature points are extracted and aligned from the processed multi-source data, and combined with joint regional network adjustment calculations to determine the device pose and the sparse 3D point cloud corresponding to the target 3D region. A pre-set geometric field construction model is used to fill and smooth the sparse 3D point cloud to obtain a continuous geometric field corresponding to the target 3D region. Semantic segmentation is performed on the full-scene images in the processed multi-source data to obtain multiple semantic labels, and these semantic labels are mapped to the continuous geometric field through spatial backprojection based on the device pose to obtain a unified 3D semantic field corresponding to the target 3D region. Block-based distributed modeling is performed based on the unified 3D semantic field to generate a 3D spatial model corresponding to the target 3D region. Thus, by jointly optimizing multi-source data to obtain a continuous geometric skeleton, embedding 2D semantics into a consistent 3D semantic field through geometric constraints, and finally performing parallel block-based modeling based on the semantic field, the efficiency and effectiveness of urban 3D reconstruction are improved. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating a three-dimensional reconstruction method based on a unified semantic field provided in an embodiment of this application; Figure 2 This is a schematic diagram of the process for constructing a sparse 3D point cloud in an embodiment of this application; Figure 3This is a schematic diagram of the process for constructing a unified three-dimensional semantic field in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of a three-dimensional reconstruction device based on a unified semantic field provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] Traditional city-level multi-scale 3D reconstruction methods typically suffer from the following drawbacks when facing increasingly complex urban environments: 1. Data fragmentation and discontinuous scenarios: Traditional solutions are based on a single data source and model each scenario separately, resulting in fragmentation of multi-scale data of urban open space and indoor spaces, and discontinuity in scenario connection. The alignment accuracy of multi-scale model fusion can only reach the meter level, which cannot meet the requirements of high-precision spatiotemporal mapping.

[0020] 2. Semantic alignment is difficult in multi-source models: Traditional solutions require semantic alignment and individualization after modeling. The model features of the same target are inconsistent at different scales, and the segmentation accuracy depends on the contour alignment accuracy extracted manually. The degree of automation is low and the semantic matching error is large.

[0021] 3. Low modeling and updating efficiency: For large-scale city-level scenarios, traditional single-unit reconstruction or integrated modeling methods often use single-machine serial computing, which takes a very long time to compute when dealing with massive amounts of data, and it is difficult to dynamically adjust computing resources according to the complexity of the scenario, thus failing to meet the needs of digital twin cities for rapid modeling and updating.

[0022] In view of this, this application proposes a three-dimensional reconstruction method based on a unified semantic field to solve the above problems.

[0023] For details, please refer to Figure 1 The specific process of this 3D reconstruction method can be summarized as follows: S110. Acquire multi-source heterogeneous data of the target three-dimensional region collected by multiple devices, and preprocess the multi-source heterogeneous data to obtain processed multi-source data with a unified format.

[0024] Among them, multi-source heterogeneous data refers to a set of data from various types of devices and different data formats (images, point clouds, videos, etc.), which may be inconsistent in resolution, coordinate system or time, but all describe the same target area.

[0025] For example, for a 10-square-kilometer urban scene, aerial photography can be conducted using drones to obtain oblique images, and point cloud and image data of outdoor scenes and building interiors can be collected using lasers and image scanners. Existing monitoring equipment can be used to supplement street-level ground images. Thus, by using these devices to collect data synchronously or temporally from the sky, ground, and interior of the same target 3D area at multiple scales and from multiple perspectives, a multi-source heterogeneous dataset covering the entire area can be formed.

[0026] Post-processed multi-source data refers to a standardized set of input data that has been processed from original multi-source heterogeneous data through a series of preprocessing operations such as cleaning, format conversion, spatiotemporal benchmark unification, and quality screening. This set of data has a consistent data structure, unified spatial reference, and can be directly used for feature extraction and geometric calculation.

[0027] For example, the preprocessing of multi-source heterogeneous data can include: 1. Data cleaning and invalid data removal: For example, for image data, remove blurry, severely motion-blurred, overexposed, and underexposed images, and remove solid colors, black screens, and redundant frames with extremely high repetition rates; for point cloud data, remove outliers (such as flying points and scanning noise) and remove obviously erroneous scanning stripes or abnormal reflection points. 2. Format unification and standardization: Convert data from different sources into a universal, interoperable format, and unify file naming rules and timestamp formats to facilitate subsequent indexing and matching. 3. Spatiotemporal reference unification: Utilize existing control points or GPS (Global Positioning System) information to uniformly convert UAV images, ground laser point clouds, and indoor scan point clouds to the same spatial coordinate system (such as an urban engineering coordinate system); unify the time zone and format alignment for acquisition times from different devices, such as using UTC (Coordinated Universal Time) timestamps, to provide a time consistency basis for subsequent multi-source data association. 4. Quality screening and data integrity verification: Check whether the image clarity, texture richness, point cloud density, coverage, and overlap with the image meet the minimum quality requirements of subsequent algorithms, and mark or remove data that does not meet the requirements to ensure that the data entering the next stage has sufficient geometric and texture information for feature extraction and registration.

[0028] This embodiment integrates existing and newly added multi-source data collected by multiple devices, and performs preprocessing such as cleaning, format and spatiotemporal benchmark unification, and quality screening to obtain a set of standardized, computable, and high-quality processed multi-source data, providing a data foundation for subsequent feature alignment, joint adjustment, and 3D reconstruction.

[0029] S120. Feature points are extracted and aligned from the processed multi-source data, and combined with joint regional network adjustment calculations, the sparse 3D point cloud corresponding to the device pose and the target 3D region is determined.

[0030] Feature point extraction and alignment of processed multi-source data refers to extracting stable and identifiable features (points, lines, corners, etc.) from images or point clouds, and finding feature correspondences for "the same physical location" across different data sources to achieve preliminary registration or alignment. For example, extracting "corner point A" from a UAV image and also extracting "corner point A" from a ground image, and marking them as the same feature; extracting "curb point" from a ground image and finding the corresponding "curb point" from a laser point cloud.

[0031] Joint regional network adjustment refers to integrating multi-view images, point clouds, and control information into a unified mathematical model. The goal is to minimize reprojection errors or geometric inconsistencies, optimizing parameters such as camera pose and point cloud coordinates to ensure consistency across multiple data sources within a unified coordinate system. For example, by acquiring 200 images and laser point clouds from three stations, joint regional network adjustment aligns all image poses and point cloud coordinates, reducing accumulated errors. Furthermore, the laser point clouds can be used as strong geometric constraints during processing to improve consistency across the sky, ground, and indoor environments.

[0032] Device pose refers to the position and orientation of a device when acquiring data, including at least translational and rotational angles. For camera-type devices, pose is expressed as its extrinsic parameters. For example, the camera pose of a drone can be: position coordinates and attitude (such as pitch angle, roll angle, yaw angle, rotation matrix, etc.); the pose of a surveillance camera can be: installation coordinates and pointing angle; the pose of a laser scanner can be: scanning station coordinates and horizontal or vertical angular reference.

[0033] Sparse 3D point clouds refer to relatively sparse sets of 3D points obtained through multi-view matching. They can be used to represent the structural skeleton of a scene, but are usually incomplete, contain holes, and are noisy. For example, hundreds of thousands to millions of 3D points obtained from image matching are unevenly spaced, with points denser at building edges and sparser in open areas. Sparse 3D point clouds can be used as input for subsequent construction of continuous geometric fields.

[0034] Please refer to Figure 2 , Figure 2 This is a schematic diagram illustrating the process of constructing a sparse 3D point cloud in an embodiment of this application. For example... Figure 2 As shown, the multiple devices include imaging devices and non-imaging scanning devices. The process of extracting and aligning feature points from the processed multi-source data, and combining this with joint regional network adjustment to determine the device pose and the sparse 3D point cloud corresponding to the target 3D region, can include: S120-1. Based on a pre-set deep learning model, feature points are extracted from the processed multi-source data to obtain multi-source feature points to be matched.

[0035] The pre-built deep learning model refers to a pre-trained neural network model capable of automatically extracting highly robust feature descriptors from images or point clouds. For example, a convolutional neural network based on the SuperPoint architecture or an image feature extractor based on Transformer can be used to extract feature vectors from salient locations such as corners and edges in the image. For laser point clouds, networks such as PointNet++ or KPConv can be used to extract their geometric features.

[0036] Multi-source matching feature points refer to sets of features independently extracted from different data sources (such as drone images, ground surveillance video frames, and laser point clouds) using deep learning models, without established correspondences. These are potential matching candidate points. For example, feature point A1, representing the "top right corner of a building," extracted from an overhead view taken by a drone; feature point G1, representing the "top right corner of the same building," extracted from a depth map generated by ground laser point cloud projection; and similar feature point C1, extracted from another ground surveillance image. In this case, feature points A1, G1, and C1 are independent of each other and are collectively referred to as multi-source matching feature points.

[0037] In the process of feature point extraction from processed multi-source data, for each image, a pre-built deep learning model can be used to extract the two-dimensional coordinates and corresponding high-dimensional feature descriptors of key points (such as building corners and road marking endpoints). For point cloud data, a pre-built deep learning model can be used to voxelize the point cloud, extract points or superpoints with significant geometric features (such as edges and abrupt changes in surface curvature), and generate geometric feature descriptors. This results in a large number of multi-source feature points scattered across different datasets, waiting to be stitched and aligned.

[0038] S120-2. Identify and align matching feature points indicating the same object from several multi-source feature points to be matched, so as to obtain several aligned multi-source feature points.

[0039] Aligned multi-source feature points refer to a set of feature points that have been identified by a pre-set matching algorithm as describing the same physical space point (i.e., indicating the same object) and have established a cross-data source correspondence. For example, by calculating the similarity between the feature vectors of feature points, it is confirmed that feature points A1, G1, and C1 are actually projections of the same physical corner point in three different datasets. They are then marked as a set of "aligned multi-source feature points" and assigned the same ID.

[0040] In some embodiments, when performing cross-modal matching and alignment of all multi-source feature points to be matched, for image-to-image matching, the cosine distance or Euclidean distance of the feature descriptors can be used for nearest neighbor search; for image-point cloud matching, the point cloud can be projected onto the image plane first, and then the matching calculation can be performed with the image, or a cross-modal metric relationship can be directly established in the feature space to achieve image-point cloud matching. After preliminary matching, the random sampling consensus algorithm and epipolar constraints can be used to eliminate mismatched multi-source feature points. For example, when matching UAV images with ground point clouds, specific perspective geometric relationships must be satisfied, so the results after preliminary matching can be verified accordingly. Thus, successfully paired corresponding points are identified as aligned multi-source feature points. These points contain multi-view, multi-scale observation information, which can provide reliable observation values ​​for subsequent pose calculation.

[0041] S120-3. Based on the projection relationship of aligned multi-source feature points in multi-view images and their geometric constraints in laser point cloud data, determine the initial pose of the imaging device and generate an initial three-dimensional point cloud covering the target three-dimensional region.

[0042] The projection relationship of a certain aligned multi-source feature point in a multi-view image refers to the constraint relationship between the same 3D point in the physical world and the pixel positions in multiple images. Its geometric constraint in laser point cloud data refers to the coordinates of the 3D point detected by the laser point cloud.

[0043] For example, suppose feature point F is the same point in the physical world, and it has projections or observations in different data (aligning multi-source feature points): it corresponds to pixel (u1, v1) in image 1, pixel (u2, v2) in image 2, and 3D point P (or may correspond to a nearest neighbor) in the point cloud. The pixel positions of this same feature point F in different images constitute the projection relationship, while its coordinates in the laser point cloud serve as geometric constraints.

[0044] In some embodiments, the process of determining the initial pose of the imaging device and generating an initial 3D point cloud covering the target 3D region based on the projection relationship of aligned multi-source feature points in multi-view images and their geometric constraints in laser point cloud data may include: Based on the aligned multi-source feature points in multi-view images of different data types, the intrinsic parameters of the imaging device are determined; based on the projection position of the aligned multi-source feature points in the multi-view images, the geometric constraints of the aligned multi-source feature points in the laser point cloud data, and the intrinsic parameters, the pose of the imaging device is determined; taking the pose of any imaging device as the initial origin, based on the geometric intersection relationship between the multi-view images, triangulation is performed on the aligned multi-source feature points to recover the coordinates of the aligned multi-source feature points in three-dimensional space, and an initial three-dimensional point cloud covering the target three-dimensional region is generated.

[0045] Intrinsic parameters refer to the internal parameters of the imaging device, such as focal length, principal point, and distortion coefficient.

[0046] During camera intrinsic parameter calibration, the intrinsic parameters of the imaging device can be estimated using a multi-view geometric self-calibration algorithm based on the aligned multi-source feature points identified in multi-view images of different data types (such as UAV oblique images and ground monitoring video frames). By utilizing the correspondence between feature points in multiple images and combining it with a lens distortion model, the intrinsic parameter matrices such as focal length, principal point coordinates, and distortion coefficients of each imaging device can be optimized and solved, providing an accurate imaging model foundation for subsequent 3D reconstruction. Alternatively, information such as focal length, image size, acquisition time, GPS, and initial attitude values ​​can be directly extracted from image files or acquisition metadata using the EXIF ​​information extraction method.

[0047] Then, based on the intrinsic parameters of the imaging device, and using the aligned multi-source feature points obtained in the aforementioned steps, a 3D model is constructed. 2D correspondence: On the one hand, it utilizes the 2D projection positions of aligned multi-source feature points in multi-view images; on the other hand, it uses the 3D geometric coordinates of these feature points in laser point cloud data or their nearest neighbors as strong geometric constraints. This is achieved through PnP (Perspective Point Mapping). n The Point algorithm and its variants, combined with bundle adjustment, are used to inversely solve for the extrinsic parameters of the imaging device when each image is captured, namely the rotation matrix and translation vector, thus obtaining the initial pose of all imaging devices in a unified world coordinate system. The introduction of laser point cloud geometric constraints effectively eliminates the scale drift and accumulated errors present in pure visual SFM (Structure from Motion), ensuring the reliability of pose calculation.

[0048] Finally, using the pose of any imaging device as the initial origin, triangulation is performed on the aligned multi-source feature points using the geometric intersection relationships between multi-view images. Specifically, for each aligned multi-source feature point, a 3D spatial ray is back-projected based on its pixel coordinates in one image, along with the image's pose and intrinsic parameters; similarly, a 3D spatial ray can be back-projected onto the corresponding pixel of the feature point in another image. By minimizing the distance between these two rays, the precise coordinates of the feature point in 3D space can be recovered. By traversing all aligned multi-source feature points, thousands upon thousands of such 3D coordinates are aggregated to form an initial 3D point cloud covering the target 3D region.

[0049] S120-4. Using the point cloud data collected by the laser scanning equipment as geometric constraints, a joint regional network adjustment model is used to jointly optimize the initial pose and initial 3D point cloud of the imaging equipment and non-imaging equipment to obtain the sparse 3D point cloud corresponding to the device pose and the target 3D region.

[0050] First, an optimization variable set is constructed, including the extrinsic parameters of the imaging device, the pose of the non-imaging device (such as a laser scanner), and the coordinates of three-dimensional points. For example, it may include: the extrinsic parameters (R) of each image or camera. i ,t i ), pose of each laser scanner or scanning station (R) s ,t s The initial 3D point cloud coordinates (X) j ,Y j Z j ).

[0051] Then, the reprojection residuals of image feature points and the points of the laser point cloud are calculated. Surface geometric constraint residuals. For example, for reprojection residuals: for each initial 3D point cloud j, first obtain its observed pixel Z in image i. ij = (u, v), and based on the 3D coordinates of the initial 3D point cloud j in the laser point cloud, the camera extrinsic parameters (R) i ,t i The intrinsic parameter K is used to predict the pixel of the aligned multi-source feature point j in the image through projection, thus obtaining the predicted pixel Z*. ij Then, calculate the observed pixel Z. ij With predicted pixel Z* ij The residual between: r ij =Z ij - Z* ij This allows for self-consistency between 3D points, camera pose, and intrinsic parameters, reducing reprojection errors. This applies to points in laser point clouds. Surface geometric constraint residuals: First, the laser point cloud is locally fitted into a plane or curved surface. Then, the minimum directed distance from the initial 3D point cloud j to the local laser surface is calculated, thus obtaining the point residuals of the initial 3D point cloud. Surface geometric constraint residuals.

[0052] Finally, a weighted nonlinear least squares optimization algorithm is used to simultaneously minimize the reprojection error and the point. The geometric constraint error is addressed by iteratively updating the pose parameters and the coordinates of the initial 3D point cloud until convergence. The final output is the optimized device pose and the sparse 3D point cloud corresponding to the target 3D region. This ensures that the optimized sparse point cloud closely fits the geometric surface represented by the laser point cloud in the normal direction, while allowing for reasonable adjustments along the surface tangent, thereby improving the consistency and accuracy of the stitching of point clouds from the sky, ground, and indoor areas.

[0053] S130. A pre-set geometric field construction model is used to fill and smooth the sparse three-dimensional point cloud to obtain the continuous geometric field corresponding to the target three-dimensional region.

[0054] In this context, the geometric field construction model refers to a learnable or computable function model that takes spatial coordinates as input and outputs geometrically relevant information (such as distance to the surface, occupancy probability, etc.) to transform discrete point clouds into continuous representations. For example, symbolic distance field (SDF) networks, NeRF-like radiation fields (which can derive geometry), and improved MLP neural networks can be used.

[0055] Continuous geometric fields refer to a continuous, queryable representation of the three-dimensional space of a target region. They can determine the relationship between any position and the surface of an object, and can be used to extract surfaces, fill holes, and smooth structures.

[0056] In some embodiments, a pre-defined geometric field construction model is used to fill and smooth the sparse 3D point cloud to obtain a continuous geometric field corresponding to the target 3D region, including: An initial geometric field construction model is constructed, and sample 3D point clouds are obtained. Layered random sampling is performed on the surface of the sample 3D point cloud and within a preset surrounding area to generate several training sample points. For each training sample point, the Euclidean distance from the training sample point to its nearest neighbor in the sample 3D point cloud is calculated, and based on the spatial location information of the training sample point, the signed Euclidean distance is used as the ground truth label. The training sample points are input into the initial geometric field construction model to predict their symbolic distance values. The loss function value is determined based on the difference between the symbolic distance value and the ground truth label, and the model parameters of the initial geometric field construction model are iteratively optimized using a backpropagation algorithm until the model converges, resulting in a preset geometric field construction model. Random sampling is performed on the sparse 3D point cloud surface to obtain query sampling points, which are then input into the preset geometric field construction model. The symbolic distance values ​​corresponding to each query sampling point are output, and isosurfaces with zero symbolic distance values ​​are extracted to obtain the continuous geometric field corresponding to the target 3D region.

[0057] During the training phase of the geometric field model, the core characteristic of the SDF network—the "signed distance from any point in space to the object surface (negative inside the object, positive outside)"—can be utilized. Discrete, sparse point cloud data can be used as a supervisory signal. By sampling points in the neighborhood of the point cloud, the network parameters are optimized using the distance labels from the point cloud to the real surface, enabling the network to learn the global geometric patterns and local detailed features of the urban scene. This approach transforms discrete 3D point sets into a continuous, differentiable, and globally consistent implicit geometric field. It can adaptively fill in empty areas of sparse point clouds and filter measurement noise, while also achieving smooth geometric transitions for key elements of the urban scene such as building walls, roof outlines, and road boundaries, accurately capturing the topological relationships and fine textures of complex structures.

[0058] In the process of constructing a model of the initial geometric field using sample 3D point clouds, such as supervising the training of an SDF network, sampling is first performed on the surface and near-surface space of the sample 3D point cloud. This includes: sampling at intervals along the normal (or approximate normal) of the point cloud at a fixed distance in the surface layer; sampling at intervals outward along the normal of the point cloud in the outer layer; and sampling at intervals inward along the normal of the point in the inner layer. Thus, by randomly sampling points at different distance layers (i.e., hierarchical random sampling), several training sample points are generated. For example, for each sample 3D point P, along its normal n, points are taken on its surface: point P itself, points P+0.02n are taken near the outer edge, points P+0.04n are taken further outward, and points P-0.02n are taken near the inner edge. The remaining sample 3D points are treated similarly, thus obtaining the training sample point set {X}. i}

[0059] Then, for each training sample point X iFind the nearest point Q in the 3D point cloud of the training sample, calculate the Euclidean distance d between them, and determine the sign. If the training sample point X... i If the point Q is outside the surface layer of a sample 3D point, its sign is "+"; if it is inside the layer, its sign is "-". Thus, each training sample point X can be obtained. i The ground truth labels are used for supervised training. For example, training sample points X i If a point is 3cm outside the wall, and its nearest point Q is on the wall surface, then its truth label is +3; if the training sample point X... i If the value is 2cm inside the wall, then its truth label is +2; if the training sample point X i If a value is very close to the surface of the sample's 3D point cloud, then its truth label is 0.

[0060] Then, the training sample points are input into the initial geometric field construction model to predict the symbolic distance value. The loss function value is determined based on the difference between the symbolic distance value and the ground truth label. The model parameters of the initial geometric field construction model are iteratively optimized through the backpropagation algorithm until the model converges, thus obtaining the preset geometric field construction model.

[0061] For example, given the input coordinates of a training sample point (10.2, 20.1, 5.0), the output symbolic distance value is... =+1.8. If its truth label is If the sum is +2.0, then the current loss function value is: - =-0.2. During updates, the loss function L can be used as follows: , in, This represents the ground truth label for the i-th training sample point. represents the symbolic distance value predicted by the i-th training sample point, and N represents the number of training sample points used in each update batch.

[0062] Thus, the initial geometric field construction model is updated in reverse until the loss no longer decreases or the maximum number of training iterations is reached, at which point the process stops, resulting in a pre-defined geometric field construction model. During this process, the predicted symbolic distance values ​​become increasingly closer to their corresponding ground truth labels, causing the zero isosurface to gradually stabilize into a continuous surface.

[0063] After obtaining the geometric field construction model, the trained model can be used to query sampling points and extract zero isosurfaces, thereby converting the sparse 3D point cloud into a continuous geometric field. Specifically, within a preset range on and around the surface of the sparse 3D point cloud, such as a minimum bounding box containing all sparse 3D point clouds, several query sampling points are randomly generated. The coordinates of each query sampling point are input into the geometric field construction model, and the corresponding symbolic distance value is output. Then, the symbolic distance value is processed using an isosurface extraction algorithm (such as the Marching Cubes algorithm): the query space (such as the minimum bounding box) is divided into a voxel mesh. Based on the positive and negative distribution of the symbolic distance values ​​at the eight vertices of each voxel, the intersection points of the zero isosurface (i.e., SDF≈0) and the voxel edges are determined. Linear interpolation is then performed on the edges according to the symbolic distance values ​​to obtain the approximate vertex positions of the zero isosurface, thereby generating triangular patches. After traversing all voxels, all triangular patches constitute the continuous geometric field corresponding to the target 3D region. This continuous geometric field is expressed in the form of a grid model or an implicit field. It has the characteristics of continuity, differentiability and global consistency, and can reflect the smooth geometric shape of urban scene elements such as building walls, roofs, and roads.

[0064] S140. Perform semantic segmentation on the full-scene image in the processed multi-source data to obtain multiple semantic labels. Based on the device pose, map the multiple semantic labels to the continuous geometric field through spatial back projection to obtain the unified three-dimensional semantic field corresponding to the target three-dimensional region.

[0065] Semantic labels refer to the category information assigned to image pixels (or points, polygons) to express "what kind of target this is". For example, pixel-level semantic labels include: buildings, roads, vegetation, sky, vehicles, pedestrians, etc. Target-level semantic labels include: a building = "commercial building", a road = "main road", etc.

[0066] A unified 3D semantic field refers to associating semantic information in 3D space, giving spatial locations, regions, or surfaces category attributes, and achieving a unified expression of "geometry + semantics." For example, in a continuous geometric field, the semantic meaning of a wall region is "building," and the semantic meaning of a road surface region is "road." Thus, given a 3D coordinate or surface, its corresponding semantic category can be retrieved.

[0067] Please refer to Figure 3 , Figure 3 This is a schematic diagram illustrating the process of constructing a unified three-dimensional semantic field in an embodiment of this application. For example... Figure 3 As shown, semantic segmentation is performed on the full-scene image in the processed multi-source data to obtain multiple semantic labels. Based on the device pose, these semantic labels are then mapped to a continuous geometric field through spatial back-projection to obtain a unified 3D semantic field corresponding to the target 3D region, including: S140-1. Based on a pre-built semantic segmentation network, perform semantic recognition on the full-scene images in the processed multi-source data to obtain multiple semantic labels carrying category information.

[0068] The pre-built semantic segmentation network refers to a pre-trained deep learning model that can directly perform pixel-by-pixel semantic classification of the input image. Its input is a full-scene image, and its output is a semantic label map at the same resolution as the input. Each pixel carries a semantic category ID (e.g., 0=background, 1=building, 2=road, 3=vegetation, etc.) and a category confidence score (i.e., probability, such as 0.8). The semantic segmentation network can employ deep learning networks such as DeepLabV3+, Mask2Former, and Fast-SCNN.

[0069] The process of processing full-scene images based on a pre-built semantic segmentation network may include: performing semantic recognition on full-scene images from different perspectives in the processed multi-source data based on the pre-built semantic segmentation network to obtain initial semantic labels and label confidence scores for each region in the full-scene images; selecting multiple initial semantic labels and label confidence scores belonging to the same target region from full-scene images from different perspectives; weighting the confidence scores of the same semantic labels among the multiple initial semantic labels based on the label confidence scores to obtain multiple aggregated semantic labels and their corresponding aggregated label confidence scores; and determining the semantic label corresponding to the target region from the multiple aggregated semantic labels based on the aggregated label confidence scores.

[0070] First, a pre-trained semantic segmentation network is used to perform pixel-by-pixel inference on full-scene images from different perspectives in the processed multi-source data, outputting initial semantic labels (such as buildings, roads, vegetation, etc.) for each region in each image, as well as label confidence scores representing the reliability of classification. Then, based on the solved device pose, pixels projected onto the same target region (such as the facade of the same building or the same road surface) in images from different perspectives are associated, and multiple initial semantic labels belonging to the target region and their corresponding confidence scores are selected.

[0071] Building upon this foundation, for the same target region, labels with the same semantic category are aggregated using a confidence-weighted method to obtain multiple aggregated semantic labels and their combined confidence scores. For example, the confidence scores of each viewpoint belonging to the "architecture" category are accumulated or weighted averaged. Finally, the semantic category with the highest confidence score among the aggregated labels is selected as the final semantic label for the target region. Thus, by introducing multi-view observation and a confidence-weighted mechanism, semantic information observed from multiple perspectives is fused, effectively suppressing interference caused by single-view occlusion and segmentation errors, reducing subsequent 3D semantic conflicts and noise, and ensuring the consistency and accuracy of semantic information in three-dimensional space.

[0072] For example, if a certain floor area is projected in all 5 images, then 5 initial semantic labels and label confidence scores will be collected for that target area. Assume that 3 of the images identify the target area as "building" with confidence scores of 0.91, 0.87, and 0.85 respectively; 1 image identifies it as "vegetation" with a confidence score of 0.6; and 1 image identifies it as "road" with a confidence score of 0.4. Then, by weighting the confidence scores, we can obtain multiple aggregated semantic labels and their corresponding aggregated label confidence scores: "building": 2.63, "vegetation": 0.6, "road": 0.4. Finally, the category with the highest aggregate confidence score is selected as the final semantic label for the target area, namely "building".

[0073] S140-2. Based on the intrinsic and extrinsic parameters of the imaging device in the device pose, establish the projection correspondence between pixels and three-dimensional spatial rays in the full scene image.

[0074] In this process, firstly, the intrinsic parameter matrix (including focal length, principal point coordinates, and distortion coefficients, etc.) and extrinsic parameter matrix (including rotation matrix and translation vector, representing the camera's position and orientation in the world coordinate system) of the imaging device at the acquisition time can be obtained. Subsequently, for each pixel in the full scene image, its two-dimensional pixel coordinates are inversely transformed through the intrinsic parameter matrix, and then back-projected from the image plane to the normalized camera coordinate system to obtain the direction vector corresponding to that pixel.

[0075] Then, the extrinsic parameter matrix is ​​used to transform the direction vector from the camera coordinate system to the world coordinate system. Combined with the camera's optical center position (defined by the extrinsic translation vector), a three-dimensional ray is constructed that originates from the camera's optical center and passes through the spatial point corresponding to the pixel. Mathematically, this ray represents all possible positions of the pixel in three-dimensional physical space, that is, the projection correspondence between the pixel and the three-dimensional spatial ray, providing a geometric basis for the subsequent back-projection of two-dimensional semantic information to the three-dimensional geometric field.

[0076] For example, in a full-scene image, pixel (960, 540) is located at the center of the image. A ray is constructed from the camera's optical center and extends in a specific direction. This ray represents all object points in the real world located directly in front of this monitoring viewpoint. The semantic information of pixel (960, 540) will be restricted to this ray for subsequent geometric matching and filtering.

[0077] S140-3. Select target pixels whose semantic labels belong to non-background categories from the full scene image, and perform dense sampling along the spatial ray direction corresponding to the target pixel in the physical space represented by the continuous geometric field to generate candidate three-dimensional sampling points.

[0078] In this context, target pixels refer to pixels in the entire scene image that do not belong to the "background or ignore" category in the semantic segmentation result and need to participate in 3D semantic mapping. For example, pixels with semantic meanings of "building, road, vegetation, vehicle", but excluding pixels with semantic meanings of "sky, pure background, uncertain", etc.

[0079] Candidate 3D sampling points refer to a number of 3D points taken along the spatial ray corresponding to a pixel within a spatial range defined by a continuous geometric field, according to a certain strategy (such as uniform or layered).

[0080] In some embodiments, semantic-driven spatial sampling can be performed after obtaining the semantic labels of the entire scene image. For example, the entire scene image is first filtered based on the semantic segmentation results, eliminating categories that do not require modeling, such as "sky," "background," or "uncertain," and retaining only target pixels carrying entity semantic information (such as "buildings," "roads," "vegetation," "vehicles," etc.). Subsequently, for each target pixel, the ray equation corresponding to that pixel in three-dimensional space is obtained using its corresponding device pose and camera intrinsic parameters. Then, along the extension direction of the ray, high-density hierarchical sampling or uniform sampling is performed within the physical space represented by continuous geometric locations (e.g., within the scene bounding box determined by sparse three-dimensional point clouds). That is, a three-dimensional point is generated on the ray at preset step lengths (e.g., 5 cm). In this way, each semantically meaningful two-dimensional pixel is transformed into a series of candidate three-dimensional sampling points distributed along the line of sight, thereby providing a sufficient sampling basis for subsequent determination of whether these points fall on the surface of real objects.

[0081] For example, in a panoramic image containing tall buildings, pixel regions representing "buildings" are identified as target pixels. For a specific pixel located at the edge of the rooftop, if the camera pose of the image is known, a 3D ray can be emitted from the camera's optical center, passing through the pixel and pointing towards the rooftop. Assuming the scene height ranges from 0 to 100 meters, a sampling point is generated every 0.05 meters along this ray, starting from the camera position, resulting in thousands of candidate 3D sampling points, such as (10.0, 20.0, 5.0), (10.01, 20.01, 5.05), (10.02, 20.02, 5.10), etc. These points cover the entire spatial range from the camera to the rooftop and beyond, and are used in subsequent steps to filter out points actually located on the rooftop surface using a continuous geometric field.

[0082] S140-4. Based on the surface constraints of the continuous geometric field, select the effective three-dimensional sampling points located on the geometric surface and near-surface region of the scene from the candidate three-dimensional sampling points.

[0083] Among them, effective 3D sampling points refer to points selected from candidate sampling points based on the surface constraints of the continuous geometric field (such as SDF≈0) that are considered to be located (or very close to) the scene's geometric surface and near-surface region.

[0084] In some embodiments, firstly, each candidate 3D sampling point generated along the spatial ray corresponding to the pixel can be input into a trained continuous geometry field construction model (such as an SDF network) to obtain the symbolic distance value (SDF value) corresponding to each candidate point. Since the zero iso-face of the symbolic distance value in the continuous geometry field corresponds to the real geometric surface of the scene, by setting a near-surface threshold (e.g., 0.03m), candidate 3D sampling points whose absolute value of the symbolic distance value is less than or equal to the threshold are selected as valid 3D sampling points located on the geometric surface and near-surface region of the scene. Optionally, the angular relationship between the surface normal and the ray direction can be further combined to exclude sampling points located behind objects or with ambiguous projections. Through this surface constraint screening, a large number of candidate points distributed along the ray can be restricted to a small number of valid points that fit or are very close to the 3D geometric surface, thereby providing reliable spatial anchor points for the accurate mapping of semantic labels from 2D images to 3D geometric fields.

[0085] S140-5. Associate the effective three-dimensional sampling points with their corresponding semantic labels to obtain the unified three-dimensional semantic field corresponding to the target three-dimensional region.

[0086] The steps described above, through spatial backprojection, have filtered out effective 3D sampling points located on the surface and near-surface regions of the scene within a continuous geometric field. These points not only possess precise 3D spatial coordinates but also carry semantic category information of their source pixels (such as "building," "road," or "vegetation"). Therefore, these discrete effective 3D sampling points can be used as "semantic seed points," binding their spatial coordinates to their corresponding semantic category labels for storage. This yields a unified 3D semantic field corresponding to the entire target 3D region, achieving deep fusion of geometric structure and semantic information.

[0087] For example, if 1500 candidate 3D sampling points are generated along the ray corresponding to a certain "building" pixel, and the SDF value is queried for each point, only 4 points have |SDF|≤0.03m and are located on the normal side outside the floor. These 4 points are then used as valid 3D sampling points and associated with the semantic label of the "building".

[0088] S150. Based on a unified three-dimensional semantic field, perform block-based distributed modeling to generate a three-dimensional spatial model corresponding to the target three-dimensional region.

[0089] Block-based distributed modeling refers to dividing a large scene into sub-blocks based on space or complexity, reconstructing them in parallel on different computing resources, and then merging them into a global model to improve efficiency and scalability. For example, densely packed high-rise buildings can be divided into several small blocks (high precision), and open squares or roads can be divided into several large blocks (lighter computation). Then, each block can be reconstructed in parallel on different GPUs (Graphics Processing Units) or computing nodes, and after boundary clipping, they can be stitched together to obtain the entire 3D spatial model.

[0090] In some embodiments, the process of performing block-based distributed modeling based on a unified three-dimensional semantic field to generate a three-dimensional spatial model corresponding to the target three-dimensional region may include: Based on the spatial range and geometric semantic distribution characteristics of a unified 3D semantic field, the target 3D region is divided into multiple computational sub-blocks according to the scene complexity. Each computational sub-block is distributed to different computing units for distributed parallel training to generate a local 3D model corresponding to each computational sub-block. The local 3D models are then clipped and registered to merge and generate a globally consistent 3D spatial model.

[0091] In this process, firstly, the scene complexity is assessed based on the spatial bounding box of a unified 3D semantic field and the geometric semantic distribution characteristics of each location within it. The spatial extent refers to the 3D area covered by the unified 3D semantic field. Geometric semantic distribution characteristics include: which areas are geometrically complex (building complexes, facade details), which areas are semantically dense or important (roads, structures), and which areas are relatively simple (open spaces, water surfaces, simple shapes), etc. Based on this, information such as semantic edges, surface curvature, point density, and number of structures per unit volume can be calculated to quantify scene complexity. For example, a higher point density and a greater number of structures per unit volume indicate that the location contains more elements and the scene is more complex.

[0092] Then, the target 3D region is adaptively divided into multiple computational sub-blocks based on complexity metrics. Regions with complex geometry and dense semantic categories are divided into smaller blocks, while open and simple regions are divided into larger blocks. The expected computational cost of each block is allocated with load balancing as the goal. For example, densely built-up blocks are divided into 50m×50m blocks, open plazas or roads into 200m×200m blocks, and indoor halls into 30m×30m blocks. Each computational sub-block can carry a "semantic label set" for subsequent modeling strategy selection.

[0093] Then, each computational sub-block is distributed to different computing units (such as GPUs or computing nodes) for distributed parallel reconstruction. Based on the geometric field and semantic information within the block, the corresponding reconstruction strategy is selected to generate a local 3D model corresponding to each computational sub-block.

[0094] Finally, the boundaries of each local 3D model are clipped to remove redundant geometry that exceeds the definition range of the sub-block. Based on the continuous geometry and semantic reference between blocks provided by the unified 3D semantic field, registration and stitching are performed to eliminate inconsistencies between blocks and fuse them to obtain a globally consistent 3D spatial model for city-level 3D visualization, analysis and digital twin applications.

[0095] This embodiment first preprocesses and unifies the format of multi-source heterogeneous data to address the differences in coordinate systems, sampling rates, and data structures between different devices and modalities. This lays a solid data foundation for subsequent collaborative computing and improves the system's compatibility and robustness. Secondly, by leveraging feature point extraction and alignment, and joint regional network adjustment, the spatial poses of each acquisition device are accurately restored, generating a high-precision sparse 3D point cloud. This ensures the accuracy of the 3D reconstruction in terms of geometric scale and spatial location, avoiding model distortion caused by accumulated errors. Based on this, a geometric field model is used to fill and smooth the sparse point cloud, transforming discrete point data into continuous and complete geometric surfaces. This effectively eliminates data holes and noise, significantly improving the integrity and realism of the target region's geometric structure.

[0096] Furthermore, this embodiment introduces semantic information. By semantically segmenting the entire scene image and back-projecting semantic labels onto the 3D geometric field using device pose, a "unified 3D semantic field" is successfully constructed. This allows the generated model to not only possess detailed geometric morphology but also endow each spatial location in the scene with clear semantic attributes (such as buildings, roads, vegetation, etc.), achieving a leap from visual to understandable. Finally, the block-based distributed modeling strategy based on the unified 3D semantic field significantly improves the processing efficiency and scalability of large-scale scenes while ensuring the model's detail accuracy. Moreover, the final generated target 3D region model possesses both high-precision geometric structure and rich semantic information, meeting the needs of high-level application scenarios such as digital twins, smart cities, autonomous driving simulation, and engineering monitoring.

[0097] To better implement the above methods, this application also provides a three-dimensional reconstruction device based on a unified semantic field. This device can be integrated into an electronic device, such as a terminal or server. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, or personal computer; the server can be a single server or a server cluster composed of multiple servers.

[0098] For example, in this embodiment, the method of this application embodiment will be described in detail by taking the three-dimensional reconstruction device specifically integrated into the server as an example.

[0099] For example, such as Figure 4As shown, the 3D reconstruction device may include a data processing module 410, a point cloud extraction module 420, a geometric field construction module 430, a semantic embedding module 440, and a distributed reconstruction module 450, as follows: (1) Data processing module 410 is used to acquire multi-source heterogeneous data of the target three-dimensional region collected by multiple devices, and to preprocess the multi-source heterogeneous data to obtain multi-source data with uniform format.

[0100] (2) Point cloud extraction module 420 is used to extract and align feature points from the processed multi-source data, and combine the joint regional network adjustment solution to determine the sparse three-dimensional point cloud corresponding to the device pose and the target three-dimensional region.

[0101] For example, the point cloud extraction module 420 is specifically used to extract feature points from the processed multi-source data based on a preset deep learning model to obtain multi-source feature points to be matched; identify matching feature points indicating the same object from several multi-source feature points to be matched and align them to obtain several aligned multi-source feature points; determine the initial pose of the imaging device based on the projection relationship of the aligned multi-source feature points in the multi-view image and their geometric constraints in the laser point cloud data, and generate an initial three-dimensional point cloud covering the target three-dimensional region; use the point cloud data collected by the laser scanning device as geometric constraints, and use a joint regional network adjustment model to jointly optimize the initial pose and initial three-dimensional point cloud of the imaging device and non-imaging devices to obtain the sparse three-dimensional point cloud corresponding to the device pose and the target three-dimensional region.

[0102] (3) Geometric field construction module 430 is used to fill and smooth sparse three-dimensional point clouds using a preset geometric field construction model in order to obtain the continuous geometric field corresponding to the target three-dimensional region.

[0103] For example, the geometry construction module 430 is specifically used to construct an initial geometry construction model and acquire sample 3D point clouds; perform hierarchical random sampling on the surface of the sample 3D point cloud and within a preset range to generate several training sample points; for each training sample point, calculate the Euclidean distance from the training sample point to the nearest neighbor in the sample 3D point cloud, and use the positive and negative Euclidean distance as the ground truth label based on the spatial location information of the training sample point; input the training sample points into the initial geometry construction model to predict the symbolic distance value of the training sample points; determine the loss function value based on the difference between the symbolic distance value and the ground truth label, and iteratively optimize the model parameters of the initial geometry construction model through the backpropagation algorithm until the model converges to obtain a preset geometry construction model; perform random sampling on the sparse 3D point cloud surface to obtain query sampling points, input the query sampling points into the preset geometry construction model, output the symbolic distance value corresponding to each query sampling point, and extract the isosurface with a symbolic distance value of zero to obtain the continuous geometry field corresponding to the target 3D region.

[0104] (4) Semantic embedding module 440 is used to perform semantic segmentation on the full scene image in the processed multi-source data to obtain multiple semantic labels, and to map the multiple semantic labels to the continuous geometric field through spatial back projection based on the device pose to obtain the unified three-dimensional semantic field corresponding to the target three-dimensional region.

[0105] For example, the semantic embedding module 440 is specifically used to perform semantic recognition on the full-scene image in the processed multi-source data based on a pre-built semantic segmentation network to obtain multiple semantic labels carrying category information; establish the projection correspondence between pixels in the full-scene image and three-dimensional spatial rays based on the intrinsic and extrinsic parameters of the imaging device in the device pose; select target pixels whose semantic labels belong to non-background categories from the full-scene image, and perform dense sampling along the spatial ray direction corresponding to the target pixels in the physical space represented by the continuous geometric field to generate candidate three-dimensional sampling points; select effective three-dimensional sampling points located on the scene geometric surface and near-surface region from the candidate three-dimensional sampling points based on the surface constraints of the continuous geometric field; associate the effective three-dimensional sampling points with their corresponding semantic labels to obtain the unified three-dimensional semantic field corresponding to the target three-dimensional region.

[0106] (4) Distributed reconstruction module 450 is used to perform block-based distributed modeling based on a unified three-dimensional semantic field to generate a three-dimensional spatial model corresponding to the target three-dimensional region.

[0107] For example, the distributed reconstruction module 450 is specifically used to divide the target 3D region into multiple computational sub-blocks based on the spatial range and geometric semantic distribution features of the unified 3D semantic field and the scene complexity; distribute each computational sub-block to different computing units for distributed parallel training to generate local 3D models corresponding to each computational sub-block; and perform boundary clipping and registration stitching on the local 3D models to fuse and generate a globally consistent 3D spatial model.

[0108] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0109] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0110] This application also provides an electronic device, which can be a terminal, a server, or other similar device. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers, etc.

[0111] In some embodiments, the three-dimensional reconstruction device can also be integrated into multiple electronic devices. For example, the three-dimensional reconstruction device can be integrated into multiple servers, and the three-dimensional reconstruction method of this application can be implemented by multiple servers.

[0112] In this embodiment, as Figure 5 As shown, it illustrates a structural schematic diagram of the electronic device involved in the embodiments of this application, specifically: The electronic device may include components such as a processor 510 with one or more processing cores, a memory 520 with one or more computer-readable storage media, a power supply 530, an input module 540, and a communication module 550. Those skilled in the art will understand that... Figure 5 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein: The processor 510 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines, and performs various functions and processes data by running or executing software programs and / or modules stored in the memory 520, and by calling data stored in the memory 520, thereby performing overall detection of the electronic device. In some embodiments, the processor 510 may include one or more processing cores; in some embodiments, the processor 510 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 510.

[0113] The memory 520 can be used to store software programs and modules. The processor 510 executes various functional applications and data processing by running the software programs and modules stored in the memory 520. The memory 520 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 520 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 520 may also include a memory controller to provide the processor 510 with access to the memory 520.

[0114] The electronic device also includes a power supply 530 that supplies power to the various components. In some embodiments, the power supply 530 can be logically connected to the processor 510 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. The power supply 530 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0115] The electronic device may also include an input module 540, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0116] The electronic device may also include a communication module 550. In some embodiments, the communication module 550 may include a wireless module, through which the electronic device can perform short-range wireless transmission, thereby providing users with wireless broadband internet access. For example, the communication module 550 can be used to help users send and receive emails, browse web pages, and access streaming media.

[0117] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 510 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 520 according to the following instructions, and the processor 510 runs the applications stored in the memory 520, thereby realizing the various functions in the above-mentioned three-dimensional reconstruction method.

[0118] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0119] To this end, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the three-dimensional reconstruction methods based on a unified semantic field provided in embodiments of this application.

[0120] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0121] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the three-dimensional reconstruction method provided in the above embodiments.

[0122] Since the instructions stored in the storage medium can execute the steps of any of the three-dimensional reconstruction methods based on a unified semantic field provided in the embodiments of this application, the beneficial effects that any of the three-dimensional reconstruction methods based on a unified semantic field provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0123] The foregoing has provided a detailed description of a three-dimensional reconstruction method, apparatus, device, and medium based on a unified semantic field provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A three-dimensional reconstruction method based on a unified semantic field, characterized in that, The method includes: Acquire multi-source heterogeneous data of a target three-dimensional region collected by multiple devices, and preprocess the multi-source heterogeneous data to obtain processed multi-source data with a unified format; Feature points are extracted and aligned from the processed multi-source data, and combined with joint regional network adjustment calculations to determine the device pose and the sparse 3D point cloud corresponding to the target 3D region. A pre-set geometric field construction model is used to fill and smooth the sparse three-dimensional point cloud to obtain the continuous geometric field corresponding to the target three-dimensional region. Semantic segmentation is performed on the full-scene image in the processed multi-source data to obtain multiple semantic labels. Based on the device pose, the multiple semantic labels are mapped to the continuous geometric field through spatial back projection to obtain a unified three-dimensional semantic field corresponding to the target three-dimensional region. Based on the spatial range and geometric semantic distribution characteristics of the unified three-dimensional semantic field, the target three-dimensional region is divided into multiple computational sub-blocks according to the scene complexity; each computational sub-block is distributed to different computing units for distributed parallel training to generate a local three-dimensional model corresponding to each computational sub-block; the local three-dimensional models are subjected to boundary clipping and registration stitching to fuse and generate a globally consistent three-dimensional spatial model.

2. The three-dimensional reconstruction method based on a unified semantic field as described in claim 1, characterized in that, The step of extracting and aligning feature points from the processed multi-source data, and combining this with joint regional network adjustment to determine the device pose and the sparse 3D point cloud corresponding to the target 3D region, includes: Based on a pre-set deep learning model, feature points are extracted from the processed multi-source data to obtain multi-source feature points to be matched. Identify and align matching feature points indicating the same object from a plurality of multi-source feature points to be matched, so as to obtain a plurality of aligned multi-source feature points; Based on the projection relationship of the aligned multi-source feature points in the multi-view image and their geometric constraints in the laser point cloud data, the initial pose of the imaging device is determined, and an initial three-dimensional point cloud covering the target three-dimensional region is generated. Using point cloud data collected by laser scanning equipment as geometric constraints, a joint regional network adjustment model is used to jointly optimize the initial pose and initial 3D point cloud of the imaging equipment and non-imaging equipment, so as to obtain the sparse 3D point cloud corresponding to the device pose and the target 3D region.

3. The three-dimensional reconstruction method based on a unified semantic field as described in claim 2, characterized in that, Based on the projection relationship of the aligned multi-source feature points in the multi-view image and their geometric constraints in the laser point cloud data, the initial pose of the imaging device is determined, and an initial 3D point cloud covering the target 3D region is generated, including: Based on the aligned multi-source feature points in multi-view images of different data types, the intrinsic parameters of the imaging device are determined. The pose of the imaging device is determined based on the projection position of the aligned multi-source feature points in the multi-view image, the geometric constraints of the aligned multi-source feature points in the laser point cloud data, and the intrinsic parameters. Using the pose of any of the imaging devices as the initial origin, and based on the geometric intersection relationship between the multi-view images, triangulation is performed on the aligned multi-source feature points to recover the coordinates of the aligned multi-source feature points in three-dimensional space, thereby generating an initial three-dimensional point cloud covering the target three-dimensional region.

4. The three-dimensional reconstruction method based on a unified semantic field as described in claim 1, characterized in that, A pre-defined geometric field model is used to fill and smooth the sparse 3D point cloud to obtain the continuous geometric field corresponding to the target 3D region, including: Construct an initial geometric field model and obtain sample 3D point clouds; Layered random sampling is performed on the surface of the sample 3D point cloud and within a preset range to generate several training sample points. For each training sample point, calculate the Euclidean distance from the training sample point to the nearest neighbor point in the sample 3D point cloud, and use the Euclidean distance with positive and negative signs as the ground truth label of the training sample point based on the spatial location information of the training sample point. The training sample points are input into the initial geometric field construction model to predict the symbolic distance value of the training sample points; The loss function value is determined based on the difference between the symbolic distance value and the truth label, and the model parameters of the initial geometric field construction model are iteratively optimized through the backpropagation algorithm until the model converges, thus obtaining the preset geometric field construction model. Random sampling is performed on the sparse three-dimensional point cloud surface and within a preset range to obtain query sampling points. The query sampling points are then input into the preset geometric field construction model. The symbolic distance value corresponding to each query sampling point is output, and the isosurface with a symbolic distance value of zero is extracted to obtain the continuous geometric field corresponding to the target three-dimensional region.

5. The three-dimensional reconstruction method based on a unified semantic field as described in claim 1, characterized in that, Semantic segmentation is performed on the full-scene image in the processed multi-source data to obtain multiple semantic labels. Based on the device pose, the multiple semantic labels are mapped to the continuous geometric field through spatial back projection to obtain a unified three-dimensional semantic field corresponding to the target three-dimensional region, including: Based on a pre-built semantic segmentation network, semantic recognition is performed on the full-scene images in the processed multi-source data to obtain multiple semantic labels carrying category information. Based on the intrinsic and extrinsic parameters of the imaging device in the device pose, establish the projection correspondence between pixels and three-dimensional spatial rays in the full scene image; From the full scene image, target pixels whose semantic labels belong to the non-background category are selected, and dense sampling is performed along the spatial ray direction corresponding to the target pixel in the physical space represented by the continuous geometric field to generate candidate three-dimensional sampling points. Based on the surface constraints of the continuous geometric field, effective three-dimensional sampling points located on the scene geometric surface and near-surface region are selected from the candidate three-dimensional sampling points. By associating the effective 3D sampling points with their corresponding semantic labels, a unified 3D semantic field corresponding to the target 3D region is obtained.

6. The three-dimensional reconstruction method based on a unified semantic field as described in claim 5, characterized in that, Based on a pre-built semantic segmentation network, semantic recognition is performed on the full-scene images in the processed multi-source data to obtain multiple semantic labels carrying category information, including: Based on a pre-built semantic segmentation network, semantic recognition is performed on full-scene images from different perspectives in the processed multi-source data to obtain the initial semantic labels and label confidence of each region in the full-scene image. Multiple initial semantic labels and label confidence scores belonging to the same target region are selected from the full-scene images from different perspectives; Based on the tag confidence, the confidence of the same semantic tags in the multiple initial semantic tags is weighted to obtain multiple aggregated semantic tags and their corresponding aggregated tag confidence; Based on the confidence level of the aggregated labels, the semantic label corresponding to the target region is determined from multiple aggregated semantic labels.

7. A three-dimensional reconstruction device based on a unified semantic field, characterized in that, The device includes: The data processing module is used to acquire multi-source heterogeneous data of a target three-dimensional region collected by multiple devices, and to preprocess the multi-source heterogeneous data to obtain processed multi-source data with a unified format. The point cloud extraction module is used to extract and align feature points from the processed multi-source data, and combine it with joint regional network adjustment calculation to determine the device pose and the sparse three-dimensional point cloud corresponding to the target three-dimensional region. The geometry field construction module is used to fill and smooth the sparse three-dimensional point cloud using a preset geometry field construction model to obtain the continuous geometry field corresponding to the target three-dimensional region. The semantic embedding module is used to perform semantic segmentation on the full-scene image in the processed multi-source data to obtain multiple semantic labels, and to map the multiple semantic labels to the continuous geometric field through spatial back projection based on the device pose to obtain a unified three-dimensional semantic field corresponding to the target three-dimensional region. The distributed reconstruction module is used to divide the target 3D region into multiple computational sub-blocks based on the spatial range and geometric semantic distribution features of the unified 3D semantic field and the scene complexity; distribute each computational sub-block to different computing units for distributed parallel training to generate local 3D models corresponding to each computational sub-block; and perform boundary clipping and registration stitching on the local 3D models to fuse and generate a globally consistent 3D spatial model.

8. An electronic device, characterized in that, The method includes a processor and a memory, the memory storing multiple instructions; the processor loads instructions from the memory to perform steps in a three-dimensional reconstruction method based on a unified semantic field as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform steps in a three-dimensional reconstruction method based on a unified semantic field as described in any one of claims 1-6.

Images