Construction method and system of real three-dimensional map based on fusion of point cloud data and panoramic image

Abstract

This invention belongs to the field of 3D modeling technology. It provides a method and system for constructing a real-scene 3D map based on the fusion of point cloud data and panoramic imagery. The method includes: obtaining candidate points based on traditional matching; outputting a set of failed anchor points according to reprojection error and geometric consistency during matching; recording the local features of each failed anchor point; using the local features of the failed anchor points as input to train a random forest network; outputting a set of cause labels for failed anchor points; simultaneously using the local features and cause labels of the failed anchor points as input to train a small MLP to output the recoverable probability of anchor point retrieval; subsequently calculating and dynamically adjusting the anchor point retrieval screening threshold based on the current scene's average point cloud density, texture intensity, and recoverable probability; for anchor points exceeding the retrieval screening threshold, applying a targeted enhancement algorithm according to the cause label; and applying an alternative strategy for the remaining anchor points; finally, outputting the anchor set and reconstructing the 3D map.

Description

Technical Field

[0001] This invention belongs to the field of 3D modeling technology, specifically a method and system for constructing real-scene 3D maps based on the fusion of point cloud data and panoramic images. Background Technology

[0002] With the continuous advancement of real-scene 3D construction, real-scene 3D maps have been widely applied in various scenarios such as urban and rural planning, traffic surveying, scenic area operation and maintenance, emergency disaster prevention and control, and park management. Currently, 3D modeling mainly relies on the fusion of laser point cloud and panoramic image data to achieve model construction. Point clouds have accurate geometric positioning and can accurately restore the structure of the site space, but they are prone to problems such as data sparsity, local missing data, and weak resistance to noise interference. Panoramic images have rich texture details and wide coverage, but they are easily affected by lighting, occlusion, and weak textures, resulting in feature matching failures and a large number of failed matching anchor points. At the same time, the existing fusion modeling methods are relatively simple, and failed anchor points are usually directly removed, which can easily lead to a waste of data resources. Furthermore, the use of fixed thresholds to filter points cannot adapt to the point cloud density and texture environment of different sites. Existing technologies cannot distinguish whether anchor point failure is due to point cloud problems, image problems, or a combination of problems, and lack differentiated repair methods. This can easily lead to problems such as holes, missing details, and misaligned stitching in the 3D model, resulting in insufficient modeling accuracy and scene integrity, making it difficult to meet the needs of constructing high-precision real-scene 3D maps for various sites.

[0003] To this end, the present invention provides a method and system for constructing a real-scene 3D map based on the fusion of point cloud data and panoramic imagery. Summary of the Invention

[0004] In order to overcome the shortcomings of the prior art, at least one technical problem raised in the background art is solved.

[0005] The technical solution adopted by this invention to solve its technical problem is: a method for constructing a real-scene 3D map based on the fusion of point cloud data and panoramic imagery, comprising the following steps: Step 1: Obtain candidate points based on traditional matching, output a set of failed anchor points according to the reprojection error and geometric consistency in the matching, and record the local features of each failed anchor point; Step 2: Use the local features of the failed anchor points as input to train a random forest network and output a set of labels for the reasons of the failed anchor points; Among them, the set of labels for reasons for failed anchor points includes point cloud-side problems, image-side problems, and point cloud-image-side composite problems; Step 3: Using the local features and cause label set of the failed anchor points as input, train a small MLP to output the recoverable probability of anchor point retrieval; Step 4: Calculate and dynamically adjust the anchor point retrieval screening threshold based on the current scene's average point cloud density, texture intensity, and recoverability probability; Step 5: For anchor points that exceed the retrieval screening threshold, apply the targeted enhancement algorithm according to the cause label; for the remaining anchor points, apply the replacement strategy; finally, output the anchor point set and reconstruct the 3D map.

[0006] Furthermore: In step one, the specific method for outputting the set of failed anchor points based on the reprojection error and geometric consistency during matching is as follows: Calculate the pixel distance between the two-dimensional feature points and the projected points of the three-dimensional points after reprojection onto the image plane in each candidate pair of corresponding points. This distance is used as the reprojection error. If the reprojection error is greater than a preset threshold, it is directly marked as a failed anchor point. If the reprojection error is less than or equal to the preset threshold, a geometric consistency judgment is performed. If any one of the three judgment conditions—epidial geometric constraint, local rigidity assumption, and normal vector consistency—is not satisfied, it is marked as a failed anchor point. If all three conditions are satisfied, it is marked as a non-failed anchor point.

[0007] Furthermore: In the geometric consistency judgment, the epipolar geometric constraint is achieved by calculating the Sampson distance between the projection points of the 3D points on the two frames of images and comparing it with a preset threshold; the local rigidity assumption is achieved by calculating the 3D nearest neighbor distance between the 3D point and its nearest neighbor point, and the image nearest neighbor distance between the corresponding image point and its nearest matching point, calculating the ratio of the absolute difference between the 3D nearest neighbor distance and the image nearest neighbor distance, and comparing it with a preset threshold to achieve the local rigidity assumption judgment; the normal vector consistency is achieved by calculating the absolute value of the cosine of the angle between the 3D point cloud normal vector projected onto the image plane and the image gradient, and comparing it with a preset threshold.

[0008] Further: In step one, the local features of each failed anchor point are recorded, including point cloud-side features, image-side features, and composite features; the point cloud-side features are extracted from the point cloud density, local curvature, and flatness within a preset radius neighborhood centered on the 3D point in the failed anchor point; the image-side features are extracted from the average gradient magnitude and local gray-level entropy within a preset window centered on the 2D feature point in the failed anchor point; the composite features include projection consistency error and neighborhood matching success rate.

[0009] Furthermore: The process of training the random forest network in step two is as follows: Construct a failed anchor point sample dataset containing input feature vectors and output labels. The input feature vectors contain the local features of the failed anchor points recorded in step one, and the output labels are enumerated values ​​representing three categories: point cloud-side problems, image-side problems, or combined problems. Divide the sample dataset into training, validation, and test sets proportionally, and perform missing value imputation and standardization preprocessing. After configuring the random forest parameters, supervised training is performed using the training set, and each tree is randomly split by sampling some samples and some features.

[0010] Furthermore: The process of finding recoverable probabilities by training the output anchor point of the small multilayer perceptron in step three is as follows: Construct a sample dataset containing input feature vectors and output labels. The input feature vectors are composed of local features of failed anchor points and cause labels, and the output labels are binary labels indicating whether the anchor point can be recovered. Divide the sample dataset into training, validation, and test sets according to the proportions, and perform missing value imputation and standardization preprocessing. Build a lightweight multilayer perceptron architecture containing several fully connected layers and an output layer. The output layer uses an activation function to ensure that the probability value is within a preset range. Use mean squared error as the loss function for iterative training until convergence.

[0011] Furthermore: The method for obtaining the average point cloud density of the current scene in step four is as follows: The calculation boundary is defined by expanding outward from the cluster area of ​​failed anchor points. Noise points and discrete isolated points are removed from the original point cloud. The total number of effective point clouds and spatial volume within the region are statistically calculated, and the average point cloud density is obtained by calculating the ratio. The texture intensity is obtained by projecting the spatial coordinates of failed anchor points onto the panoramic image and extracting pixel blocks. The average grayscale gradient and texture contrast of each block are calculated. After normalization, the mean value is taken as the texture intensity of the block. After removing abnormal blocks, the average texture intensity of all blocks is calculated to obtain the average texture intensity of the scene.

[0012] Furthermore: The method for dynamically adjusting the anchor point to retrieve the filtering threshold in step four is as follows: The point cloud density quality factor is calculated based on the ratio of the scene's average point cloud density to the preset reference density. The normalized scene average texture intensity is taken as the texture intensity quality factor, and the smaller value between the point cloud density quality factor and the texture intensity quality factor is taken as the dynamic threshold.

[0013] Furthermore: The specific method for invoking the targeted enhancement algorithm according to the cause label in step five is as follows: When the cause label is a point cloud-side problem, the point cloud-guided interpolation enhancement algorithm is invoked; when the cause label is an image-side problem, the local texture sharpening re-matching algorithm is invoked; when the cause label is a composite problem, both the point cloud-guided interpolation enhancement algorithm and the local texture sharpening re-matching algorithm are invoked simultaneously; for the remaining anchor points, virtual anchor points are generated using an alternative strategy; finally, the non-failed anchor points, the anchor points retrieved through targeted enhancement, and the virtual anchor points generated by the alternative strategy are gathered into the final anchor point set.

[0014] A real-scene 3D map construction system based on the fusion of point cloud data and panoramic imagery includes the following modules: Failed anchor point location and extraction module: Based on traditional matching, candidate points of the same name are obtained. The set of failed anchor points is output according to the reprojection error and geometric consistency in the matching, and the local features of each failed anchor point are recorded. Failure Cause Classification Output Module: Takes the local features of the failure anchor points as input, trains a random forest network, and outputs a set of failure anchor point cause labels; Among them, the set of labels for reasons for failed anchor points includes point cloud-side problems, image-side problems, and point cloud-image-side composite problems; Recovery probability prediction and evaluation module: Takes the local features and cause label set of the failed anchor point as input, trains a small MLP to output the recoverability probability of the anchor point; Adaptive threshold dynamic calculation module: Calculates and dynamically adjusts the anchor point retrieval screening threshold based on the current scene's average point cloud density, texture intensity, and the probability of recovery. The difference repair 3D reconstruction module: For anchor points that exceed the retrieval screening threshold, a targeted enhancement algorithm is called according to the cause label, and the alternative strategy is called for the remaining anchor points. Finally, the anchor point set is output and the 3D map is reconstructed.

[0015] The beneficial effects of this invention are as follows: This invention fully utilizes the local environmental information surrounding failed anchor points, accurately distinguishes between three types of fault causes—point cloud side, image side, and composite—through a random forest model, and introduces a multilayer perceptron to quantitatively evaluate the repair feasibility of each anchor point, providing a refined basis for subsequent handling strategies based on individual differences. Simultaneously, this invention overcomes the limitations of traditional fixed threshold screening schemes by introducing a scene-adaptive dynamic threshold calculation mechanism, enabling the screening criteria to automatically adjust with changes in point cloud density and texture intensity, balancing recall and precision in different quality scenarios. Furthermore, this invention implements differentiated repair strategies based on fault type and recovery probability, employing targeted enhancement algorithms to accurately retrieve high-probability anchor points, and using substitution strategies to generate virtual anchor points to fill data gaps for low-probability anchor points. This maximizes the recovery of failed anchor point data resources that would otherwise be discarded, effectively reducing holes and splicing misalignments in the 3D model, and significantly improving the geometric accuracy and texture integrity of the real-world 3D map. Attached Figure Description

[0016] The invention will now be further described with reference to the accompanying drawings.

[0017] Figure 1 This is a flowchart of the steps in the method for constructing a real-scene 3D map based on the fusion of point cloud data and panoramic imagery according to an embodiment of the present invention. Figure 2 This is a flowchart of a real-scene 3D map construction system based on the fusion of point cloud data and panoramic images, as described in this embodiment of the invention. Detailed Implementation

[0018] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0019] Example 1: Please refer to Figure 1As shown in the embodiment of the present invention, the method for constructing a real-scene 3D map based on the fusion of point cloud data and panoramic imagery includes the following steps: Step 1: Obtain candidate points based on traditional matching, output a set of failed anchor points according to the reprojection error and geometric consistency in the matching, and record the local features of each failed anchor point; In step one, the corresponding point candidate means: a point in the same real-world site and at the same physical location, that is, it has coordinates in the 3D point cloud and a corresponding pixel position in the panoramic image. In step one, the methods for obtaining candidate points with the same name based on traditional matching include, but are not limited to: A1 uses feature extraction algorithms (such as classic algorithms like SIFT, SURF, and ORB) to detect feature points (corners, spots, etc.) on panoramic images and calculates a descriptor (such as a 128-dimensional SIFT vector) for each feature point. A2 uses known camera intrinsic parameters (focal length, distortion parameters) and camera extrinsic parameters (rotation matrix, translation vector) relative to the point cloud coordinate system to project each 3D point in the point cloud onto the panoramic image to obtain its corresponding pixel coordinates. A3. For each image pixel obtained from projection, search for the existence of a two-dimensional feature point in the neighborhood. If it exists, form a "three-dimensional-two-dimensional candidate pair" with the three-dimensional point and the two-dimensional feature point. Calculate the Euclidean distance between the descriptor of the two-dimensional feature point and the description of the local image patch at the corresponding position of the three-dimensional point, and retain candidate pairs with an Euclidean distance lower than a preset threshold (such as 0.7) as candidates for corresponding points; In step one, the process of outputting the set of failed anchor points based on the reprojection error and geometric consistency during matching is as follows: For each candidate pair, calculate the reprojection error, which is the pixel distance between the two-dimensional feature point and the projection point in the candidate pair. The projection points are obtained by reprojecting the 3D points in the candidate pair onto the image plane using the camera pose. If the reprojection error is greater than the preset threshold, the candidate pair is marked as a failed anchor point; otherwise, if the reprojection error is less than or equal to the preset threshold, the geometric consistency judgment is entered. The specific process of geometric consistency determination is as follows: B1, epipolar geometry constraint: For the projection points of a 3D point onto two frames of images, calculate the Sampson distance; Set the pass condition: Sampson distance is less than 3 pixels; B2, Local stiffness assumption: In the point cloud, for a 3D point and its three nearest neighbors (e.g., Q1, Q2, Q3), on the holographic image, the image point corresponding to the 3D point and the matched three nearest neighbors (q1, q2, q3) are taken. Calculate the distance from a 3D point to its three nearest neighbors (e.g., Q1, Q2, Q3) to obtain the 3D nearest neighbor distance di, where i represents the nearest neighbor number of the 3D point; Calculate the distance from the image point to the three nearest neighbors (q1, q2, q3) to obtain the image nearest neighbor distance Li, where i represents the nearest neighbor number of the image point; Calculate the ratio of the absolute difference between the three-dimensional nearest neighbor distance di and the image nearest neighbor distance Li, i.e.: |di-Li| / di; The pass condition is set as follows: all absolute difference ratios are less than 0.1; B3, Normal Vector Consistency: The point cloud normal vector n of the three-dimensional points pc The normal vector n obtained by projecting onto the image plane 2d Image gradient I calculates the position of image points in the image and obtains the point cloud normal vector n of the 3D points. 2d and image gradient The absolute value of the cosine of the angle between I and I: The pass conditions are set as follows: absolute cosine value < 0.2 and image gradient > 5; If the polar geometry constraint, the local rigidity assumption, and the pass condition of normal vector consistency are all satisfied, then the candidate pair is marked as a non-failed anchor point; otherwise, if any one of them fails, the candidate pair is marked as a failed anchor point. In step one, the local features of each failed anchor point are recorded, specifically including: point cloud side features (within a neighborhood of radius r (5 * average point spacing of the point cloud) centered on the 3D point in the failed anchor point), image side features (within a window w × w (value 21 pixels) centered on the 2D feature point in the failed anchor point), and composite features. Point cloud side features include, but are not limited to: point cloud density ( ), local curvature ( and flatness ( Image-side features include, but are not limited to: average gradient magnitude ( ) etc. ), local gray-level entropy ( Composite features include, but are not limited to: projection consistency error and ( ) and neighborhood matching success rate, that is, the success rate of other candidate pairs around the failed anchor point; Understandably, the significance of step one is as follows: First, using existing mature visual matching and point cloud matching algorithms, pairing and matching are achieved between various feature points. From this, effective anchor points with stable matching status and good alignment are selected. At the same time, failed points with large matching deviations and various anomalies are accurately identified and uniformly collected into a failed anchor point dataset. On this basis, multiple local environmental and feature information around each failed anchor point are collected simultaneously and used as basic raw data sources. This facilitates in-depth analysis of the specific causes of anchor point matching failures and provides sufficient data support for determining whether failed anchor points can be repaired and recovered. Step 2: Use the local features of the failed anchor points as input to train a random forest network and output a set of labels for the reasons of the failed anchor points; In step two, the process of training the random forest network using the local features of the failed anchor points as input is as follows: C1, Dataset Construction: Collect a large number of failure anchor point samples with known cause labels from historical or simulation data. Each sample contains: The input feature vector specifically includes the local features of the failed anchor points recorded in step one (such as point cloud density, local curvature, flatness, average gradient magnitude, local gray-level entropy, projection error, etc.). Output labels, specifically the known cause labels for failed anchor point samples (enumerated values: 0 = point cloud side problem, 1 = image side problem, 2 = combined problem). C2, Dataset Partitioning: Divide the failed anchor sample dataset into training set, validation set, and test set according to a ratio of 7:1.5:1.5; C3, Feature Preprocessing: Preprocessing the dataset, including: median imputation of missing values, Z-score standardization, etc. C4, Random Forest Network Model Configuration, sets the number of decision trees to 100, the maximum depth to 20, constrains the model hierarchy structure through hard truncation, and specifies that the minimum number of samples for a node to split is 5; C5, Supervised training of the random forest using the training set: For each tree, a subset of samples and features (such as...) are randomly selected. (where d is the total number of features), select the optimal splitting feature and threshold based on Gini impurity or information gain, and repeat the splitting until the maximum depth is reached or the number of node samples is lower than the threshold. C6, after training, yields a random forest network; In step two, the method for outputting the failure anchor point reason tag set is as follows: For each failed anchor point, the local features of the failed anchor point are used as the input of the trained random forest model, and the category with the highest probability is taken as the reason label of the failed anchor point. Summarize the reason labels for all failed anchor points to obtain the set of reason labels for failed anchor points (3D one-hot encoding). ); Understandably, the significance of step two lies in: using the trained random forest model to intelligently classify the failure causes of all failed anchor points, accurately dividing them into three categories: point cloud-side failures, image-side failures, and composite failures that combine both. This enables precise location and tracing of failure problems, laying the groundwork for matching corresponding enhancement and repair algorithms and developing differentiated treatment plans according to different failure types. It also provides accurate category reference data for the recoverable probability prediction model, making the goals of subsequent anchor point repair work clearer and more targeted. Step 3: Using the local features and cause label set of the failed anchor points as input, train a small MLP to output the recoverable probability of anchor point retrieval; In step three, the process of training the small MLP output anchor point to retrieve the recoverable probability is as follows: D1, Input Data: The input data specifically includes the local features of the failed anchor points and the reason labels; D2, Sample Dataset Construction: Collect a large number of failed anchor point samples with known retrieval results from historical or simulation data. Each sample contains: Input feature vector x: It is composed of two parts: local features of the failure anchor point and cause label, and is a floating-point vector. Specifically, it is expressed as follows: Among them, R 9 It is a dimensional representation (if the local feature is 6-dimensional, plus 3-dimensional one-hot encoding, the total is 9-dimensional). Output label y: a binary label (0 or 1) used to indicate whether the failed anchor can be successfully retrieved; D3, Dataset Partitioning: The failed anchor sample dataset (containing input feature vector x and output label y) is split into three subsets: training set, validation set, and test set according to 70%:15%:15%; D4, Feature preprocessing: Missing values ​​are filled with the median (the median of local features at failed anchors over the entire training dataset), and continuous features are standardized using Z-score. D5: Build a lightweight multilayer perceptron architecture and adopt a lightweight multilayer perceptron set. The specific configuration is shown in Table 1 below. Table 1: Lightweight Multilayer Perception Set Configuration; The network depth does not exceed 4 layers, and the number of parameters is controlled within ~10. 3In terms of magnitude, the Sigmoid (activation function) output layer ensures that the probability value is in the [0-1] range, while Dropout (a neural network regularization technique) is used for regularization to improve generalization ability; D5, Training process of a small MLP model: Hyperparameter settings: Limit the core hyperparameters of model training, with the learning rate ranging from 0.001 to 0.01, the number of iterations from 50 to 100, and the batch size from 16 to 32. At the same time, an early stopping strategy is introduced, which terminates the training process when the validation set loss does not decrease for 5 consecutive iterations. Loss function selection: Mean squared error (MSE) is selected as the model loss function, and the weight parameters of each layer of the model are iteratively updated according to the backpropagation algorithm; Iterative training and parameter optimization: The joint features of the training set are input into the MLP model. First, the recoverable probability prediction value (a number between 0 and 1 output by Sigmoid) is calculated through forward propagation. It is compared with the true label and the loss error is calculated through mean squared error (MSE). Then, the neuron weights and bias parameters are adjusted through the backpropagation algorithm. Each round of iterative training is evaluated using a validation set. Model convergence determination: If the fluctuation range of the training set loss error and the validation set loss error does not exceed 1% of the initial loss value (the loss error calculated after the initial parameters are used to predict the training set) within 10 consecutive rounds, and the validation set prediction error meets the preset threshold (≤5%), then iterative training is stopped. D6, the method for finding the recoverable probability of the output anchor point is as follows: input the feature vector x into the trained small MLP model for forward propagation, and the scalar value output by the model is the recoverable probability p of that anchor point. rec ∈[0,1]; Understandably, the significance of step three lies in the following: based on the failure cause classification results clearly defined in step two, a lightweight multilayer perceptron model is further introduced to conduct a quantitative assessment of the recoverability of each failure anchor point. By jointly inputting local environmental features and cause labels, the output is a continuous probability value between zero and one, thereby transforming the original black-and-white binary classification problem into a probability assessment problem with confidence representation. This provides a refined basis for individual differences in the setting of subsequent dynamic thresholds, avoiding a one-size-fits-all approach to all anchor points. Step 4: Calculate and dynamically adjust the anchor point retrieval screening threshold based on the current scene's average point cloud density, texture intensity, and recoverability probability; In step four, the average point cloud density and texture intensity of the current scene are obtained as follows: E1, Average Point Cloud Density Acquisition: Taking the clustering range of failed anchor points as the core, extend outward by 5 to 10 centimeters to define the calculation boundary. Remove noise points and isolated discrete points from the original point cloud, retaining only valid point cloud data. Statistically calculate the total number of valid point clouds N and the spatial volume V within the measurement area. Calculate the scene's average point cloud density p (unit: points / m³) using the following formula: Then, the calculated average point cloud density value p is bound to the scene and stored. E2, Texture Intensity Acquisition: Map the spatial coordinates of each failed anchor point to the panoramic image of the scene through projection transformation, extract (32×32 or 64×64) pixel blocks, and for each block: The Sobel operator is used to calculate the gray-level gradient of a single pixel, and the arithmetic mean of the gray-level gradients of all pixels is calculated to obtain the overall gray-level gradient. The region texture contrast is calculated using the grayscale variance formula; After normalizing the overall grayscale gradient and regional texture contrast to [0,1], the mean value is taken as the texture intensity of the block. After removing abnormal blocks (the image is too bright or too dark), the arithmetic mean of the texture intensity of all blocks is calculated to obtain the average texture intensity Z of the scene. In step four, the process of calculating and dynamically adjusting the anchor point retrieval screening threshold is as follows: D1 defines the point cloud density quality factor: Where ρ ref For a preset reference density (e.g., 100 particles / m³), when p < ρ ref When p ≥ ρ, the quality factor decreases linearly with decreasing density; when p ≥ ρ ref At that time, the quality factor saturates to 1; The texture intensity quality factor is directly taken as the normalized scene average texture intensity Z: Dynamic threshold T rec Defined as the minimum of the two: ; Understandably, the significance of step four lies in: fully taking into account the significant differences in point cloud distribution density and image texture richness in different scenes, introducing a scene-adaptive dynamic threshold calculation mechanism, and obtaining a recall screening threshold by fusing point cloud density quality factor and texture intensity quality factor, so that the screening criteria can be automatically adjusted according to changes in scene conditions, strictly screening in high-quality scenes to ensure accuracy, and appropriately relaxing in low-quality scenes to avoid missed detections, thereby breaking through the limitation of poor adaptability of traditional fixed threshold schemes to different sites; Step 5: For anchor points that exceed the retrieval screening threshold, apply the targeted enhancement algorithm according to the cause label; for the remaining anchor points, apply the replacement strategy; finally, output the anchor point set and reconstruct the 3D map. Among them, the targeted enhancement algorithms include point cloud guided interpolation and local texture sharpening rematch, while the alternative strategies are planar hypothetical interpolation and depth completion network to generate virtual anchor points; In step five, the remaining anchor points are those that are lower than or equal to the retrieval screening threshold; In step five, the targeted enhancement algorithm is invoked according to the cause label as follows: The random forest network outputs a cause label of 0 (point cloud side problem), and the system calls the point cloud guided interpolation enhancement algorithm. The random forest network outputs a cause label of 1 (affecting side issues), and the system calls the local texture sharpening rematch algorithm. The random forest network outputs a cause label of 2 (composite problem), and the system calls the point cloud guided interpolation + local texture sharpening rematching algorithm; In step five, the remaining anchor points apply an alternative strategy, and the final process of outputting the anchor point set and reconstructing the 3D map is as follows: The alternative strategy does not distinguish between cause labels and uses a low-cost method to generate virtual anchors: Centered on the failed anchor point in three dimensions, with a radius of... ( Define a local neighborhood for the average point spacing of the point cloud; perform principal component analysis (PCA) on successfully matched points within the neighborhood to calculate the three eigenvalues ​​of the covariance matrix. ( This represents the direction in which the point cloud is most dispersed, i.e., the direction in which the relative change in data is greatest. The direction of secondary dispersion of the point cloud is perpendicular to the first direction. The flatness is defined as the direction in which the point cloud is most concentrated (i.e., the direction with the least relative change in data). : like If the plane fitting RMSE ≤ 0.5cm, then the region within a certain spatial range centered on the failed anchor point is determined to be approximately a plane, and the system calls the plane hypothesis interpolation algorithm; otherwise, it is determined to be a structurally complex region, and the depth completion network algorithm is called. Based on the final set of output anchor points (including non-failed anchor points, targeted enhanced retrieved anchor points, and virtual anchor points generated by alternative strategies), combined with camera pose and panoramic images, a dense color point cloud is generated through multi-view stereo matching, and then a three-dimensional mesh model is constructed through Poisson surface reconstruction. Finally, a real-world three-dimensional map is output through texture mapping. Understandably, the significance of step five lies in implementing differentiated anchor point repair strategies based on the cause classification results of step two and the dynamic threshold judgment results of step four. For anchor points with high recovery probability, targeted enhancement algorithms are used to achieve accurate retrieval. For anchor points with low recovery probability, alternative strategies are called to generate virtual anchor points to fill data gaps. Finally, all repaired anchor points are collected and output to participate in 3D reconstruction. This not only maximizes the recovery of failed anchor point data resources that would otherwise be directly discarded, but also ensures the efficiency of the repair process and the accuracy of the reconstruction results through the classification and disposal mechanism.

[0020] Example 2: Based on the same inventive concept as the method for constructing a real-scene 3D map based on the fusion of point cloud data and panoramic imagery in Example 1 above, such as... Figure 2 As shown, this application provides a real-scene 3D map construction system based on the fusion of point cloud data and panoramic imagery, wherein the system specifically includes the following modules: Failed anchor point location and extraction module: Based on traditional matching, candidate points of the same name are obtained. The set of failed anchor points is output according to the reprojection error and geometric consistency in the matching, and the local features of each failed anchor point are recorded. Failure Cause Classification Output Module: Takes the local features of the failure anchor points as input, trains a random forest network, and outputs a set of failure anchor point cause labels; Among them, the set of labels for reasons for failed anchor points includes point cloud-side problems, image-side problems, and point cloud-image-side composite problems; Recovery probability prediction and evaluation module: Takes the local features and cause label set of the failed anchor point as input, trains a small MLP to output the recoverability probability of the anchor point; Adaptive threshold dynamic calculation module: Calculates and dynamically adjusts the anchor point retrieval screening threshold based on the current scene's average point cloud density, texture intensity, and the probability of recovery. The difference repair 3D reconstruction module: For anchor points that exceed the retrieval screening threshold, a targeted enhancement algorithm is called according to the cause label, and the alternative strategy is called for the remaining anchor points. Finally, the anchor point set is output and the 3D map is reconstructed.

[0021] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for constructing a real scene three-dimensional map based on fusion of point cloud data and panoramic images, characterized in that: Includes the following steps: Step 1: Obtain candidate points based on traditional matching, output a set of failed anchor points according to the reprojection error and geometric consistency in the matching, and record the local features of each failed anchor point; Step 2: Use the local features of the failed anchor points as input to train a random forest network and output a set of labels for the reasons of the failed anchor points; Among them, the set of labels for reasons for failed anchor points includes point cloud-side problems, image-side problems, and point cloud-image-side composite problems; Step 3: Using the local features and cause label set of the failed anchor points as input, train a small MLP to output the recoverable probability of anchor point retrieval; Step 4: Calculate and dynamically adjust the anchor point retrieval screening threshold based on the current scene's average point cloud density, texture intensity, and recoverability probability; Step 5: For anchor points that exceed the retrieval screening threshold, apply the targeted enhancement algorithm according to the cause label; for the remaining anchor points, apply the replacement strategy; finally, output the anchor point set and reconstruct the 3D map.

2. The method for constructing a real-scene 3D map based on the fusion of point cloud data and panoramic imagery according to claim 1, characterized in that: In step one, the specific method for outputting the set of failed anchor points based on the reprojection error and geometric consistency during matching is as follows: Calculate the pixel distance between the two-dimensional feature points and the three-dimensional points in each candidate pair of corresponding points after reprojection onto the image plane, and use it as the reprojection error. If the reprojection error is greater than a preset threshold, it is directly marked as a failed anchor point. If the reprojection error is less than or equal to a preset threshold, a geometric consistency judgment is performed. If any one of the three judgment conditions—epidial geometric constraint, local rigidity assumption, and normal vector consistency—is not met, it is marked as a failed anchor point; if all three conditions are met, it is marked as a non-failed anchor point.

3. The method for constructing a real-scene 3D map based on the fusion of point cloud data and panoramic imagery according to claim 1, characterized in that: In the geometric consistency judgment, the epipolar geometric constraint is achieved by calculating the Sampson distance between the projection points of the 3D points on the two frames of images and comparing it with a preset threshold. The local rigidity assumption is achieved by calculating the 3D nearest neighbor distance between the 3D point and its nearest neighbor, and the image nearest neighbor distance between the corresponding image point and its nearest matching point. The ratio of the absolute difference between the 3D nearest neighbor distance and the image nearest neighbor distance is calculated and compared with a preset threshold to achieve the local rigidity assumption judgment. The normal vector consistency is achieved by calculating the absolute value of the cosine of the angle between the 3D point cloud normal vector projected onto the image plane and the image gradient, and comparing it with a preset threshold.

4. The method for constructing a real-scene 3D map based on the fusion of point cloud data and panoramic imagery according to claim 1, characterized in that: Step 1 records the local features of each failed anchor point, including point cloud features, image features, and composite features. Point cloud features are extracted from the three-dimensional points in the failed anchor point within a preset radius neighborhood, including point cloud density, local curvature, and flatness. Image features are extracted from the two-dimensional feature points in the failed anchor point within a preset window, including average gradient magnitude and local grayscale entropy. Composite features include projection consistency error and neighborhood matching success rate.

5. The method for constructing a real-scene 3D map based on the fusion of point cloud data and panoramic imagery according to claim 1, characterized in that: The process of training the random forest network in step two is as follows: Construct a failed anchor point sample dataset containing input feature vectors and output labels. The input feature vectors contain the local features of the failed anchor points recorded in step one, and the output labels are enumerated values ​​representing three categories: point cloud-side problems, image-side problems, or combined problems. Divide the sample dataset into training, validation, and test sets proportionally, and perform missing value imputation and standardization preprocessing. After configuring the random forest parameters, supervised training is performed using the training set, and each tree is randomly split by sampling some samples and some features.

6. The method for constructing a real-scene 3D map based on the fusion of point cloud data and panoramic imagery according to claim 1, characterized in that: Step three involves training the small multilayer perceptron to retrieve the recoverable probability from the output anchor point. Construct a sample dataset containing input feature vectors and output labels. The input feature vectors are composed of local features of failed anchor points and cause labels, and the output labels are binary labels indicating whether the anchor point can be recovered. Divide the sample dataset into training, validation, and test sets according to the proportions, and perform missing value imputation and standardization preprocessing. Build a lightweight multilayer perceptron architecture containing several fully connected layers and an output layer. The output layer uses an activation function to ensure that the probability value is within a preset range. Use mean squared error as the loss function for iterative training until convergence.

7. The method for constructing a real-scene 3D map based on the fusion of point cloud data and panoramic imagery according to claim 1, characterized in that: The method for obtaining the average point cloud density of the current scene in step four is as follows: The calculation boundary is defined by expanding outward from the cluster area of ​​failed anchor points. Noise points and discrete isolated points are removed from the original point cloud. The total number of effective point clouds and spatial volume within the region are statistically calculated, and the average point cloud density is obtained by calculating the ratio. The texture intensity is obtained by projecting the spatial coordinates of failed anchor points onto the panoramic image and extracting pixel blocks. The average grayscale gradient and texture contrast of each block are calculated. After normalization, the mean value is taken as the texture intensity of the block. After removing abnormal blocks, the average texture intensity of all blocks is calculated to obtain the average texture intensity of the scene.

8. The method and system for constructing a real-scene 3D map based on the fusion of point cloud data and panoramic imagery as described in claim 1, characterized in that: The method for dynamically adjusting the anchor point to retrieve the filtering threshold in step four is as follows: The point cloud density quality factor is calculated based on the ratio of the scene's average point cloud density to the preset reference density. The normalized scene average texture intensity is taken as the texture intensity quality factor, and the smaller value between the point cloud density quality factor and the texture intensity quality factor is taken as the dynamic threshold.

9. The method and system for constructing a real-scene 3D map based on the fusion of point cloud data and panoramic imagery as described in claim 1, characterized in that: The specific method for invoking the targeted enhancement algorithm based on the cause label in step five is as follows: When the cause label is a point cloud-side problem, the point cloud-guided interpolation enhancement algorithm is invoked; when the cause label is an image-side problem, the local texture sharpening re-matching algorithm is invoked; when the cause label is a composite problem, both the point cloud-guided interpolation enhancement algorithm and the local texture sharpening re-matching algorithm are invoked simultaneously; for the remaining anchor points, virtual anchor points are generated using an alternative strategy; finally, the non-failed anchor points, the anchor points retrieved through targeted enhancement, and the virtual anchor points generated by the alternative strategy are gathered into the final anchor point set.

10. A real-scene 3D map construction system based on the fusion of point cloud data and panoramic imagery, characterized in that, The system is used to execute the construction method according to any one of claims 1-9, and includes the following modules: Failed anchor point location and extraction module: Based on traditional matching, candidate points of the same name are obtained. The set of failed anchor points is output according to the reprojection error and geometric consistency in the matching, and the local features of each failed anchor point are recorded. Fault Cause Classification Output Module: Takes the local features of the failure anchor points as input, trains a random forest network, and outputs a set of failure anchor point cause labels; Among them, the set of labels for reasons for failed anchor points includes point cloud-side problems, image-side problems, and point cloud-image-side composite problems; Recovery probability prediction and evaluation module: Takes the local features and cause label set of the failed anchor point as input, trains a small MLP to output the recoverability probability of the anchor point; Adaptive threshold dynamic calculation module: Calculates and dynamically adjusts the anchor point retrieval screening threshold based on the current scene's average point cloud density, texture intensity, and the probability of recovery. The difference repair 3D reconstruction module: For anchor points that exceed the retrieval screening threshold, a targeted enhancement algorithm is called according to the cause label, and the alternative strategy is called for the remaining anchor points. Finally, the anchor point set is output and the 3D map is reconstructed.

Images