Multi-modal fusion intelligent synchronization three-dimensional reconstruction method

Abstract

The invention discloses a three-dimensional reconstruction method for multi-modal fusion intelligent synchronization. The method comprises the following steps: acquiring a panoramic image and a laser radar point cloud, optionally acquiring an infrared image and millimeter wave radar data, and realizing multi-source data timestamp synchronization through an intelligent calibration synchronization mechanism; carrying out feature extraction on the panoramic image and the laser radar point cloud by adopting a deep learning and traditional algorithm fusion mode to respectively obtain panoramic image features and radar point cloud features, and optionally fusing infrared image and millimeter wave radar features; based on the panoramic image features, the radar point cloud features and optional multi-mode features, initial camera pose estimation and laser radar pose iterative optimization are completed, and a convergent laser radar pose is obtained; and performing adaptive weight joint pose optimization on the initial camera pose, the sparse three-dimensional point cloud and the converged laser radar pose, and executing closed-loop detection to eliminate accumulative errors and dynamic object elimination to remove noise interference at the same time.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of three-dimensional reconstruction, in particular to a multi-modal fusion intelligent synchronization three-dimensional reconstruction method. BACKGROUND

[0002] Indoor three-dimensional reconstruction technology is the core support of smart office, digital building, residential digitization, etc. Its core goal is to restore the three-dimensional structure and object information of the indoor scene through acquisition equipment, providing a data foundation for spatial planning, asset inventory, virtual simulation, etc. With the acceleration of digital transformation, the demand for reconstruction accuracy, completeness and intelligence is increasingly urgent in office space, industrial plant, smart community, etc. Multi-modal data fusion, such as the combination of panoramic images and laser radar point clouds, has become a key direction to improve reconstruction results, promoting the development of technology towards high precision and automation.

[0003] The current mainstream fusion scheme takes panoramic camera and laser radar as the core, realizes camera pose estimation through global SfM, extracts laser radar curvature features and combines joint pose optimization, relaxes the data synchronization requirement and does not require additional sensors, reducing the complexity of the equipment. However, this type of scheme still has obvious limitations: data synchronization relies on rough timestamp matching, feature extraction mainly uses traditional algorithms, pose optimization uses a fixed weight residual model, and there is no effective solution designed for indoor dynamic noise and long distance cumulative error, resulting in insufficient reconstruction accuracy, model completeness and semantic utility, making it difficult to adapt to the high-precision reconstruction needs of complex indoor scenes.

[0004] The existing technology generally first acquires panoramic images and laser radar point clouds, and roughly synchronizes the data through screen timestamps; uses global SfM algorithm to extract feature points from panoramic images, and through relative pose estimation, pose graph optimization and bundle adjustment, obtains initial camera pose and sparse three-dimensional point cloud; calculates the curvature of laser radar point cloud to extract edge feature points, fuses them into straight line features, matches with adjacent point clouds and optimizes the pose by minimizing the distance, iteratively de-distorts until convergence; finally, performs joint pose optimization based on a fixed weight three-class residual, and outputs the target pose and sparse point cloud.

[0005] However, the existing technology has significant shortcomings: first, data synchronization only relies on rough timestamps, lacks dynamic correction mechanism, and error can reach 100-200 milliseconds, resulting in inconsistent data in space and time; second, feature extraction relies on traditional algorithms, which have poor robustness in low light and textureless areas, making it difficult to balance semantic and geometric features; third, pose optimization uses a fixed weight, which cannot adapt to the error characteristics of different scenes, limiting the optimization accuracy; fourth, there is no effective closed-loop detection and dynamic noise removal mechanism, and long distance acquisition is prone to cumulative error and dynamic targets will introduce noise; fifth, only sparse point clouds are output, lacking dense completion and semantic labeling functions, resulting in insufficient model completeness and utility, limiting subsequent intelligent applications. SUMMARY

[0006] The application provides a multi-modal fusion intelligent synchronization three-dimensional reconstruction method, which is used to solve the technical problems that the pure vision method in the prior art is susceptible to textureless light, the pure laser radar method is susceptible to degradation in the absence of structural feature scenes, the fusion method requires strict synchronization and additional sensors (such as IMU and GNSS) to cause high cost and complexity.

[0007] In view of the above problems, the application provides a multi-modal fusion intelligent synchronization three-dimensional reconstruction method.

[0008] The method comprises the following steps: acquiring panoramic images and laser radar point clouds, and optionally acquiring infrared images and millimeter wave radar data; achieving multi-source data timestamp synchronization through an intelligent calibration synchronization mechanism; performing feature extraction on the panoramic images and the laser radar point clouds in a deep learning and traditional algorithm fusion manner to obtain panoramic image features and radar point cloud features, and optionally fusing infrared image and millimeter wave radar features; completing initial camera pose estimation and laser radar pose iterative optimization based on the panoramic image features, the radar point cloud features and the optional multi-modal features to obtain a converged laser radar pose; performing adaptive weight joint pose optimization on the initial camera pose, the sparse three-dimensional point cloud and the converged laser radar pose, simultaneously performing loop closure detection to eliminate accumulated errors and dynamic object removal to remove noise interference to obtain an optimized camera pose, an optimized laser radar pose and an optimized sparse three-dimensional point cloud; and based on the optimized pose and the sparse three-dimensional point cloud, realizing dense completion through multi-view stereo matching and neural radiation field fusion, combining a semantic segmentation model to complete semantic labeling and outputting a target camera pose, a target laser radar pose and a dense semantic three-dimensional model.

[0009] One or more technical solutions provided in the application have at least the following technical effects or advantages:

[0010] This application embodiment acquires panoramic images and LiDAR point clouds, and optionally infrared images and millimeter-wave radar data, achieving multi-source data timestamp synchronization through an intelligent calibration synchronization mechanism. Feature extraction is performed on the panoramic images and LiDAR point clouds using a fusion of deep learning and traditional algorithms to obtain panoramic image features and LiDAR point cloud features respectively, optionally fusing infrared image and millimeter-wave radar features. Based on the panoramic image features, LiDAR point cloud features, and optional multimodal features, initial camera pose estimation and iterative optimization of the LiDAR pose are completed to obtain a converged LiDAR pose. Adaptive weight joint positioning is then applied to the initial camera pose, sparse 3D point cloud, and converged LiDAR pose. Pose optimization is performed simultaneously with closed-loop detection to eliminate accumulated errors and dynamic object culling to remove noise interference, resulting in optimized camera pose, optimized LiDAR pose, and optimized sparse 3D point cloud. Based on the optimized pose and sparse 3D point cloud, dense completion is achieved through multi-view stereo matching and neural radiation field fusion. Semantic annotation is completed by combining a semantic segmentation model, outputting the target camera pose, target LiDAR pose, and dense semantic 3D model. This solves the technical problems in existing technologies, such as pure vision methods being susceptible to textureless lighting, pure LiDAR methods being prone to degradation in scenes lacking structural features, and the high cost and complexity caused by the need for strict synchronization and additional sensors (such as IMU and GNSS) in fusion methods.

[0011] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart illustrating a multimodal fusion intelligent synchronization three-dimensional reconstruction method provided in this application. Detailed Implementation

[0014] This application provides a multimodal fusion intelligent synchronization 3D reconstruction method to address the technical problems in existing technologies, such as the susceptibility of pure vision methods to textureless lighting, the easy degradation of pure LiDAR methods in scenes lacking structural features, and the high cost and complexity caused by the strict synchronization and additional sensors (such as IMU and GNSS) required for fusion methods.

[0015] Example 1

[0016] like Figure 1 As shown, this application provides a multimodal fusion intelligent synchronization three-dimensional reconstruction method, characterized in that the method includes:

[0017] S100: Acquires panoramic images and LiDAR point clouds, and optionally acquires infrared images and millimeter-wave radar data. It achieves multi-source data timestamp synchronization through an intelligent calibration and synchronization mechanism.

[0018] This process collects various data types, including panoramic images and LiDAR point clouds, and eliminates time differences between different devices through an intelligent calibration mechanism, ensuring that all data are accurately aligned in the time dimension. This step is applicable to indoor 3D reconstruction scenarios such as indoor office space modeling, residential floor plan reconstruction, and industrial plant scanning. The core equipment involved includes panoramic cameras, LiDAR, infrared cameras (optional, suitable for dimly lit indoor environments), and millimeter-wave radar (optional, improving short-range ranging accuracy). The core technologies are PTP (Precise Time Protocol) and mutual information calculation. The former achieves initial time alignment, while the latter corrects subtle time offsets through data feature correlation, ultimately ensuring that the data synchronization error is controlled within 10ms, meeting the accuracy requirements of subsequent feature fusion and pose estimation.

[0019] S100 also includes:

[0020] S110: The hardware layer is configured with a PTP precision time protocol module to perform preliminary calibration of the timestamps of the panoramic camera, LiDAR, infrared camera and millimeter-wave radar to obtain initial timestamp calibration data;

[0021] S120: Define the set of edge features for the panoramic image Line feature set of lidar point cloud Calculate the mutual information between the two. The formula is as follows:

[0022] ;

[0023] in, This represents the mutual information value between the panoramic image features and the lidar point cloud features; The information entropy of the panoramic image feature set represents the uncertainty of the panoramic image features; The information entropy of the lidar point cloud feature set represents the uncertainty of lidar point cloud features; The joint information entropy of panoramic image features and LiDAR point cloud features represents the common uncertainty of the two types of features;

[0024] S130: Dynamically correct timestamp offset based on mutual information value , Where t is the initial timestamp. The candidate offset is used to correct the synchronization error to within 10ms;

[0025] S140: If infrared images and millimeter-wave radar data are acquired, the same mutual information calculation method is used to synchronize them with the panoramic image timestamps to obtain multi-source synchronized data.

[0026] First, a PTP (Precision Time Protocol) module is configured for the panoramic camera and LiDAR. The PTP protocol acts like a unified clock synchronization mechanism for all acquisition devices. Through clock synchronization signals between devices, the system time of each device is initially aligned, reducing initial time deviations and obtaining initial timestamp calibration data. This process is automated at the hardware level without manual intervention, laying the foundation for subsequent accurate synchronization.

[0027] Optionally, an infrared camera can be added to adapt to dimly lit environments such as windowless offices. Another option is to add millimeter-wave radar to improve the ranging accuracy of objects such as tables and chairs.

[0028] Define the set of edge features of a panoramic image It includes all obvious edge information in the image, such as the outline of the desk, the boundary between the wall and the floor, and the edge of the door frame; it defines the set of line features of the LiDAR point cloud. It includes features that exhibit linear distribution in point clouds, such as the point cloud distribution of table legs, filing cabinet edges, and wall corners.

[0029] Calculate the mutual information between the two. Mutual information is an indicator that measures the correlation between two types of features. The formulas represent the information entropy of the two types of features, reflecting the uncertainty of the features. It is the joint information entropy, reflecting the shared uncertainty of the two types of features. The higher the mutual information value, the stronger the correlation between the image features and point cloud features at the same time point, and the more accurate the timestamp.

[0030] Based on the initial timestamp t, set the candidate offset. Calculate different Given the given mutual information values, find the value that maximizes mutual information. As the final offset, i.e. This operation corrects the time difference through feature correlation, ensuring precise temporal matching between image and point cloud data, and ultimately corrects the synchronization error to within 10ms, fully meeting the data synchronization requirements for close-range, high-precision reconstruction in indoor scenes.

[0031] If the project requires the acquisition of infrared images for supplemental lighting in low-light scenarios such as offices without main lights, and millimeter-wave radar data for accurate measurement of the distance between desks and filing cabinets, then the same mutual information calculation method as S120 and S130 is adopted. The mutual information of infrared image features, millimeter-wave radar features and panoramic image features are calculated separately, and their timestamp offsets are dynamically corrected. Finally, multi-source data with complete synchronization of panoramic image, lidar, infrared image and millimeter-wave radar are obtained.

[0032] For example, in a 3D modeling project of an open-plan office in a company, a mobile data acquisition robot equipped with a panoramic camera, LiDAR, infrared camera, and millimeter-wave radar was used. The clocks of the four devices were initially calibrated using a PTP protocol module. The edges of the desks, door frames, and wall intersections in the panoramic image were defined. The data consists of the table legs and filing cabinet lines obtained from LiDAR scanning. After calculating the mutual information values ​​between the two sets of data, the offset is dynamically corrected to control the synchronization error within 7ms. Subsequently, the same method is used to adapt infrared images to dimly lit corners of the office and millimeter-wave radar data to improve the ranging accuracy of desks and partitions and the time synchronization with the panoramic image, providing time-aligned multi-source data for subsequent feature extraction.

[0033] S200: Feature extraction is performed on the panoramic image and the lidar point cloud using a fusion of deep learning and traditional algorithms to obtain panoramic image features and lidar point cloud features respectively. Infrared image and millimeter-wave radar features can be optionally fused together.

[0034] Valuable features are extracted from the synchronized multi-source data, and multimodal information is fused as needed to provide high-quality data support for subsequent pose estimation and 3D modeling. This step adopts a fusion scheme of "deep learning + traditional algorithms". Deep learning models such as DETR-Lite and PointPillars are good at extracting deep semantic features of data, such as distinguishing category information such as "desk", "file cabinet" and "wall", while traditional algorithms such as DeepLSD and curvature calculation are good at capturing geometric features such as lines and edge points. The combination of the two can balance the richness and accuracy of features.

[0035] S200 also includes:

[0036] S210: The DETR-Lite model is used to extract deep features from panoramic images, and the model outputs feature maps. Where C is the number of feature channels, H and W are the feature map dimensions, and the feature extraction formula is as follows:

[0037] ;

[0038] Among them, the DETR-Lite deep learning model is used to extract deep features from panoramic images, providing high-quality image features for subsequent feature fusion; The deep feature map of the panoramic image output by the DETR-Lite model; This is a lightweight DETR (Detection Transformer) feature extraction model; The input is a panoramic image; This is the set of learnable parameters for the DETR-Lite model;

[0039] S220: Detecting line features in panoramic images using the DeepLSD algorithm. The deep features extracted by DETR-Lite are stitched together with the line features detected by DeepLSD to obtain the final features of the panoramic image. ;

[0040] S230: The LiDAR terminal learns the deep semantic features of the point cloud, Fpillar, through the PointPillars network, as shown in the following formula:

[0041] ;

[0042] in, The deep semantic features of the LiDAR point cloud output by the PointPillars network; This is a point cloud feature extraction network based on pillars; The input set of lidar point clouds; This is the set of learnable parameters for the PointPillars network.

[0043] S240: Calculate the curvature of each point in the lidar point cloud. The formula is as follows:

[0044] ;

[0045] in, Let be the curvature value of the i-th point in the lidar point cloud. The larger the curvature, the more likely the point is to be an edge point. The set of neighboring points of the i-th point The number of elements; For the i-th point The magnitude of the spatial coordinate vector; q is the set of neighboring points. Any neighboring point in the array; Let be the vector sum of all neighboring points relative to the i-th point; Let be the magnitude of the sum of the vectors; where, For point The set of neighboring points that satisfies the condition of... Same scan line, at least 5 adjacent points on each side, distance between the farthest points on both sides A minimum of 10 centimeters is required;

[0046] S250: Edge feature point extraction based on curvature The features are fused into linear features using a region growing algorithm. The features are then concatenated with the deep features extracted by PointPillars to obtain the final radar point cloud features. ;

[0047] S260: If infrared images and millimeter-wave radar data are acquired, DETR-Lite is used to extract infrared image features. Extracting millimeter-wave radar range features using convolutional neural networks The attention mechanism is used to fuse multimodal features, as shown in the following formula:

[0048] ;

[0049] in, The fused multimodal features; For attention mechanism functions; For the final features of the panoramic image, The final features of the lidar point cloud, Infrared image features, Range characteristics for millimeter-wave radar; Modal type identifier (img represents panoramic image, lidar represents LiDAR, ir represents infrared, mmw represents millimeter-wave radar); Let be the attention weight for the k-th modality feature, satisfying that the sum of all modality weights is 1; The features of the k-th mode;

[0050] in, satisfy , ;

[0051] in, The attention weights for the k-th modality feature are... The significance score S(k) for the feature of the kth mode. The exponential function of ) is used to calculate the relative proportion of attention weights; Features The significance score; among which, It is the unified ergodic variable of modal features, and the same as in the formula. They have the same meaning; they are only used to distinguish between the "current term" and the "traversal term" in summation traversal to avoid symbol confusion.

[0052] A lightweight DETR-Lite model is used to process panoramic images, with the panoramic image as the model input. The output is a deep feature map F_deep ∈ ℝ^(C×H×W). The advantages of this model are low computational requirements, fast execution speed, and the ability to capture deep semantic information of objects in an image, such as distinguishing between desks, filing cabinets, and green plants. The feature extraction formula is... In this model, θ is a learnable parameter that, after training, can accurately adapt to the feature extraction requirements of indoor panoramic images.

[0053] DETR-Lite is a "lightweight object detection and feature extraction model," based on the DETR (Detection Transformer). "Lite" signifies lightweight, meaning it maintains performance while requiring minimal computational resources, making it suitable for mobile data collection robots and other devices. Its core function is to "understand" image content: it can not only identify objects in images, such as desks and filing cabinets, but also extract deep features of these objects, such as the shape of the desk and the texture of the cabinet. These features are crucial for subsequently determining the correlation between data from different devices.

[0054] Detecting line features in panoramic images using the DeepLSD algorithm Geometric features such as the intersection of walls and ceilings, the outline of desks, and the vertical lines of door frames are crucial for structural reconstruction in interior 3D reconstruction. The deep semantic features F_deep extracted by DETR-Lite and the line features detected by DeepLSD are compared. By stitching the images together, the final features of the panoramic image are obtained. This preserves both semantic category information and geometric structure information, thus enhancing the comprehensiveness of the features.

[0055] DeepLSD, or "Depth Line Detection Algorithm," is specifically designed to accurately locate straight line features in images. Indoor scenes often contain many regular straight lines, such as the boundaries between walls and floors, the vertical lines of door frames, and the edges of desks. These lines form the "skeleton" of the 3D spatial structure. DeepLSD can automatically identify these lines, consistently detecting them regardless of image lighting conditions or slight occlusion. It acts like a "line detector," providing accurate structural references for 3D reconstruction.

[0056] Processing LiDAR point cloud datasets using PointPillars network The network divides the point cloud into multiple cylinders for feature learning and outputs deep semantic features. After training, it can distinguish the categories of point clouds, such as ground points, desk points, filing cabinet points, and corner points, solving the problem that traditional point cloud processing is difficult to capture the semantic information of indoor objects.

[0057] PointPillars is a feature extraction network specifically designed for LiDAR point clouds. LiDAR emits lasers and receives reflected signals, forming a large number of three-dimensional points, or point clouds. These point clouds, like three-dimensional pixels, reflect the spatial location of objects. The core idea of ​​PointPillars is to divide the three-dimensional space into vertical pillars, each containing a point cloud within a certain area. It then analyzes the point cloud distribution within each pillar to determine the corresponding object in that area, such as the ground, a table, or a cabinet. This allows for efficient extraction of semantic information from the point cloud, enabling the computer to identify which points belong to the same object.

[0058] Calculate the curvature of each point in the lidar point cloud The greater the curvature, the more likely the point is to be an edge point, such as the edge of a table leg, the corner of a filing cabinet, or the side of a door frame, providing a basis for subsequent feature extraction.

[0059] Curvature can be understood as the degree of bending of a point, used to determine the positional characteristics of points in a point cloud. For example, in the point cloud of a table leg, the points on the surface will have obvious bends, resulting in a large curvature value; while the point cloud of a tabletop is relatively flat, resulting in a small curvature value. By calculating curvature, edge points of objects can be quickly identified. These edge points are key to forming the outline of an object, much like the outline drawn in a sketch, helping computers accurately reconstruct the three-dimensional shape of an object.

[0060] A curvature threshold of 0.05 is set, and a set E of edge feature points with a curvature greater than 0.05 that are not ground points is selected. These points are then fused into line features using a region growing algorithm. Examples include the point cloud lines of table legs, the outline of filing cabinets, and the intersection lines of wall corners. These line features are then compared with the deep semantic features extracted by PointPillars. By stitching together the data, the final features of the radar point cloud are obtained. It takes into account both semantic and geometric structural information.

[0061] Region growing is a point cloud clustering tool that groups similar points into a single class. For example, selecting table leg edge points that are close in space and have similar curvature characteristics will be automatically connected by the algorithm to form continuous line features, such as a complete table leg point cloud line. By stringing together scattered edge points into line features, the discrete point cloud forms a meaningful geometric structure, facilitating subsequent matching with image features.

[0062] If infrared images and millimeter-wave radar data have been acquired, the DETR-Lite model can be used to extract infrared image features. It is adapted for feature capture in low-light scenarios such as windowless offices, and extracts distance features from millimeter-wave radar through convolutional neural networks. It accurately reflects the distance relationships between desks and filing cabinets, and between partitions and walls. An attention mechanism is used to fuse multimodal features, with higher saliency features receiving greater weight. This ensures that the fused data highlights key information, such as higher weighting for infrared image features in dimly lit corners and higher weighting for LiDAR features in densely furnished areas.

[0063] In dimly lit office corners, infrared images offer clearer features, compensating for the shortcomings of visible light images. Therefore, the attention mechanism assigns higher weight to infrared image features. Conversely, in densely furnished areas, LiDAR point cloud features are more accurate, reflecting the spatial location of objects, thus assigning higher weight to LiDAR features. It automatically adapts to different scenarios, ensuring the fused features better meet actual needs.

[0064] For example, in a company's open-plan office 3D modeling project, the DETR-Lite model was used to extract deep features from the synchronized panoramic image, capturing semantic information such as desks, filing cabinets, greenery, and partitions. Then, the DeepLSD algorithm was used to detect line features such as the intersection of walls and ceilings, the outline of desks, and the vertical edges of door frames, which were then stitched together to obtain the final panoramic image features. For the LiDAR point cloud, the PointPillars network was used to distinguish semantic features such as ground points, desk points, filing cabinet points, and corner points. After calculating the curvature of each point, the edge points of desk legs and filing cabinet corners were selected and fused into line features, which were then stitched together to obtain the final LiDAR point cloud features. Considering that the north side of the office has no windows and is relatively dark, additional infrared image illumination features and millimeter-wave radar near-range ranging features were extracted. An attention mechanism was used to assign weights, with higher weights to infrared image features in dimly lit areas and higher weights to LiDAR features in densely furnished areas, resulting in fused multimodal features.

[0065] S300: Based on the panoramic image features, radar point cloud features, and optional multimodal features, complete the initial camera pose estimation and lidar pose iterative optimization to obtain the converged lidar pose.

[0066] Determining the position and orientation of the camera and LiDAR in space, i.e., pose, involves rapidly estimating the initial camera pose using image sequences and achieving convergence of the LiDAR pose through iterative optimization. This ultimately provides the spatial coordinate reference for the equipment in indoor 3D reconstruction. Pose includes the device orientation (rotation matrix) and the device position (translation vector), and is a crucial prerequisite for fusing multiple frames of indoor data into a 3D model. This step is applicable to mobile acquisition scenarios, such as a robot scanning within an office. By dynamically optimizing the pose, spatial deviations caused by device movement are offset, ensuring the stitching accuracy of the indoor 3D model, such as seamless wall connections and accurate furniture placement.

[0067] The S300 also includes:

[0068] S310: Using the DPVO model to quickly estimate the initial pose sequence of the panoramic camera. ;

[0069] The pose estimation formula is as follows:

[0070] ;

[0071] in, , Let be a rotation matrix. The translation vector is used; the relative camera pose between the i-th frame and the (i-1)-th frame panoramic image is calculated using the DPVO (DeepVisualOdometry) model to generate the initial camera pose sequence; where, Let i be the camera pose corresponding to the i-th frame of the panoramic image, including the rotation matrix. Translation vector ; For depth visual odometry model; For the i-th frame of the panoramic image, This is the (i-1)th frame of the panoramic image; This is the set of learnable parameters for the DPVO model.

[0072] S320: Utilizes depth information from LiDAR point clouds to calibrate pose scale, setting up LiDAR points... The depth in the camera coordinate system is Camera feature points The initial depth is , scale factor The calibrated camera pose is ;

[0073] S330: Based on radar point cloud final features For current lidar point clouds Point clouds of 5 neighboring lidar units Perform feature matching to establish the correspondence between lines and between points and surfaces;

[0074] S340: Define line-to-line distance ( , , , and point-to-surface distance ( , (p, N), the formula is as follows:

[0075] ;

[0076] Among them, the matching distance (in terms of angle) of the centerline features of two LiDAR point clouds is calculated, which is used as an error metric for subsequent LiDAR pose optimization. This is a function for calculating line-to-line distance. Let be the pose of the lidar in the i-th frame. Let j be the pose of the lidar in the j-th frame; The line features in the i-th frame of the lidar are... For the j-th frame of the lidar and Matching line features; p is any point on; p′ is a point on p in the... Projection points on; Let p be the coordinates of point p after the pose transformation of the i-th frame of the lidar. Let p′ be the coordinates of the projection point p′ after the pose transformation of the lidar in the j-th frame. Line features The normal vector; The inverse cosine function is used to convert the angle between vectors into an angle value; the numerator is... and The dot product of the two vectors, with the denominator being the product of their magnitudes, represents the cosine of the angle between the two vectors.

[0077] ;

[0078] Among them, the matching distance (in angular dimensions) from the midpoint of the two frames of lidar point cloud to the plane is calculated, which together with the line-to-line distance constitutes the error term for lidar pose optimization; This is a function for calculating the distance from a point to a surface. , Let p be the pose of the LiDAR in frames i and j, respectively; p be a point in the LiDAR in frame i; N be the plane that matches point p in the LiDAR in frame j; and p′′ be the projection point of p onto plane N. Let p be the coordinates of point p after the pose transformation in the i-th frame. Let p′′ be the coordinates of the projection point after the pose transformation in the j-th frame; N is the normal vector of the plane. The function is an inverse cosine function, and the numerator is... The dot product of the vectors, with the denominator being the product of their magnitudes, represents the cosine of the angle between the two vectors, and ultimately outputs the distance in angular form.

[0079] S350: The LiDAR pose is optimized by minimizing the sum of distances. The objective function is as follows:

[0080] ;

[0081] Among them, by minimizing the sum of the line-to-line distance and the point-to-surface distance of adjacent LiDAR frames, the LiDAR pose set is optimized to achieve the best spatiotemporal consistency of the LiDAR point cloud. This indicates the search for the set of lidar poses that minimizes the objective function. ; is the set of poses of all frames of LiDAR; n is the total number of frames of the LiDAR point cloud; i and j are the frame indices; Let i be the set of neighboring frame indices of the i-th frame; Let be the line-to-line distance of the lidar in frames i and j. Let be the point-to-area distance of the lidar in frames i and j; double summation means accumulating the sum of the distances of all adjacent frame pairs;

[0082] S360: Obtain the pose of the candidate lidar. ,based on Distortion correction is performed on the lidar point cloud, and the point cloud is defined. The transformation between the first and last points is ΔT= Calculate each point using spherical linear interpolation Transform the coordinates of the first point to the coordinate system of the first point; determine whether the lidar pose has converged, the convergence condition is... - <ϵ, where ϵ= , For the current pose, The previous LiDAR pose is used; if convergence is not achieved, feature matching, pose optimization, and distortion correction steps are performed iteratively based on the current LiDAR pose until the pose converges, resulting in a converged LiDAR pose. .

[0083] The DPVO depth visual odometry model is used to calculate the camera's relative pose using two consecutive panoramic images, generating an initial pose sequence. This model can quickly process image sequences and output initial poses in real time, meeting the real-time requirements of indoor mobile image acquisition.

[0084] DPVO, or Depth Visual Odometry, is a model whose core function is to calculate the device's movement trajectory using image sequences. As the robot moves within the office, the camera continuously captures images, and DPVO compares the differences between adjacent frames. For example, if the first frame captures the left side of the desk and the second frame captures the front, this difference can be used to calculate the distance and angle the camera moved and rotated between those two frames—its pose change. This eliminates the need for GPS; real-time positioning is achieved solely through images, providing a positional reference for subsequent 3D modeling.

[0085] The initial pose of the camera, i.e., the scale of the actual physical distance, contains errors and needs to be calibrated using depth information from the LiDAR. Let the LiDAR point... For example, the depth of the center point of the office desk in the camera coordinate system is... The actual physical distance is, for example, 2.5 meters, and the camera feature points. The initial depth is Model estimates distance, calculates scale factor Multiplying the camera translation vector by S yields the calibrated camera pose. Ensure that the dimensions of the position are consistent with the actual interior space, such as avoiding rebuilding a 1.2-meter-wide desk into a 1.8-meter-wide one.

[0086] Scale calibration, in particular, corrects for distance scaling errors. The distance estimated by the camera from the image may be inaccurate; for example, a desk that is actually 1.2 meters wide might be estimated by the camera as 1.0 meter – this is a scale error. LiDAR, with its high depth measurement accuracy, can directly obtain the true physical distance. By comparing the actual distance from the LiDAR with the estimated distance from the camera, a correction coefficient, or scale factor, is calculated. This coefficient is then used to correct the camera's pose data, ensuring that the size of the subsequently reconstructed 3D model matches the real scene and avoids scaling issues.

[0087] Based on the final features of radar point clouds Select the current lidar point cloud Point clouds of the 5 neighboring frames Feature matching is performed to establish line-to-line correspondences, for example, the correspondence between table leg lines and door frame lines in two point clouds, and point-to-plane correspondences, for example, the correspondence between ground points in the current frame and ground planes in adjacent frames, and the correspondence between desk desktop points and desktop planes in adjacent frames, providing a matching basis for subsequent pose optimization.

[0088] Define line-to-line distance Distance from point to surface Matching errors are all measured in angular units.

[0089] Line-to-line distance calculation calculates the angle error between matching line features in two frames of point cloud, such as the matching angle of table leg lines and door frame lines. The vector angle is converted into an angle using the inverse cosine function, and the total error is obtained by accumulating the errors of all points, which reflects the matching accuracy of the line features.

[0090] The point-to-plane distance calculation measures the angular error between a point in the current frame and the matching plane in adjacent frames, such as the matching angle between a ground point and the ground plane, or between a desktop point and the desktop plane. This angle is also output as an angle, reflecting the matching accuracy between the point and the plane. These two types of distances together constitute the error metric for pose optimization.

[0091] With the goal of minimizing the sum of distances between adjacent frames, an optimization objective function is constructed to find the lidar pose set that minimizes the total error. By minimizing the total error, the spatiotemporal consistency of the lidar pose is optimized, meaning that indoor point clouds from different frames can be accurately stitched together.

[0092] Based on candidate LiDAR pose The LiDAR point cloud is subjected to distortion correction processing. The transformation of each point is calculated using spherical linear interpolation and unified to the coordinate system of the initial point, eliminating point cloud distortion caused by device movement. The pose is then checked for convergence. If convergence is not achieved, the feature matching, pose optimization, and distortion correction steps are repeated until convergence is obtained, resulting in a stable LiDAR pose. .

[0093] Spherical linear interpolation is a method for smoothing attitude transitions. A LiDAR scanner needs time to scan a frame of point cloud data; during this time, the acquisition device may move, causing positional deviations (distortions) in the point cloud within the same frame. Spherical linear interpolation can calculate the true position of each point at the instant of scanning based on the device's movement trajectory, correcting the distorted point cloud to the ideal point cloud acquired at the same moment. It's like performing corrective surgery on the point cloud, eliminating errors caused by device movement and allowing the point cloud to more accurately reflect the scene structure.

[0094] For example, in a 3D modeling project of an open-plan office in a company, a data acquisition robot moves within the office at a speed of 0.5 m / s. The DPVO model processes continuous panoramic images in real time to generate an initial camera pose sequence, including the shooting location of each frame: the desk area on the east side of the office, 3 meters from the north wall, and the filing cabinet facing west. The width and actual depth of the desks are measured using LiDAR. =1.2 meters, the scale factor S=1.1 for calibrating the initial pose of the camera, and the corrected pose accurately reflects the actual driving trajectory of the acquisition robot. Feature matching is performed on the current frame of the LiDAR point cloud and the neighboring point clouds of the previous and next two frames, for a total of five frames. The correspondence between the desk leg line and the desk leg line of the adjacent frame, and the ground point and the ground plane of the adjacent frame are established. By calculating the line-to-line and point-to-plane distances, the LiDAR pose is optimized. After distortion removal, the pose convergence error is judged to be less than 1.2 meters. This allows for the acquisition of a stable lidar pose.

[0095] S400: Adaptive weighted joint pose optimization is performed on the initial camera pose, sparse 3D point cloud and converged lidar pose, while loop closure detection is performed to eliminate accumulated errors and dynamic object removal to remove noise interference, so as to obtain the optimized camera pose, optimized lidar pose and optimized sparse 3D point cloud.

[0096] By jointly optimizing the pose of the camera and LiDAR, the overall accuracy of the pose is improved, while cumulative errors and dynamic noise are eliminated, providing high-quality pose and sparse point cloud for subsequent dense reconstruction. Joint optimization dynamically adjusts the weights of different errors to adapt to different indoor scenarios, such as densely furnished open office areas and simply structured conference rooms; closed-loop detection solves the cumulative errors of large-scale indoor data acquisition, such as trajectory deviations after the data acquisition robot circles the office; dynamic object culling removes point cloud noise from moving targets, such as walking employees and temporarily moved chairs, ensuring that the sparse point cloud only contains static indoor scene information such as walls, fixed desks, and filing cabinets.

[0097] The S400 also includes:

[0098] S410: The first residual between panoramic cameras The second residual between the panoramic camera and the LiDAR The third residual between lidars ;

[0099] The first residual is the angle-based reprojection error, as shown in the following formula:

[0100]

[0101] Among them, the reprojection error (angular dimension) between panoramic cameras is calculated as the first type of residual for joint pose optimization, which measures the accuracy of camera pose. denoted as the inter-camera reprojection error (first residual); p represents a point in three-dimensional space, and P is the set of points in three-dimensional space. Let i be the set of camera frame indices for the observed 3D point p; i is the camera frame index. Let be the rotation matrix of the camera in the i-th frame. Let be the translation vector of the camera in the i-th frame; Let be the vector of the 3D point p in the camera coordinate system after the camera pose transformation in the i-th frame; x is the corresponding feature point of the 3D point p in the i-th frame image; π(x) is a function that projects the 2D feature point x onto the surface of a unit sphere, and outputs a unit vector. The inverse cosine function is used to calculate the angle between two vectors to obtain the reprojection error in angular form. The double summation represents the sum of the errors of all three-dimensional points under all observation frames.

[0102] The second residual is the sum of the angle error and the range constraint error of the straight line matching, as shown in the following formula:

[0103] ;

[0104] Among them, the straight-line matching error (angle error and range constraint error) between the panoramic camera and the LiDAR is calculated as the second type of residual for joint pose optimization, which measures the cooperative accuracy of the pose of the camera and the LiDAR. is the matching error between the camera and the lidar (second residual); i and j are the frame indices, corresponding to the camera frame and the lidar frame, respectively; m and n are the line feature indices; For the line features in the i-th frame of the camera image, The line feature in the j-th frame of the lidar; for The corresponding normal vector of the "derived plane", nn, is The corresponding "derived plane" normal vector; the first inverse cosine term is the angle between the two derived plane normal vectors, representing the angle matching error of the line feature; The angle is the projection of the midpoint of the lidar line feature onto the derived plane of the camera line feature. The characteristic half-angle of the lidar line; For range constraint error, when > There is an error if there is one; otherwise, the error is 0. Double summation represents the sum of the errors of all frame pairs and all matching line feature pairs. Wherein, , Where p1 is the projection point of the midpoint of Ln onto the "derived plane" of Lm, pc is the midpoint of Ln, and ps and pe are the endpoints of Ln.

[0105] The third residual is the sum of the line-to-line and point-to-surface distances based on the normalized angle, as shown in the following formula:

[0106] ;

[0107] Specifically, the normalized angular error between lidars (line-to-line and point-to-surface) is calculated as the third type of residual in the joint pose optimization, measuring the accuracy of the lidar pose; variable interpretation: Let represent the matching error between lidars (third residual); i and j are lidar frame indices; Ni is the set of neighboring frames of the i-th frame; Li is the set of line features of the lidar in the i-th frame; Lm is the line feature in Li; Lj is the set of line features of the lidar in the j-th frame; and Ln is the line feature in Lj that matches Lm. For the normalized lidar center, =1 indicates that point p is normalized to the surface of a unit sphere; p is a point on Lm. Let p be the projection point onto Ln, and the first arccosine term be the normalized value. and The included angle represents the line-to-line normalized angle error; Si is the set of points of the i-th frame of the lidar, Pj is the set of planes of the j-th frame of the lidar, and N is the plane in Pj. Let p be the projection of p onto N, and the second inverse cosine term be the normalized value. and The included angle represents the normalized angle error from point to plane; multiple summation represents the sum of errors of all adjacent frame pairs, all matching line feature pairs, and point-plane pairs;

[0108] S420: Employs a reinforcement learning model to dynamically adjust weights. and State s = (C, S, E), where C is the scene texture density, S is the number of structural features, E is the residual error variance, and reward... Where reer represents the reconstruction error, maxer represents the maximum allowable error, and the error weight update formula is as follows:

[0109] ;

[0110] Among these methods, the optimal weights are found through a reinforcement learning model. It is used to dynamically adjust the proportion of the three types of residuals in joint pose optimization to adapt to different scenarios; The optimal weights for the reinforcement learning output (corresponding to the weights of the third and second residuals, respectively). This indicates the search for the weights that maximize the Q-function value. and ; The Q-function is used to evaluate the choice of weights in the current state s. , The value of ; s=(C,S,E) is the current state, where C is the scene texture density, S is the number of structural features, and E is the residual error variance; This is the set of learnable parameters for the Q-network;

[0111] The objective function for joint optimization is as follows:

[0112] ;

[0113] Among them, by minimizing the sum of the three types of weighted residuals, the joint optimization of the camera pose set, the lidar pose set, and the 3D point cloud set is achieved, thereby improving the accuracy of the overall pose and point cloud. This represents the search for the set of camera poses Tc, the set of lidar poses Tl, and the set of 3D point clouds P that minimize the objective function; For the first residual, The third residual, The second residual; , To reinforce the optimal weights obtained through learning, they are used to adjust... and The contribution of the residuals; the objective function is the weighted sum of the three types of residuals, and the residual terms are all wrapped in the Huber loss function to suppress the influence of outliers. The optimization problem is solved by CeresSolver.

[0114] S430: Loop closure detection extracts the global VLAD feature Vimg from the panoramic image and the global FVF feature Vlidar from the LiDAR, and fuses them to obtain the joint descriptor Vfusion=γVimg+(1−γ)Vlidar, where γ=0.5;

[0115] S440: Calculate the descriptor similarity between the current frame and historical frames. If the descriptor similarity is >0.85, it is determined to be a closed-loop frame. The accumulated error is eliminated by optimizing the pose between closed-loop frames.

[0116] S450: Dynamic object culling uses the pyramid LK optical flow method to calculate the motion vector of pixels in the panoramic image. If the motion vector is greater than 5 pixels per frame, it is determined to be a dynamic pixel.

[0117] S460: Analyze the motion consistency of the lidar point cloud. If the displacement of point p in adjacent frames is greater than 0.3 meters, it is determined to be a dynamic point.

[0118] S470: Remove the 3D points and dynamic points corresponding to dynamic pixels to obtain the denoised point cloud;

[0119] S480: Outputs optimized camera pose Toptc, optimized LiDAR pose Toptl, and optimized sparse 3D point cloud Popp; where Toptc is the set of camera poses optimized by weighted residual minimization, Toptl is the set of optimized LiDAR poses, and Popp is the set of sparse 3D point clouds with improved accuracy and removed dynamic noise after optimization.

[0120] Residual is an error metric that measures pose accuracy. Three types of residuals are calculated:

[0121] First residual Between cameras: Angle-based reprojection error measures the consistency of camera pose. It is calculated by adding up the angle error between the 3D point after pose transformation and the image feature point to obtain the total error, such as the projection error of the corner point of the desk in multiple frames of images.

[0122] Second residual Between camera and lidar: The sum of the angular error and the range constraint error in straight-line matching measures the coordination accuracy of the poses of the two types of devices. The angular error reflects the deviation of the matching angle of line features, such as the door frame line, while the range constraint error reflects the positional deviation of the lidar line features and the camera line features, such as the positional deviation of the edge line of the filing cabinet.

[0123] Third residual Between lidars: The sum of line-to-line and point-to-surface distances with normalized angles measures the consistency of lidar pose. Normalization eliminates scale effects and ensures uniformity of error measurement, such as matching errors of table leg lines and ground planes in different frames.

[0124] The residual can be understood as the error value of the pose, used to quantify the accuracy of the pose data. For example, a small residual for the camera pose indicates that the camera's position and orientation are calculated more accurately; a large residual indicates that there is a significant error. The three types of residuals evaluate the error from three dimensions: camera consistency, camera-LiDAR coordination, and LiDAR consistency, respectively, comprehensively covering the key aspects of pose calculation and providing a clear direction for error in subsequent optimization.

[0125] A reinforcement learning model is used to dynamically adjust the weights of the third and second residuals, enabling adaptive adjustment of error weights in different indoor scenarios. For example, in a sparsely furnished conference room, the LiDAR residual has a higher weight. The reinforcement learning model is an intelligent weighting tool whose core function is to automatically adjust error weights based on the scenario. The sources of error differ in different indoor scenarios: for instance, in a densely furnished office area, the camera image has rich texture, making the camera residual more valuable; while in a sparsely furnished conference room, the structural features of the LiDAR are more reliable, making the LiDAR residual more valuable. The reinforcement learning model learns through trial and error; if the reconstruction error decreases after adjusting the weights, this weight allocation is retained; if the error increases, the direction is adjusted. Ultimately, the optimal weights for the current scenario are found, making joint optimization more accurate.

[0126] A joint optimization objective function is constructed to find the camera pose set T^c, LiDAR pose set, and sparse 3D point cloud that minimize the weighted total residual. The residual term is wrapped in the Huber loss function to suppress outliers such as occasional equipment jitter during acquisition. The optimization problem is solved using the CeresSolver tool to obtain the pose and point cloud with improved accuracy.

[0127] CeresSolver is a professional optimization tool whose core function is to find the optimal solution that minimizes the error. The goal of joint optimization is to minimize the weighted sum of the three types of residuals. This problem involves a large amount of complex computation, making manual solution virtually impossible. CeresSolver can automatically handle these calculations, quickly finding the pose and point cloud data that minimize the total error, providing powerful computational support for pose optimization.

[0128] The Huber loss function is an error suppression tool that primarily reduces the interference of outliers. Random errors may occur during data acquisition, such as sudden device jitter causing significant deviations in a particular frame—outliers. Directly incorporating these outliers into optimization can severely impact the results. The Huber loss function identifies these outliers, reduces their weight in error calculation, and makes the optimization results more robust, preventing them from being skewed by a few bad data points.

[0129] The VLAD global feature Vimg and the FVF global feature Vlidar from the LiDAR are extracted from the panoramic image and fused to obtain the joint descriptor Vfusion=γVimg+(1−γ)Vlidar, where γ=0.5, and the two types of features have equal weights. The descriptor similarity between the current frame and historical frames is calculated. If the similarity is >0.85, it is determined to be a closed-loop frame, meaning that the acquisition device has returned to a previously visited position, such as the office entrance area. By optimizing the pose between closed-loop frames, the cumulative error of long-distance acquisition is corrected, such as the acquisition robot accurately closing its trajectory after circling the office.

[0130] VLAD and FVF are both feature compression and description tools, both designed to concisely describe the core features of an entire frame of data. VLAD is used for image processing, while FVF is used for LiDAR point clouds. They can compress the complex features of a frame of data into a single feature vector. When the data acquisition robot returns to its previous location, the feature snapshot of the current frame will be highly similar to that of historical frames. By comparing this similarity, loop closure can be determined, thereby correcting previously accumulated pose errors and making the trajectory more accurate.

[0131] The pyramid LK optical flow method is used to calculate the motion vectors of pixels in the panoramic image. If the motion vector is greater than 5 pixels per frame, it is determined to be a dynamic pixel, such as the pixel movement of walking employees or moving vehicles in the image. The 3D points corresponding to these pixels belong to dynamic noise and need to be removed later.

[0132] Among them, the Pyramid LK optical flow method is a dynamic object detection tool that tracks the movement of pixels in an image. It compares two adjacent frames to calculate the direction and distance of movement for each pixel, i.e., the motion vector. In indoor scenes, the pixel movement of static objects such as walls and desks is very small, while the pixel movement of dynamic objects such as walking employees and moving chairs is very large. By setting a threshold, dynamic and static pixels can be quickly distinguished, providing a basis for subsequent removal of dynamic noise.

[0133] Analyzing the motion consistency of LiDAR point clouds, for example, if the displacement of point p in adjacent frames is greater than 0.3 meters, it is determined to be a dynamic point, such as a water cup moved by an employee or a chair temporarily dragged. The position change of the point cloud in adjacent frames exceeds 0.3 meters. Such point clouds will interfere with the reconstruction of static scenes and need to be removed.

[0134] The 3D points corresponding to the dynamic pixels determined by S450 are removed together with the dynamic points of the LiDAR determined by S460 to obtain a denoised sparse 3D point cloud, while only the point cloud information of static scenes such as walls, fixed desks, filing cabinets and corners is retained.

[0135] The output includes the optimized camera pose Toptc, the optimized LiDAR pose Toptl, and the optimized sparse 3D point cloud Popt.

[0136] For example, in a company's open-plan office 3D modeling project, after a data acquisition robot completes the acquisition of data along an 80-meter path within the office, it calculates the residuals between cameras, between cameras and LiDAR sensors, and between LiDAR sensors. Using a reinforcement learning model, the weights are adjusted based on the scene's dense texture and numerous structural features within the open office area. =0.4、 =0.3, construct and solve a joint optimization function to improve pose accuracy. When the acquisition robot returns to the entrance area after circling the office, a closed-loop frame is detected with a descriptor similarity of 0.91. Optimize the pose between closed-loop frames to eliminate accumulated errors. The pyramid LK optical flow method is used to determine the dynamic pixels of moving employees in the image, with a motion vector of 9 pixels / frame. Dynamic points of temporarily dragging chairs are determined through point cloud displacement analysis, with a displacement of 0.4 meters. After removing these dynamic noises, the optimized camera pose, LiDAR pose, and sparse 3D point cloud containing only walls, fixed desks, filing cabinets, and corners are output.

[0137] S500: Based on the optimized pose and sparse 3D point cloud, dense completion is achieved through multi-view stereo matching and neural radiation field fusion. Semantic annotation is completed by combining the semantic segmentation model, and the target camera pose, target LiDAR pose, and dense semantic 3D model are output.

[0138] The optimized sparse 3D point cloud is completed into a dense point cloud, and each point cloud is assigned a semantic label, ultimately outputting a dense 3D model containing accurate pose and semantic information. Dense completion is achieved through multi-view stereo matching and neural radiation field fusion, solving the problem of holes in the sparse point cloud, such as gaps in the point cloud under an office desk or inside a filing cabinet. Semantic annotation is achieved through a deep learning model, enabling the 3D model to distinguish different categories of indoor objects, such as office desks, filing cabinets, walls, and floors, making it suitable for scenarios requiring detailed 3D information, such as interior design, office space planning, and smart building management.

[0139] The S500 also includes:

[0140] S510: Based on optimized camera pose Optimized LiDAR pose And optimized sparse 3D point cloud The initial dense point cloud was calculated using multi-view stereo matching. The formula is as follows:

[0141] ;

[0142] in, This is the initial dense point cloud output by the MVS algorithm; This is a multi-view stereo matching algorithm; A collection of panoramic images; This is the optimized set of camera poses; This is the optimized sparse 3D point cloud set; This is the matching threshold for the MVS algorithm, used to filter reliable matching pairs;

[0143] S520: Construct a neural radiation field (NeRF) model, to To constrain and optimize the radiation field parameters The model outputs the color c and density σ of any point p(x,y,z) in 3D space, as shown in the following formula:

[0144] ;

[0145] Where c is The model predicts the color of the 3D points; σ is The predicted 3D point density from the model; This is a neural radiation field model; p(x,y,z) represents the coordinates of any point in three-dimensional space. for The set of learnable parameters for the model, including parameters for the positional encoding layer and the fully connected layer;

[0146] S530: Obtaining dense 3D point clouds through volume rendering ,in The density threshold is used; a semantic segmentation model is constructed using MobileNetV3+PointNet to perform semantic segmentation on panoramic images and obtain pixel semantic labels. The formula for mapping labels to dense point clouds is as follows:

[0147] ;

[0148] in, A set of semantic labels for dense 3D point clouds; This is a label mapping function; It is a set of pixel semantic labels for panoramic images, obtained by the MobileNetV3+PointNet semantic segmentation model; This is the optimized set of camera poses, used to provide the camera projection matrix; It is a dense 3D point cloud set; the mapping function establishes the correspondence between pixels and 3D points through the camera projection matrix and assigns pixel labels to the corresponding 3D points;

[0149] S540: Output target camera pose Target lidar pose Dense semantic 3D model .

[0150] Based on the optimized camera pose LiDAR pose Sparse 3D point cloud The initial dense point cloud was calculated using a multi-view stereo matching algorithm. The MVS algorithm supplements a large number of intermediate points to a sparse point cloud by matching pixels of multiple frames of images, thus initially improving the point cloud density, such as increasing the sparse point cloud from 15 points per square meter to 150 points per square meter.

[0151] Among them, the multi-view stereo matching algorithm is a point cloud completion tool. Its core is to generate missing point clouds through interpolation of multiple frames of images. Sparse point clouds cannot fully reflect the surface of an object; for example, an office desk desktop may only have a few points, while multiple frames of images capture the same object from different angles. The MVS algorithm can compare these images, find the corresponding pixels for each location on the object's surface in different images, and calculate the 3D coordinates of that location through pixel matching. This fills in a large number of intermediate points for the sparse point cloud, making the point cloud denser and initially restoring the complete surface of the object.

[0152] Construct a neural radiation field model, in order to To constrain and optimize the radiation field parameters The NeRF model can learn the color c and density σ of any point in 3D space. After training and optimization, the model can accurately predict the color and density of areas not covered by point cloud in space, filling the point cloud gaps left by the MVS algorithm, such as inside a filing cabinet or in the shadow of a corner.

[0153] NeRF, or Neural Radiation Field Model, is an intelligent tool for 3D scene completion and reconstruction. Its core idea is to learn the radiation field patterns of a scene, which contains the color and density information of each point in 3D space. By learning from existing dense point cloud and image data, NeRF can fill in areas that were not previously captured, such as the 3D information under a table or inside a cabinet. It can not only fill in holes in the point cloud but also restore the color and texture of objects, making the reconstructed 3D model more complete and realistic. Simply put, it can reconstruct a complete 3D scene based on existing data.

[0154] Density was selected using volume rendering technology. The three-dimensional points are used to generate a dense three-dimensional point cloud. At this point, the point cloud density is sufficient to reproduce indoor details such as the texture of the desk desktop and the outline of the filing cabinet door.

[0155] A semantic segmentation model is constructed using MobileNetV3+PointNet. First, the panoramic image is semantically segmented to obtain pixel semantic labels. Then, based on the optimized camera projection matrix, a label mapping function is used to establish the correspondence between pixels and dense point clouds. The pixel labels are assigned to the corresponding 3D points to complete the semantic annotation of dense point clouds.

[0156] Volume rendering is a tool for generating point clouds from radiation fields. The radiation field learned by the NeRF model contains density information for every point in space. Volume rendering calculates the density at each location in space along the camera's line of sight, selecting points with a density greater than a threshold. These points represent the actual surface of the object, ultimately generating a complete dense point cloud. It can accurately extract the point cloud constituting the object's surface from the massive amount of information predicted by NeRF, forming a high-quality dense point cloud.

[0157] MobileNetV3+PointNet is a semantic segmentation model that jointly uses images and point clouds. Its core function is to label 3D point clouds. MobileNetV3 processes panoramic images, identifying the object category corresponding to each pixel in the image; PointNet processes dense point clouds, accurately mapping the category labels of image pixels to their corresponding 3D points. For example, if a pixel in an image represents a desk, the corresponding 3D point is found using the camera projection matrix, and this point is labeled as a desk. Ultimately, all 3D points have clear category labels, giving the 3D model not only shape but also semantic information, allowing the computer to clearly identify what each part represents.

[0158] Output target camera pose Target lidar pose Dense semantic 3D model This model contains both accurate spatial location information and clear semantic classification, and can be directly used for subsequent applications such as office space replanning and smart building asset inventory.

[0159] For example, in a company's open-plan office 3D modeling project, based on optimized poses and sparse point clouds, the MVS algorithm was used to calculate an initial dense point cloud, increasing the point cloud density in the desk and filing cabinet areas to 160 points per square meter. A NeRF model was constructed, and after optimizing the radiation field parameters, point cloud holes were filled under desks, inside filing cabinets, and in shadowed corners. A uniformly dense point cloud was generated through volume rendering. Using a MobileNetV3+PointNet model, semantic segmentation was performed on the panoramic image to obtain pixel labels. These labels were then assigned to the dense point cloud through a mapping function, ultimately outputting the target camera pose, the target LiDAR pose, and a dense semantic 3D model containing "desk (brown label), filing cabinet (gray label), wall (white label), floor (light gray label), and door frame (dark brown label)." This model accurately recreates the static scene of the open-plan office and can be directly used for digital management and replanning of office spaces.

[0160] Those skilled in the art will understand that the various numerical designations, such as "first," "second," etc., used in this application are merely for descriptive convenience and are not intended to limit the scope of this application, nor do they indicate a chronological order. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one" refers to one or more. "At least two" refers to two or more. "At least one," "any one," or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can represent: a, b, c, a-b, ac, bc, or abc, where a, b, and c can be single or multiple.

[0161] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0162] The instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0163] The various illustrative logic units and circuits described in this application may be implemented or operate the described functions using a general-purpose processor, digital signal processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof. The general-purpose processor may be a microprocessor, and optionally, it may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented using a combination of computing devices, such as a digital signal processor and a microprocessor, multiple microprocessors, one or more microprocessors combined with a digital signal processor core, or any other similar configuration.

[0164] The steps of the methods or algorithms described in this application can be directly embedded in hardware, a software unit executed by a processor, or a combination of both. The software unit can be stored in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other storage medium of any form in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from the storage medium and write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and storage medium can be disposed in an ASIC, which can be disposed in a terminal. Optionally, the processor and storage medium can also be disposed in different components within the terminal. These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0165] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely illustrative examples of this application and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from its scope. Thus, if such modifications and modifications fall within the scope of this application and its equivalents, this application intends to include such modifications and modifications.

Claims

1. A multimodal fusion intelligent synchronization three-dimensional reconstruction method, characterized in that, The method includes: It acquires panoramic images and LiDAR point clouds, and optionally infrared images and millimeter-wave radar data, and achieves multi-source data timestamp synchronization through an intelligent calibration and synchronization mechanism; Feature extraction is performed on the panoramic image and the lidar point cloud using a fusion of deep learning and traditional algorithms to obtain panoramic image features and lidar point cloud features respectively. Infrared image and millimeter-wave radar features can be optionally fused together. Based on the panoramic image features, radar point cloud features, and optional multimodal features, the initial camera pose estimation and lidar pose iterative optimization are completed to obtain the converged lidar pose. Adaptive weighted joint pose optimization is performed on the initial camera pose, sparse 3D point cloud and converged lidar pose. At the same time, closed-loop detection is performed to eliminate accumulated errors and dynamic object removal to remove noise interference, resulting in optimized camera pose, optimized lidar pose and optimized sparse 3D point cloud. Based on the optimized pose and sparse 3D point cloud, dense completion is achieved through multi-view stereo matching and neural radiation field fusion. Semantic annotation is completed by combining the semantic segmentation model, and the target camera pose, target LiDAR pose, and dense semantic 3D model are output.

2. The method according to claim 1, characterized in that, The acquisition of panoramic images and LiDAR point clouds, optionally including infrared images and millimeter-wave radar data, and the synchronization of multi-source data timestamps through an intelligent calibration and synchronization mechanism, includes: The hardware layer is configured with a PTP precision time protocol module to perform preliminary calibration of the timestamps of the panoramic camera, LiDAR, infrared camera and millimeter-wave radar, and obtain the initial timestamp calibration data; Define the set of edge features of a panoramic image Line feature set of lidar point cloud Calculate the mutual information between the two. The formula is as follows: ； in, This represents the mutual information value between the panoramic image features and the lidar point cloud features; The information entropy of the panoramic image feature set represents the uncertainty of the panoramic image features; The information entropy of the lidar point cloud feature set represents the uncertainty of lidar point cloud features; The joint information entropy of panoramic image features and LiDAR point cloud features represents the common uncertainty of the two types of features; Dynamically correct timestamp offset based on mutual information value , Where t is the initial timestamp. The candidate offset is used to correct the synchronization error to within 10ms; If infrared images and millimeter-wave radar data are acquired, the same mutual information calculation method is used to synchronize them with the panoramic image timestamps to obtain multi-source synchronized data.

3. The method according to claim 2, characterized in that, The panoramic image and the lidar point cloud are subjected to feature extraction using a fusion of deep learning and traditional algorithms to obtain panoramic image features and lidar point cloud features, respectively. Optionally, infrared image and millimeter-wave radar features are fused, including: The DETR-Lite model is used to extract deep features from panoramic images, and the model outputs a feature map. Where C is the number of feature channels, H and W are the feature map dimensions, and the feature extraction formula is as follows: ； Among them, the DETR-Lite deep learning model is used to extract deep features from panoramic images, providing high-quality image features for subsequent feature fusion; The deep feature map of the panoramic image output by the DETR-Lite model; This is a lightweight DETR (Detection Transformer) feature extraction model; The input is a panoramic image; This is the set of learnable parameters for the DETR-Lite model; Detecting line features in panoramic images using the DeepLSD algorithm The deep features extracted by DETR-Lite are stitched together with the line features detected by DeepLSD to obtain the final features of the panoramic image. ; The LiDAR unit learns the deep semantic features of the point cloud, Fpillar, through the PointPillars network, as shown in the following formula: ； in, The deep semantic features of the LiDAR point cloud output by the PointPillars network; This is a point cloud feature extraction network based on pillars; The input is a set of lidar point clouds; This is the set of learnable parameters for the PointPillars network. Calculate the curvature of each point in the lidar point cloud The formula is as follows: ; in, Let be the curvature value of the i-th point in the lidar point cloud. The larger the curvature, the more likely the point is to be an edge point. The set of neighboring points of the i-th point The number of elements; For the i-th point The magnitude of the spatial coordinate vector; q is the set of neighboring points. Any neighboring point in the array; The vector sum of all neighboring points relative to the i-th point; Let be the magnitude of the sum of the vectors; where, For point The set of neighboring points that satisfies the condition of... Same scan line, at least 5 adjacent points on each side, distance between the farthest points on both sides A minimum of 10 centimeters is required; Edge feature point extraction based on curvature The features are fused into linear features using a region growing algorithm. The features are then concatenated with the deep features extracted by PointPillars to obtain the final radar point cloud features. ; If infrared images and millimeter-wave radar data are acquired, DETR-Lite can be used to extract infrared image features. Extracting millimeter-wave radar range features using convolutional neural networks The attention mechanism is used to fuse multimodal features, as shown in the following formula: ； in, The fused multimodal features; For attention mechanism functions; For the final features of the panoramic image, The final features of the lidar point cloud, Infrared image features, Range characteristics for millimeter-wave radar; Modal type identifier (img represents panoramic image, lidar represents LiDAR, ir represents infrared, mmw represents millimeter-wave radar); Let be the attention weight for the k-th modality feature, satisfying that the sum of all modality weights is 1; The features of the k-th mode; in, satisfy , ; in, The attention weights for the k-th modality feature are... The significance score S(k) for the feature of the kth mode. The exponential function of ) is used to calculate the relative proportion of attention weights; Features The significance score; among which, It is the unified ergodic variable of modal features, and the same as in the formula. They have the same meaning; they are only used to distinguish between the "current term" and the "traversal term" in summation traversal to avoid symbol confusion.

4. The method according to claim 3, characterized in that, The process of initial camera pose estimation and iterative optimization of lidar pose based on the panoramic image features, radar point cloud features, and optional multimodal features to obtain a converged lidar pose includes: The DPVO model is used to quickly estimate the initial pose sequence of the panoramic camera. ; The pose estimation formula is as follows: ； in, , For rotation matrix, The translation vector is used; the relative camera pose between the i-th frame and the (i-1)-th frame panoramic image is calculated using the DPVO (DeepVisualOdometry) model to generate the initial camera pose sequence; where, Let i be the camera pose corresponding to the i-th frame of the panoramic image, including the rotation matrix. Translation vector ; For depth visual odometry model; For the i-th frame of the panoramic image, This is the (i-1)th frame of the panoramic image; This is the set of learnable parameters for the DPVO model. Using depth information from lidar point clouds to calibrate pose scale, let lidar points... The depth in the camera coordinate system is Camera feature points The initial depth is , scale factor The calibrated camera pose is ; Based on the final features of radar point clouds For current lidar point clouds Point clouds of 5 neighboring lidar units Perform feature matching to establish the correspondence between lines and between points and surfaces; Define line-to-line distance ( , , , and point-to-surface distance ( , (p, N), the formula is as follows: ； Among them, the matching distance (in terms of angle) of the centerline features of two LiDAR point clouds is calculated, which is used as an error metric for subsequent LiDAR pose optimization. This is a function for calculating line-to-line distance. Let be the pose of the lidar in the i-th frame. Let j be the pose of the lidar in the j-th frame; The line feature in the i-th frame of the lidar is... For the j-th frame of the lidar, and Matching line features; p is any point on; p′ is a point on p in the interval [1, 2, 3]; Projection points on; Let p be the coordinates of point p after the pose transformation of the i-th frame of the lidar. Let p′ be the coordinates of the projection point p′ after the pose transformation of the lidar in the j-th frame. Line features The normal vector; The inverse cosine function is used to convert the angle between vectors into an angle value; the numerator is... and The dot product of the two vectors, with the denominator being the product of their magnitudes, represents the cosine of the angle between the two vectors. ； Among them, the matching distance (in angular dimensions) from the midpoint of the two frames of lidar point cloud to the plane is calculated, which together with the line-to-line distance constitutes the error term for lidar pose optimization; This is a function for calculating the distance from a point to a surface. , Let p be the pose of the LiDAR in frames i and j, respectively; p be a point in the LiDAR in frame i; N be the plane that matches point p in the LiDAR in frame j; and p′′ be the projection point of p onto plane N. Let p be the coordinates of point p after the pose transformation in the i-th frame. Let p′′ be the coordinates of the projection point after the pose transformation in the j-th frame; N is the normal vector of the plane. The function is an inverse cosine function, and the numerator is... The dot product of the vectors, with the denominator being the product of their magnitudes, represents the cosine of the angle between the two vectors, and ultimately outputs the distance in angular form. The LiDAR pose is optimized by minimizing the sum of distances. The optimization objective function is as follows: ； Among them, by minimizing the sum of the line-to-line distance and the point-to-surface distance of adjacent LiDAR frames, the LiDAR pose set is optimized to achieve the best spatiotemporal consistency of the LiDAR point cloud. This indicates the search for the set of lidar poses that minimizes the objective function. ; is the set of poses of all frames of LiDAR; n is the total number of frames of the LiDAR point cloud; i and j are the frame indices; Let i be the set of neighboring frame indices of the i-th frame; Let be the line-to-line distance of the lidar in frames i and j. Let be the point-to-area distance of the lidar in frames i and j; double summation means accumulating the sum of the distances of all adjacent frame pairs; Obtain the pose of the candidate lidar ,based on Distortion correction is performed on the lidar point cloud, and the point cloud is defined. The transformation between the first and last points is ΔT= Calculate each point using spherical linear interpolation Transform the coordinates of the first point to the coordinate system of the first point; determine whether the lidar pose has converged, the convergence condition is... - <ϵ, where ϵ= , For the current pose, The previous LiDAR pose is used; if convergence is not achieved, feature matching, pose optimization, and distortion correction steps are performed iteratively based on the current LiDAR pose until the pose converges, resulting in a converged LiDAR pose. .

5. The method according to claim 4, characterized in that, The process involves adaptive weighted joint pose optimization of the initial camera pose, sparse 3D point cloud, and converged LiDAR pose, while simultaneously performing loop closure detection to eliminate accumulated errors and dynamic object culling to remove noise interference, resulting in optimized camera pose, optimized LiDAR pose, and optimized sparse 3D point cloud. The first residual between panoramic cameras The second residual between the panoramic camera and the LiDAR The third residual between lidars ; The first residual is the angle-based reprojection error, as shown in the following formula:

6. Among them, Calculate the reprojection error (in angular dimensions) between panoramic cameras, and use it as the first type of residual for joint pose optimization to measure the accuracy of camera pose. denoted as the inter-camera reprojection error (first residual); p represents a point in three-dimensional space, and P is the set of points in three-dimensional space. The set of camera frame indices for observing the 3D point p; i is the camera frame index; Let be the rotation matrix of the camera in the i-th frame. Let be the translation vector of the camera in the i-th frame; Let be the vector of the 3D point p in the camera coordinate system after the camera pose transformation in the i-th frame; x is the corresponding feature point of the 3D point p in the i-th frame image; π(x) is a function that projects the 2D feature point x onto the surface of a unit sphere, and outputs a unit vector. The inverse cosine function is used to calculate the angle between two vectors to obtain the reprojection error in angular form. The double summation represents the sum of the errors of all three-dimensional points under all observation frames. The second residual is the sum of the angle error and the range constraint error of the straight line matching, as shown in the following formula: ； Among them, the straight-line matching error (angle error and range constraint error) between the panoramic camera and the LiDAR is calculated as the second type of residual for joint pose optimization, which measures the cooperative accuracy of the pose of the camera and the LiDAR. is the matching error between the camera and the lidar (second residual); i and j are the frame indices, corresponding to the camera frame and the lidar frame, respectively; m and n are the line feature indices; For the line features in the i-th frame of the camera image, The line feature in the j-th frame of the lidar; for The corresponding normal vector of the "derived plane", nn, is The corresponding "derived plane" normal vector; the first inverse cosine term is the angle between the two derived plane normal vectors, representing the angle matching error of the line feature; The angle is the projection of the midpoint of the lidar line feature onto the derived plane of the camera line feature. This refers to the half-angle of the lidar line characteristics; For range constraint error, when > There is an error if there is one; otherwise, the error is 0. Double summation represents the sum of the errors of all frame pairs and all matching line feature pairs. Wherein, , Where p1 is the projection point of the midpoint of Ln onto the "derived plane" of Lm, pc is the midpoint of Ln, and ps and pe are the endpoints of Ln. The third residual is the sum of the line-to-line and point-to-surface distances based on the normalized angle, as shown in the following formula: ； Specifically, the normalized angular error between lidars (line-to-line and point-to-surface) is calculated as the third type of residual in the joint pose optimization, measuring the accuracy of the lidar pose; variable interpretation: Let represent the matching error between lidars (third residual); i and j are lidar frame indices; Ni is the set of neighboring frames of the i-th frame; Li is the set of line features of the lidar in the i-th frame; Lm is the line feature in Li; Lj is the set of line features of the lidar in the j-th frame; and Ln is the line feature in Lj that matches Lm. For the normalized lidar center, =1 indicates that point p is normalized to the surface of a unit sphere; p is a point on Lm. Let p be the projection point onto Ln, and the first arccosine term be the normalized value. and The included angle represents the line-to-line normalized angle error; Si is the set of points of the i-th frame of the lidar, Pj is the set of planes of the j-th frame of the lidar, and N is the plane in Pj. Let p be the projection of p onto N, and the second inverse cosine term be the normalized value. and The included angle represents the normalized angle error from point to plane; multiple summation represents the sum of errors of all adjacent frame pairs, all matching line feature pairs, and point-plane pairs; Employing a reinforcement learning model to dynamically adjust weights and State s = (C, S, E), where C is the scene texture density, S is the number of structural features, E is the residual error variance, and reward... Where reer represents the reconstruction error, maxer represents the maximum allowable error, and the error weight update formula is as follows: ； Among these methods, the optimal weights are found through a reinforcement learning model. It is used to dynamically adjust the proportion of the three types of residuals in joint pose optimization to adapt to different scenarios; The optimal weights for the reinforcement learning output (corresponding to the weights of the third and second residuals, respectively). This indicates the search for the weights that maximize the Q-function value. and ; The Q-function is used to evaluate the choice of weights in the current state s. , The value of ; s=(C,S,E) is the current state, where C is the scene texture density, S is the number of structural features, and E is the residual error variance; This is the set of learnable parameters for the Q-network; The objective function for joint optimization is as follows: ； Among them, by minimizing the sum of the three types of weighted residuals, the joint optimization of the camera pose set, the lidar pose set, and the 3D point cloud set is achieved, thereby improving the accuracy of the overall pose and point cloud. This represents the search for the set of camera poses Tc, the set of lidar poses Tl, and the set of 3D point clouds P that minimize the objective function; For the first residual, The third residual, The second residual; , To reinforce the optimal weights obtained through learning, they are used to adjust... and The contribution of the residuals; the objective function is the weighted sum of the three types of residuals, and the residual terms are all wrapped in the Huber loss function to suppress the influence of outliers. The optimization problem is solved by CeresSolver. Loop closure detection extracts the global VLAD feature Vimg from the panoramic image and the global FVF feature Vlidar from the LiDAR, and fuses them to obtain the joint descriptor Vfusion=γVimg+(1−γ)Vlidar, where γ=0.5; Calculate the descriptor similarity between the current frame and historical frames. If the descriptor similarity is >0.85, it is determined to be a closed-loop frame. The accumulated error is eliminated by optimizing the pose between closed-loop frames. Dynamic object culling uses the pyramid LK optical flow method to calculate the motion vector of pixels in the panoramic image. If the motion vector is greater than 5 pixels per frame, it is determined to be a dynamic pixel. Analyze the motion consistency of the lidar point cloud. If the displacement of point p in adjacent frames is greater than 0.3 meters, it is determined to be a dynamic point. Remove the 3D points and dynamic points corresponding to the dynamic pixels to obtain the denoised point cloud; The output includes the optimized camera pose Toptc, the optimized LiDAR pose Toptl, and the optimized sparse 3D point cloud Popp; where Toptc is the set of camera poses optimized by weighted residual minimization, Toptl is the set of optimized LiDAR poses, and Popp is the set of sparse 3D point clouds with improved accuracy and removed dynamic noise after optimization.

7. The method according to claim 1, characterized in that, Based on the optimized pose and sparse 3D point cloud, dense completion is achieved through multi-view stereo matching and neural radiation field fusion. Semantic annotation is then completed using a semantic segmentation model, outputting the target camera pose, target LiDAR pose, and a dense semantic 3D model, including: Based on the optimized camera pose Optimized LiDAR pose And optimized sparse 3D point cloud The initial dense point cloud was calculated using multi-view stereo matching. The formula is as follows: ； in, This is the initial dense point cloud output by the MVS algorithm; This is a multi-view stereo matching algorithm; A collection of panoramic images; This is the optimized set of camera poses; This is the optimized sparse 3D point cloud set; This is the matching threshold for the MVS algorithm, used to filter reliable matching pairs; Construct a neural radiation field (NeRF) model to... To constrain and optimize the radiation field parameters The model outputs the color c and density σ of any point p(x,y,z) in 3D space, as shown in the following formula: ； c is The model predicts the color of the 3D points; σ is The predicted 3D point density by the model; This is a neural radiation field model; p(x,y,z) represents the coordinates of any point in three-dimensional space. for The set of learnable parameters for the model, including parameters for the positional encoding layer and the fully connected layer; Dense 3D point cloud obtained through volume rendering ,in The density threshold is used; a semantic segmentation model is constructed using MobileNetV3+PointNet to perform semantic segmentation on panoramic images and obtain pixel semantic labels. The formula for mapping labels to dense point clouds is as follows: ； in, A set of semantic labels for dense 3D point clouds; This is a label mapping function; It is a set of pixel semantic labels for panoramic images, obtained by the MobileNetV3+PointNet semantic segmentation model; This is the optimized set of camera poses, used to provide the camera projection matrix; It is a dense 3D point cloud set; the mapping function establishes the correspondence between pixels and 3D points through the camera projection matrix and assigns pixel labels to the corresponding 3D points; Output target camera pose Target lidar pose Dense semantic 3D model .

Images