Slam pose optimization method and three-dimensional reconstruction device

Abstract

The application relates to the technical field of three-dimensional reconstruction, in particular to a SLAM pose optimization method and a three-dimensional reconstruction device. After a three-dimensional scene is marked, a SAM segmentation model is introduced, three-dimensional point cloud data in disorder is projected into an ordered two-dimensional intensity image, the SAM segmentation model can process laser radar data, the three-dimensional coordinates of the marked points can be more accurately extracted by using the SAM model, and the calculation accuracy of the marked points is significantly improved. A global factor graph is generated according to the semantic three-dimensional coordinates of the extracted marked points as semantic landmark factors, the pose of the laser radar is constrained and optimized according to the global factor graph, and in the scenes of tunnels, long corridors and the like, the constraint is provided when the geometric features are missing, and the drift of the SLAM is significantly inhibited.

Description

Technical Field

[0001] This application relates to the field of 3D reconstruction technology, specifically to a SLAM pose optimization method, a 3D reconstruction device, and a computer-readable storage medium. Background Technology

[0002] LiDAR-based SLAM (Simultaneous Localization and Mapping) technology is a localization and navigation technology widely used in robots and autonomous vehicles. Its basic principle is to acquire 3D point cloud data of the surrounding environment using LiDAR, and then simultaneously build a map in the unknown environment while estimating its own position and orientation.

[0003] LiDAR SLAM technology is widely used in tunnel inspection, underground mines and other fields due to its high precision and resistance to light interference. However, in geometrically degraded scenarios such as long straight tunnels and smooth corridors, the lack of sufficient geometric features, such as corner points and surface features, makes LiDAR point cloud registration prone to slippage in the extension direction of the same scene, resulting in severe drift. Summary of the Invention

[0004] In view of the above problems, embodiments of this application provide a SLAM pose optimization method, a 3D reconstruction device, and a computer-readable storage medium to solve the above-mentioned technical problems existing in the prior art.

[0005] According to one aspect of the embodiments of this application, a SLAM pose optimization method is proposed for application to LiDAR, comprising: Acquire 3D point cloud data of multiple keyframes in a 3D scene, wherein the 3D scene includes multiple preset marker points, and the 3D point cloud data includes 3D coordinates and intensity information; A two-dimensional intensity image is generated based on the three-dimensional coordinates and intensity information of the keyframe; The two-dimensional intensity image is semantically segmented according to a preset segmentation model to generate multiple masks corresponding to the marker points; The pixels within the mask are back-projected back into the 3D scene to generate semantic 3D coordinates corresponding to multiple marker points; Construct semantic landmark factors based on the observation constraints between the keyframe and the semantic 3D coordinates corresponding to the marker points; Construct a global factor graph based on the semantic landmark factors; The global factor graph is optimized using a nonlinear least squares method, thereby optimizing the SLAM pose.

[0006] Optionally, in some embodiments, acquiring the 3D point cloud data of multiple keyframes in a 3D scene includes: The three-dimensional point cloud data of the lidar is sampled at intervals according to a preset distance; The 3D point cloud data obtained by the interval sampling is accumulated to generate the 3D point cloud data of the key frame.

[0007] Optionally, in some embodiments, generating a two-dimensional intensity image based on the three-dimensional coordinates and intensity information of the keyframe includes: Obtain the horizontal field of view, vertical field of view, horizontal resolution, and vertical resolution of the lidar; The resolution of the two-dimensional image is determined based on the horizontal resolution and the vertical resolution; The actual range of the two-dimensional image in the horizontal and vertical directions is determined based on the horizontal field of view, the vertical field of view, and the resolution of the two-dimensional image.

[0008] Optionally, in some embodiments, generating a two-dimensional intensity image based on the three-dimensional coordinates and intensity information of the keyframe further includes: Based on the three-dimensional coordinates and the actual range of the two-dimensional image in the horizontal and vertical directions, calculate the two-dimensional coordinates of the point cloud data points of the key frame on the two-dimensional image. The intensity information corresponding to the point cloud data points of the key frame is assigned to the pixel points corresponding to the two-dimensional coordinates to generate a two-dimensional intensity image.

[0009] Optionally, in some embodiments, constructing semantic landmark factors based on the observation constraints between the keyframe and the semantic three-dimensional coordinates corresponding to the marker points includes: Find related keyframes with the same marker point in the keyframes; Establish pose constraint relationships between the associated keyframes based on the poses of each associated keyframe; Semantic landmark factors are constructed based on the pose constraints and the observation constraints.

[0010] Optionally, in some embodiments, constructing semantic landmark factors based on the pose constraint relationship and the observation constraint includes: Determine the actual location of the marker point observed in the associated keyframe; The predicted position of the marker point is generated based on the pose of the associated keyframe and the semantic 3D coordinates of the marker point; The semantic landmark factor is determined based on the error between the predicted location and the actual location.

[0011] Optionally, in some embodiments, constructing a global factor graph based on the semantic landmark factors includes: Obtain the pose information of the lidar; The relative pose between adjacent keyframes is obtained based on the pose information, and the odometry factor of the lidar is determined based on the relative pose. A global factor graph is constructed based on the positions of multiple marker points, the pose information of the lidar, the odometer factor, and the semantic landmark factor.

[0012] Optionally, in some embodiments, optimizing the global factor graph using a nonlinear least squares method includes: The global factor graph is optimized according to the preset Levenberg-Marquardt algorithm.

[0013] According to another aspect of the embodiments of this application, a three-dimensional reconstruction device is proposed, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus; The memory is used to store at least one program that causes the processor to execute the SLAM pose optimization method described in the above embodiments.

[0014] According to a third aspect of the embodiments of this application, a readable computer storage medium is provided, wherein the storage medium stores at least one program, which, when run on a 3D reconstruction device, causes the 3D reconstruction device to perform the SLAM pose optimization method described in the above embodiments.

[0015] In summary, this embodiment, after marking the 3D scene, introduces the SAM segmentation model. By projecting the disordered 3D point cloud data into an ordered 2D intensity image, the SAM segmentation model can process the LiDAR data. Utilizing the powerful edge segmentation capabilities of the SAM model, masks corresponding to each marker point are accurately segmented on the 2D intensity image. These masks are then back-projected into 3D space to generate semantic 3D coordinates for each marker point, significantly improving the calculation accuracy of the marker points. Furthermore, this embodiment uses the extracted semantic 3D coordinates of the marker points as semantic landmark factors to generate a global factor map. This global factor map constrains and optimizes the LiDAR pose, providing anchor point constraints in scenarios such as tunnels and long corridors where geometric features are missing, significantly suppressing SLAM drift.

[0016] The above description is merely an overview of the technical solutions of the embodiments of this application. In order to better understand the technical means of the embodiments of this application and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of this application more obvious and understandable, specific implementation methods of this application are described below. Attached Figure Description

[0017] The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 This is a schematic diagram of the SLAM pose optimization scenario proposed in the embodiments of this application; Figure 2 This is a flowchart illustrating the SLAM pose optimization method proposed in an embodiment of this application. Figure 3 This is a schematic diagram of the two-dimensional intensity image generation method proposed in the embodiments of this application; Figure 4 This is a schematic diagram of the semantic landmark factor generation process proposed in the embodiments of this application; Figure 5 This is a schematic diagram of the structure of the three-dimensional reconstruction device proposed in the embodiments of this application. Detailed Implementation

[0018] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein.

[0019] In geometrically degraded scenarios such as long straight tunnels and smooth corridors, LiDAR point cloud registration is prone to slippage along the tunnel's extension direction, leading to severe drift. On one hand, these environments are typically relatively homogeneous, lacking rich geometric features (such as corners and edges). LiDAR point cloud registration algorithms (such as the ICP algorithm) rely on these features for matching and localization. When sufficient geometric features are lacking, the algorithm struggles to find reliable matching points, increasing registration errors. On the other hand, long straight tunnels and smooth corridors often exhibit high axial symmetry, meaning environmental features are very similar along the tunnel's extension direction. This symmetry makes it difficult for point cloud registration algorithms to distinguish between points at different locations, easily leading to mismatches along the axial direction and drift. Furthermore, LiDAR SLAM systems estimate the robot's absolute pose by accumulating inter-frame pose changes. In geometrically degraded scenarios, due to inaccuracies in point cloud registration, errors from each registration accumulate, causing the localization error to increase over time, ultimately resulting in severe drift. Meanwhile, the lidar sensor itself contains noise, including measurement noise and motion noise. In geometrically degraded scenarios, this noise is amplified, further reducing the accuracy of point cloud registration and thus causing drift.

[0020] To address these issues, existing solutions typically employ multi-sensor fusion, such as fused IMUs, wheel speedometers, or cameras. However, introducing cameras requires complex intrinsic and extrinsic parameter calibration, and cameras are prone to failure in dimly lit tunnels or environments with alternating strong and weak light. Another approach is to use manual road signs, such as reflectors. However, these traditional methods usually rely on manually designed feature extraction algorithms, such as high-brightness threshold detection. This method is sensitive to environmental noise and struggles to distinguish other highly reflective objects (such as people wearing reflective clothing), resulting in low accuracy in extracting the center of the road sign.

[0021] To address the geometric degradation and drift issues encountered by laser SLAM in scenarios such as tunnels and long corridors, this application proposes a SLAM pose optimization method and a 3D reconstruction device. After setting marker points in the 3D scene, a SAM (Segment Anything Model) is introduced to project the disordered 3D point cloud data into an ordered 2D intensity image. This allows the SAM segmentation model to process the lidar data. Utilizing the powerful edge segmentation capabilities of the SAM model, masks corresponding to each marker point are accurately segmented on the 2D intensity image. These masks are then back-projected into 3D space to generate semantic 3D coordinates for each marker point. Compared to traditional thresholding methods, this application can extract the 3D coordinates of marker points more accurately using the SAM model, significantly improving the calculation accuracy of marker points. Furthermore, the extracted semantic 3D coordinates of the marker points are used as semantic landmark factors to generate a global factor map. This global factor map is used to constrain and optimize the lidar pose, providing anchor point constraints in scenarios such as tunnels and long corridors where geometric features are missing, significantly suppressing SLAM drift.

[0022] The SLAM pose optimization method and 3D reconstruction equipment proposed in this application can be applied to scenarios requiring modeling in tunnels and long corridors, including autonomous driving, robot navigation, and map building. In the field of autonomous driving, LiDAR SLAM technology is used for vehicle localization and environmental perception; in the field of robotics, it is used for autonomous navigation of service robots, inspection robots, etc.; and in the field of map building, it is used to build high-precision maps, such as indoor maps and city maps.

[0023] Figure 1 This application scenario illustrates an embodiment of the present application, using a tunnel as an example. To achieve tunnel inspection and 3D modeling, this embodiment first sets marker points in the tunnel at predetermined intervals or heights, at both ends, the top, or inflection points. These marker points can be designed using a structured approach, meaning they need to have relatively obvious structural features, such as... Figure 1The different shapes of the markers are shown. Furthermore, when placing these markers, materials with a certain reflective intensity can be used to facilitate acquisition and identification by 3D modeling equipment, providing an absolute position reference in the environment. After placing the markers, the tunnel is modeled using LiDAR SLAM through the 3D reconstruction equipment proposed in this embodiment, and the SLAM pose optimization method proposed in this embodiment is executed. Specifically, as... Figure 2 As shown, the method includes: Step S100: Acquire 3D point cloud data of multiple keyframes in the 3D scene; The tunnel is scanned and sampled by the odometer front end of the lidar to obtain three-dimensional point cloud data of the tunnel, and multiple key frames are obtained from the three-dimensional point cloud data.

[0024] The keyframes are representative frames selected from consecutive sensor data frames. Keyframes typically contain rich environmental information, reflecting the environmental characteristics of the lidar at different locations. In this embodiment, keyframes typically include information about marker points in the tunnel.

[0025] To better select keyframes and reduce data processing volume, this embodiment of the application samples the 3D point cloud data of the lidar at preset distances, for example, sampling once every 1 meter. The 3D point cloud data obtained by the interval sampling is accumulated to generate the 3D point cloud data of the keyframe. The keyframe contains a large number of 3D point cloud data points, each with corresponding X, Y, and Z coordinates, as well as intensity and color information. The intensity information represents the reflection intensity of the point cloud data point, indicating the intensity of light reflected back after the laser beam hits the object's surface, reflecting the surface's reflectivity. After linear normalization, the reflection intensity value is typically between 0 and 1, where 0 represents no reflection and 1 represents complete reflection. These 3D point cloud data points can accurately describe the 3D structure of the environment. By selecting keyframes, the amount of data to be processed can be significantly reduced, improving the system's real-time performance and efficiency.

[0026] Step S200: Generate a two-dimensional intensity image based on the three-dimensional coordinates and intensity information of the keyframe; In order to process the 3D point cloud data acquired by the LiDAR using the segmentation model SAM, in this embodiment, the 3D image is first converted into a 2D intensity image, that is, the 3D coordinates and intensity information of the keyframe are converted into a 2D intensity image, and then the 2D intensity image is identified by the segmentation model SAM.

[0027] Specifically, Figure 3 The process of converting the three-dimensional point cloud data into a two-dimensional intensity image is illustrated, including: Step S210: According to the specifications of the lidar, obtain the horizontal field of view (HFOV), vertical field of view (VFOV), horizontal resolution, and vertical resolution of the lidar.

[0028] Step S220: Determine the resolution of the two-dimensional image based on the horizontal resolution and the vertical resolution; The number of pixels in a 2D image is determined based on the horizontal and vertical resolution of the LiDAR. Horizontal resolution is typically expressed as the number of pixels per row, while vertical resolution is expressed as the number of pixels per column.

[0029] Step S230: Determine the actual range of the two-dimensional image in the horizontal and vertical directions based on the horizontal field of view, the vertical field of view, and the resolution of the two-dimensional image.

[0030] The actual range of the two-dimensional image in the horizontal and vertical directions helps to map the three-dimensional point cloud data onto the two-dimensional image. The three-dimensional coordinates of each point cloud data point need to be converted into the pixel position on the two-dimensional image. By determining the actual range of the two-dimensional image in the horizontal and vertical directions, the correct coordinate transformation can be ensured.

[0031] Step S240: Calculate the two-dimensional coordinates of the point cloud data points of the keyframe on the two-dimensional image based on the three-dimensional coordinates and the actual range of the two-dimensional image in the horizontal and vertical directions.

[0032] For each 3D point cloud data point, its corresponding position on the 2D image is calculated based on its 3D coordinates. The 3D coordinates can be mapped to the 2D coordinate system by using polar coordinates or orthogonal projection.

[0033] Step S250: Assign the intensity information corresponding to the point cloud data points of the key frame to the pixel points corresponding to the two-dimensional coordinates to generate a two-dimensional intensity image.

[0034] The intensity information of each point cloud data point is assigned to the corresponding pixel position in the two-dimensional image. The projection of all point cloud data points is completed, resulting in a two-dimensional intensity image. The intensity value of each pixel reflects the reflection intensity of the point cloud data point at the corresponding position.

[0035] After generating a two-dimensional intensity image, further image smoothing or filtering can be performed to improve image quality and reduce image noise.

[0036] The embodiments of this application efficiently convert the three-dimensional coordinates and intensity information of keyframes into two-dimensional intensity images through the above method, laying the foundation for recognizing laser three-dimensional data through segmentation models.

[0037] Step S300: Perform semantic segmentation on the two-dimensional intensity image according to the preset segmentation model to generate multiple masks corresponding to the marker points; In this embodiment, the preferred segmentation model is the SAM (Segment Anything Model). The SAM segmentation model is a powerful segmentation model based on the Transformer architecture, capable of segmenting objects in an image with zero samples, meaning it can accurately segment objects in an image even without any specific class labels. A key feature of the SAM segmentation model is its zero-sample segmentation capability, which means the model can segment objects in an image without any specific class labels. This capability allows the SAM model to adapt to various application scenarios without needing to retrain the model for each scenario. However, its original design was for processing 2D RGB images. In this embodiment, by generating a 2D intensity image from the 3D coordinates and intensity information of the keyframes in step S200, the processing of 3D images using the SAM segmentation model is indirectly achieved.

[0038] When processing a two-dimensional intensity image using the SAM segmentation model, the two-dimensional intensity image corresponding to the keyframe in step S200 is input into the SAM model. The SAM segmentation model outputs the semantic category to which each pixel belongs. For example, different marker points have different semantic categories. A corresponding mask is generated based on the different semantic categories. The mask is a binary image, where each pixel value indicates whether the pixel belongs to a specific semantic landmark. In this embodiment, the two-dimensional intensity image corresponding to the keyframe is semantically segmented using the SAM segmentation model to generate masks corresponding to each marker point.

[0039] Step S400: Project the pixels within the mask back into the 3D scene to generate semantic 3D coordinates corresponding to multiple marker points; After the SAM segmentation model generates masks corresponding to each marker point, the pixels within the masks are again mapped back to the three-dimensional space. For each pixel within the mask, the parameters related to the LiDAR obtained in step S210 are used for reverse transformation to calculate its semantic three-dimensional coordinates in the three-dimensional space. In this embodiment, this can be calculated using methods such as triangulation or depth mapping, which will not be elaborated here.

[0040] Through the above steps S300 and S400, the semantic segmentation of the 3D image by the SAM segmentation model is realized. In the tunnel modeling scenario, prior image features such as marker points are introduced to generate the semantic 3D coordinates corresponding to the marker points. Compared with the traditional thresholding method, the SAM model can extract the 3D coordinates of the marker points more accurately, significantly improving the calculation accuracy of the marker point extraction.

[0041] Step S500: Construct semantic landmark factors based on the observation constraints between the keyframe and the semantic 3D coordinates corresponding to the marker points; After semantic segmentation of the two-dimensional intensity image using the SAM segmentation model and regenerating the semantic three-dimensional coordinates, the semantic three-dimensional coordinates corresponding to the marker points are used as feature prior points, and semantic landmark factors are constructed based on the observation constraints between the keyframes and the semantic three-dimensional coordinates.

[0042] In a SLAM system, the observation constraints refer to the relationship between the data observed by the lidar and the lidar pose or environmental features. These relationships can be used to constrain and optimize lidar pose estimation and environmental map construction.

[0043] Observation constraints typically include observed values, predicted values, and an error term. The observed values ​​represent the data observed by the LiDAR at a certain moment, such as point cloud data scanned by the LiDAR or images captured by a camera. The predicted values ​​are the data that the sensor should observe, predicted based on the LiDAR's pose estimation and the environment map. The error term is the difference between the observed and predicted values, usually represented as a residual. The error term is the objective function in the optimization process; minimizing the error term optimizes the LiDAR's pose and the environment map. The semantic landmark factor is a factor describing the observation constraints between the semantic 3D coordinates corresponding to keyframes and marker points, and is usually represented by the aforementioned error term.

[0044] Furthermore, in this embodiment of the application, in order to establish constraint relationships between keyframes and thus more accurately estimate the pose of the lidar, the following steps are further proposed when constructing semantic landmark factors, specifically as follows: Figure 4 As shown, it includes: Step S510: Find related keyframes with the same marker point in the keyframes; When searching for associated keyframes with the same marker points, feature descriptors, including SIFT, SURF, ORB, etc., can be extracted from each keyframe. Interest points are detected for each keyframe and their descriptors are calculated.

[0045] By using a feature matching algorithm to match feature descriptors between different keyframes, and based on the matching results, finding features that appear repeatedly in different keyframes, i.e. the same marker points, and associating these repeated features, associated keyframes are generated.

[0046] Step S520: Establish pose constraint relationships between the associated keyframes according to the poses of each associated keyframe; After determining the associated keyframes, a relative pose constraint relationship is established between these keyframes with the same marker points, thereby constraining the pose of the lidar and improving the accuracy of pose estimation.

[0047] Step S530: Construct semantic landmark factors based on the pose constraint relationship and the observation constraint.

[0048] When constructing semantic landmark factors, firstly, the actual location of the marker point observed in the associated keyframe is determined, where the actual location is the observed value of the marker point. For example, if marker point j is observed in the Kth frame, and the observed value is... This observation typically represents the actual location observed in keyframe K.

[0049] Secondly, the predicted position of the marker point is generated based on the pose of the associated keyframe and the semantic three-dimensional coordinates of the marker point; Predicted value ,in, Let k be the rotation matrix of keyframe k. Let k be the translation vector of keyframe k. Let j be the semantic 3D coordinates of the marker point j. This predicted value indicates the position of the marker point j estimated based on the pose of the keyframe k.

[0050] Third, the semantic landmark factor is determined based on the error value between the predicted location and the actual location.

[0051] Error term is The error term represents the difference between the predicted and observed values ​​and is the objective function in the optimization process. This error term is used as a semantic landmark factor.

[0052] Step S600: Construct a global factor graph based on the semantic landmark factors; The global factor graph is used to represent the relationship between the lidar pose and environmental features. By optimizing the global factor graph, the accuracy of 3D modeling can be improved.

[0053] When constructing the global factor graph, firstly, variable nodes are defined. These variable nodes include the pose information of the LiDAR and environmental features, where the environmental features include the positions of marker points.

[0054] Secondly, factor nodes need to be defined. These factor nodes include odometry factors and semantic landmark factors corresponding to the odometry front end of the lidar. The odometry factors are used to constrain the relative motion between adjacent keyframes, and a factor node can be defined to constrain the relative pose of two keyframes. The semantic landmark factors, as described in the previous steps, will not be repeated here. Each factor node has an error function representing the difference between the observed and predicted values. In this embodiment, the relative pose between adjacent keyframes is obtained based on the pose information, and the odometry factors of the lidar are determined based on the relative pose.

[0055] Third, a global factor graph is constructed based on the positions of multiple marker points, the pose information of the lidar, the odometry factor, and the semantic landmark factor. In constructing the factor graph, variable nodes and factor nodes are connected to form a graph structure, with each factor node connected to related variable nodes via edges, representing the constraint relationships between these variable nodes.

[0056] Step S700: Optimize the global factor graph using the nonlinear least squares method to optimize the SLAM pose; In this embodiment, the optimization of the global factor graph is transformed into a problem solved using nonlinear least squares. Each factor node in the global factor graph corresponds to an error function, and the optimization objective is to minimize the sum of squares of all error functions.

[0057] Preferably, in this embodiment, the global factor graph is optimized according to a preset Levenberg-Marquardt (LM) algorithm. The LM algorithm has good convergence and robustness and is often used to solve this type of problem. By iteratively updating the pose of the lidar and the position of environmental features, the LM algorithm continuously corrects the three-dimensional semantic coordinates of the marker points according to the observation constraints. At the same time, it uses the corrected position of the marker points to correct the trajectory of the lidar and eliminate drift.

[0058] In summary, the SLAM pose optimization method proposed in this application can improve the localization accuracy of laser SLAM in degraded scenes and enhance the overall accuracy of laser SLAM in complex scenes, making it more robust. Secondly, it innovatively projects disordered 3D laser point clouds into ordered 2D intensity images, enabling the SAM model, which originally could only process images, to be directly applied to lidar data, achieving cross-modal data conversion. Simultaneously, it utilizes the powerful edge segmentation capability of the SAM model to accurately segment the masks corresponding to the marker points on the 2D intensity image, and then back-projects them back into 3D space. This allows for more precise extraction of the 3D coordinates of the marker points using the SAM model, significantly improving the accuracy of landmark calculation. Furthermore, the extracted high-precision marker points are added to the optimization graph as semantic landmark factors, constraining the pose together with odometry factors. This provides constraints when geometric features are missing, significantly suppressing lidar drift and solving problems such as geometric degradation and drift that occur in laser SLAM in scenes such as tunnels and long corridors.

[0059] In other embodiments, such as Figure 4 As shown in the embodiments of this application, a three-dimensional reconstruction device is also proposed. The three-dimensional reconstruction device is used to run the SLAM pose optimization method proposed in the above embodiments. The three-dimensional reconstruction device may include: a processor 402, a memory 406, a communication interface 404, and a communication bus 408.

[0060] The processor 402, memory 406, and communication interface 404 communicate with each other via communication bus 408. The memory 406 stores at least one program 410, which causes the processor 402 to execute steps related to the SLAM pose optimization method proposed in this application embodiment.

[0061] Specifically, program 410 may include program code, which includes computer-executable instructions.

[0062] Processor 402 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The one or more processors included in the 3D reconstruction device may be processors of the same type, such as one or more CPUs; or they may be processors of different types, such as one or more CPUs and one or more ASICs.

[0063] Memory 406 is used to store program 410. Memory 406 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0064] Specifically, program 410 can be called by processor 402 to cause the 3D reconstruction device to execute the above-described SLAM pose optimization method proposed in the embodiments of this application, which will not be repeated here.

[0065] This application also provides a computer-readable storage medium storing executable instructions. When the executable instructions are run on a 3D reconstruction device, the 3D reconstruction device performs the SLAM pose optimization method provided in any of the above embodiments.

[0066] This application also provides a SLAM pose optimization program, which is used to execute the SLAM pose optimization method provided in the above embodiments.

[0067] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is apparent from the above description. Furthermore, the embodiments of this application are not directed to any particular programming language. It should be understood that the content of this application described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of this application.

[0068] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0069] Similarly, it should be understood that, in order to simplify this application and aid in understanding one or more of the various aspects of the invention, in the above description of exemplary embodiments of this application, various features of the embodiments of this application are sometimes grouped together into a single embodiment, figure, or description thereof.

[0070] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying abstract and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying abstract and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0071] It should be noted that the above embodiments are illustrative of this application and not restrictive, and those skilled in the art can design alternative embodiments without departing from the scope. Unless otherwise specified, the steps in the above embodiments should not be construed as limiting the order of execution.

Claims

1. A SLAM pose optimization method, characterized in that, Applied to lidar, the method includes: Acquire 3D point cloud data of multiple keyframes in a 3D scene, wherein the 3D scene includes multiple preset marker points, the marker points are made of reflective material and are used to provide absolute position references, and the 3D point cloud data includes 3D coordinates and intensity information; A two-dimensional intensity image is generated based on the three-dimensional coordinates and intensity information of the keyframe; The two-dimensional intensity image is semantically segmented according to a preset segmentation model to generate multiple masks corresponding to the marker points; The pixels within the mask are back-projected back into the 3D scene to generate semantic 3D coordinates corresponding to multiple marker points; Construct semantic landmark factors based on the observation constraints between the keyframe and the semantic 3D coordinates corresponding to the marker points; Construct a global factor graph based on the semantic landmark factors; The global factor graph is optimized using the nonlinear least squares method; The step of constructing semantic landmark factors based on the observation constraints between the keyframe and the semantic 3D coordinates corresponding to the marker points includes: Find related keyframes with the same marker point in the keyframes; Establish pose constraint relationships between the associated keyframes based on the poses of each associated keyframe; Semantic landmark factors are constructed based on the pose constraints and the observation constraints.

2. The method according to claim 1, characterized in that, The acquisition of 3D point cloud data of multiple keyframes in a 3D scene includes: The three-dimensional point cloud data of the lidar is sampled at intervals according to a preset distance; The 3D point cloud data obtained by the interval sampling is accumulated to generate the 3D point cloud data of the key frame.

3. The method according to claim 1, characterized in that, The step of generating a two-dimensional intensity image based on the three-dimensional coordinates and intensity information of the keyframe includes: Obtain the horizontal field of view, vertical field of view, horizontal resolution, and vertical resolution of the lidar; The resolution of the two-dimensional image is determined based on the horizontal resolution and the vertical resolution; The actual range of the two-dimensional image in the horizontal and vertical directions is determined based on the horizontal field of view, the vertical field of view, and the resolution of the two-dimensional image.

4. The method according to claim 3, characterized in that, The step of generating a two-dimensional intensity image based on the three-dimensional coordinates and intensity information of the keyframe further includes: Based on the three-dimensional coordinates and the actual range of the two-dimensional image in the horizontal and vertical directions, calculate the two-dimensional coordinates of the point cloud data points of the key frame on the two-dimensional image. The intensity information corresponding to the point cloud data points of the key frame is assigned to the pixel points corresponding to the two-dimensional coordinates to generate a two-dimensional intensity image.

5. The method according to claim 1, characterized in that, Based on the pose constraints and the observation constraints, semantic landmark factors are constructed, including: Determine the actual location of the marker point observed in the associated keyframe; The predicted position of the marker point is generated based on the pose of the associated keyframe and the semantic 3D coordinates of the marker point; The semantic landmark factor is determined based on the error between the predicted location and the actual location.

6. The method according to claim 5, characterized in that, The step of constructing a global factor graph based on the semantic landmark factors includes: Obtain the pose information of the lidar; The relative pose between adjacent keyframes is obtained based on the pose information, and the odometry factor of the lidar is determined based on the relative pose. A global factor graph is constructed based on the positions of multiple marker points, the pose information of the lidar, the odometer factor, and the semantic landmark factor.

7. The method according to claim 1, characterized in that, The optimization of the global factor graph using the nonlinear least squares method includes: The global factor graph is optimized according to the preset Levenberg-Marquardt algorithm.

8. A three-dimensional reconstruction device, characterized in that, include: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one program that causes the processor to execute the SLAM pose optimization method as described in any one of claims 1-7.

9. A readable computer storage medium, characterized in that, The storage medium stores at least one program, which, when run on a 3D reconstruction device, causes the 3D reconstruction device to perform the SLAM pose optimization method as described in any one of claims 1-7.

Images