Unmanned loader positioning mapping method, device, equipment and medium
By using a drone equipped with a lidar and RGB camera combined with IMU data, and employing 3D Gaussian splashing and volume rendering, the problem of scale drift and unstable attitude estimation of unmanned loaders in complex environments was solved, achieving high-precision 3D map construction and improving positioning accuracy and operational efficiency.
Patent Information
- Application Number
- CN202511713369.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-20
- Publication Date
- 2026-02-24
AI Technical Summary
Existing localization and mapping methods for unmanned loaders are prone to scale drift and unstable attitude estimation in complex outdoor environments, making it difficult to meet the requirements for high-precision real-time mapping.
By using a drone equipped with a LiDAR to collect point cloud data and an RGB camera to collect image data, combined with IMU motion data, and employing 3D Gaussian splashing and volume rendering methods, a high-precision 3D map is generated through tight coupling fusion and optimization.
It significantly improves the positioning accuracy and map integrity of unmanned loaders in dynamic and complex environments, supports autonomous path planning and obstacle avoidance, and enhances operational safety and automation.
Smart Images
Figure CN121564260A_ABST
Abstract
Description
Technical Field
[0001] This application relates to positioning and mapping technology for unmanned loaders, and more particularly to a method, apparatus, equipment and medium for positioning and mapping of unmanned loaders. Background Technology
[0002] Localization and mapping of unmanned loaders (UAVs) are fundamental supporting technologies for achieving autonomous operation, safe obstacle avoidance, and efficient scheduling. By constructing high-precision 3D maps in real time and accurately estimating its own pose in complex outdoor environments, UAVs can maintain stable perception and autonomous decision-making capabilities in the presence of dynamic obstacles, strong light interference, dust obstruction, and undulating terrain. This enables autonomous planning of material handling paths, environmental perception of the work area, and collaborative operation of multiple devices, significantly improving operational safety, automation level, and overall operational efficiency.
[0003] In existing technologies, the commonly used localization and mapping method for unmanned loaders is to use Simultaneous Localization and Mapping (SLAM). For example, the Fast LiDAR-Inertial Odometry 2 (Fast-LIO2) system based on point cloud data works as follows: continuous point cloud data is collected by LiDAR, motion compensation is performed using an inertial measurement unit (IMU), and front-end feature matching and back-end factor map optimization are performed within a sliding window to achieve real-time localization and local map construction of the unmanned loader.
[0004] However, existing positioning and mapping methods for unmanned loaders are prone to scale drift and unstable attitude estimation, making it difficult to meet the high-precision real-time mapping requirements of unmanned loaders in complex outdoor environments. Summary of the Invention
[0005] This application provides a method, apparatus, equipment, and medium for unmanned loader positioning and mapping, which solves the problems of scale drift and unstable attitude estimation in the prior art, making it difficult to meet the high-precision real-time mapping requirements of unmanned loaders in complex outdoor environments.
[0006] In a first aspect, embodiments of this application provide a method for positioning and mapping an unmanned loader, including:
[0007] Point cloud data of the area to be mapped, collected by the laser point cloud of the UAV, and image data of the area to be mapped, collected by the RGB camera;
[0008] The point cloud data is projected onto the image data to obtain depth map data;
[0009] An initial Gaussian map is obtained by sampling Gaussian primitives based on the depth map and setting initial parameters using a 3D Gaussian splashing method.
[0010] The initial Gaussian map is rendered and optimized using volumetric rendering to obtain a 3D map corresponding to the region to be mapped.
[0011] In one possible implementation, the method further includes:
[0012] While the drone collects point cloud data, motion data is collected via IMU;
[0013] The relative attitude transformation is performed based on the motion data to obtain relative attitude data;
[0014] Accordingly, the rendering optimization of the initial Gaussian map using volume rendering includes:
[0015] A volume rendering approach is adopted, based on the Gaussian primitives in the initial Gaussian map through tile rasterization iteration, and motion constraints are performed based on the relative pose data.
[0016] In one possible implementation, the volume rendering method iterates over Gaussian primitives in the initial Gaussian map based on tiles and applies motion constraints based on the relative pose data, including:
[0017] Based on the pre-constructed loss function and the objective of minimizing the loss value, a volume rendering method and a pyramid training method are used to iterate the Gaussian primitives in the initial Gaussian map based on tile rasterization.
[0018] The loss function includes a photometric loss function and a depth consistency loss function, the value of which is determined based on the vibration offset in the relative attitude data.
[0019] In one possible implementation, the photometric loss function is:
[0020]
[0021] in, This is the weighting factor for photometric loss; This is a photometric error term based on pixel differences; Based on Gaussian primitive G and current pose The rendered image; I represents the image consistency error term based on structural similarity; I represents the image data.
[0022] In one possible implementation, the deep consistency loss function is:
[0023]
[0024] Where D is the rendered depth map; A depth map generated from the point cloud data; The compensation amount, determined based on the vibration offset in the relative attitude data, is used to correct the depth error caused by vibration.
[0025] In one possible implementation, after optimizing the initial Gaussian map using volume rendering, the method further includes:
[0026] Based on the minimization objective of the loss function, the pose parameters of the current RGB camera are optimized to obtain the optimized pose parameters;
[0027] When the change between the optimized attitude parameters and the attitude parameters of the previous frame is greater than a preset threshold, the current frame is determined as a key frame.
[0028] The Gaussian primitives corresponding to the keyframes are added to the initial Gaussian map to update the 3D map.
[0029] In one possible implementation, before projecting the point cloud data onto the image to obtain the depth map, the method further includes:
[0030] The point cloud data, the image data, and the motion data are time-aligned.
[0031] Secondly, embodiments of this application provide an unmanned loader positioning and mapping device, comprising:
[0032] The acquisition module is used to acquire point cloud data of the area to be mapped by the laser point cloud of the UAV, and image data of the area to be mapped by the RGB camera.
[0033] The acquisition module is used to project the point cloud data onto the image data to acquire depth map data;
[0034] The processing module is used to obtain an initial Gaussian map by sampling Gaussian primitives based on the depth map and setting initial parameters using a 3D Gaussian splashing method.
[0035] The rendering module is used to optimize the initial Gaussian map using volume rendering to obtain a 3D map corresponding to the area to be mapped.
[0036] In one possible implementation, the unmanned loader positioning and mapping device further includes an optimization module for:
[0037] While the drone collects point cloud data, motion data is collected via IMU;
[0038] The relative attitude transformation is performed based on the motion data to obtain relative attitude data;
[0039] Accordingly, the rendering optimization of the initial Gaussian map using volume rendering includes:
[0040] A volume rendering approach is adopted, based on the Gaussian primitives in the initial Gaussian map through tile rasterization iteration, and motion constraints are performed based on the relative pose data.
[0041] In one possible implementation, the optimization module is configured to:
[0042] Based on the pre-constructed loss function and the objective of minimizing the loss value, a volume rendering method and a pyramid training method are used to iterate the Gaussian primitives in the initial Gaussian map based on tile rasterization.
[0043] The loss function includes a photometric loss function and a depth consistency loss function, the value of which is determined based on the vibration offset in the relative attitude data.
[0044] In one possible implementation, the photometric loss function is:
[0045]
[0046] in, This is the weighting factor for photometric loss; This is a photometric error term based on pixel differences; Based on Gaussian primitive G and current pose The rendered image; I represents the image consistency error term based on structural similarity; I represents the image data.
[0047] In one possible implementation, the deep consistency loss function is:
[0048]
[0049] Where D is the rendered depth map; A depth map generated from the point cloud data; The compensation amount, determined based on the vibration offset in the relative attitude data, is used to correct the depth error caused by vibration.
[0050] In one possible implementation, the processing apparatus is further configured to:
[0051] Based on the minimization objective of the loss function, the pose parameters of the current RGB camera are optimized to obtain the optimized pose parameters;
[0052] When the change between the optimized attitude parameters and the attitude parameters of the previous frame is greater than a preset threshold, the current frame is determined as a key frame.
[0053] The Gaussian primitives corresponding to the keyframes are added to the initial Gaussian map to update the 3D map.
[0054] In one possible implementation, before projecting the point cloud data onto the image to obtain the depth map, the acquisition module is further configured to:
[0055] The point cloud data, the image data, and the motion data are time-aligned.
[0056] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;
[0057] The memory stores computer-executed instructions;
[0058] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.
[0059] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.
[0060] The unmanned loader positioning and mapping method, apparatus, equipment, and medium provided in this application firstly collect point cloud data of the area to be mapped using a lidar mounted on the UAV to obtain high-precision three-dimensional geometric information. Simultaneously, an RGB camera collects image data of the same area to provide texture and color information. Then, the collected point cloud data is projected onto the camera plane to generate depth map data, achieving spatial alignment between the three-dimensional geometry and the two-dimensional image, ensuring the synchronous fusion of multimodal information. Next, a 3D Gaussian splashing method is used to sample Gaussian primitives in the depth map, and a setting is applied to each Gaussian primitive. The corresponding parameters are used to construct an initial Gaussian map, which uses anisotropic Gaussian to represent complex terrain and uneven ground, improving the modeling accuracy of structures such as slopes and material piles. Finally, the initial Gaussian map is optimized using volume rendering. By iteratively calculating the rendered image and depth map and minimizing photometric and depth consistency losses, the position, attitude, and color of Gaussian primitives are optimized and updated, thereby generating a high-precision, continuous, and differentiable 3D map. This significantly improves the positioning accuracy and map integrity of UAVs in dynamic and complex environments, providing a reliable data foundation for subsequent path planning and obstacle avoidance. Attached Figure Description
[0061] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0062] Figure 1 A flowchart illustrating the unmanned loader positioning and mapping method provided in this application embodiment. Figure 1 ;
[0063] Figure 2 A flowchart illustrating the unmanned loader positioning and mapping method provided in this application embodiment. Figure 2 ;
[0064] Figure 3 This is a schematic diagram of the unmanned loader positioning and mapping device provided in the embodiments of this application;
[0065] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.
[0066] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation
[0067] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.
[0068] Localization and mapping of unmanned loaders (UAVs) are fundamental supporting technologies for achieving autonomous operation, safe obstacle avoidance, and efficient scheduling. By constructing high-precision 3D maps in real time and accurately estimating its own pose in complex outdoor environments, UAVs can maintain stable perception and autonomous decision-making capabilities in the presence of dynamic obstacles, strong light interference, dust obstruction, and undulating terrain. This enables autonomous planning of material handling paths, environmental perception of the work area, and collaborative operation of multiple devices, significantly improving operational safety, automation level, and overall operational efficiency.
[0069] In existing technologies, the commonly used method for localization and mapping of unmanned loaders is to use SLAM, such as Fast-LIO2 based on point cloud data. Specifically, continuous point cloud data is collected by lidar, motion compensation is performed using IMU, and front-end feature matching and back-end factor map optimization are performed within a sliding window to achieve real-time localization and local map construction of the unmanned loader.
[0070] However, existing localization and mapping methods for unmanned loaders suffer from several drawbacks. First, the point cloud quality deteriorates under dusty, highly reflective, or strong light interference conditions, which can easily lead to feature matching failures. Second, the loosely coupled IMU constraint and point cloud optimization methods are prone to scale drift and unstable attitude estimation under heavy load vibration or sharp turning conditions, making it difficult to meet the high-precision real-time mapping requirements of unmanned loaders in complex outdoor environments.
[0071] Based on this, this application proposes a localization and mapping method for unmanned loaders. Addressing the problems of point cloud occlusion, visual interference, and loose coupling leading to drift in complex outdoor environments, existing technologies employ this method. The inventors conceived of a method that tightly couples and fuses LiDAR point clouds, RGB images, and IMU motion data, utilizing anisotropic Gaussian primitives for this tight coupling and fusion on a unified 3D Gaussian splash map. Simultaneously, volume rendering and differentiable optimization are used for attitude tracking and map construction. This improves localization accuracy and map integrity under dynamic obstacles, strong light and dust, and heavy-load vibration conditions, ensuring the unmanned loader can generate high-precision 3D maps in real time to support autonomous path planning and obstacle avoidance. This effectively solves the problems of accuracy degradation and drift in traditional SLAM under complex environments.
[0072] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.
[0073] Figure 1 A flowchart illustrating the positioning and mapping method for the unmanned loader provided in this application embodiment. Figure 1 ;like Figure 1 As shown, the method includes:
[0074] S101. Point cloud data of the area to be mapped, collected by the laser point cloud of the UAV, and image data of the area to be mapped, collected by the RGB camera.
[0075] In one feasible approach, motion data is acquired via an IMU while the UAV is collecting point cloud data; then, relative attitude transformation is performed based on the motion data to obtain relative attitude data.
[0076] It should be understood that in this embodiment, 3D point cloud data of the area to be mapped is acquired using a LiDAR system mounted on the UAV, while image data from corresponding viewpoints is acquired using an RGB camera to provide texture information of the scene. During point cloud and image acquisition, the IMU module records the UAV's acceleration and angular velocity information in real time to reflect changes in flight attitude. The inertial data over continuous time intervals is integrated using an IMU pre-integration algorithm to obtain the relative attitude transformation matrix, thereby enabling continuous estimation of the UAV's trajectory.
[0077] Understandably, this method enables the simultaneous acquisition and fusion of geometric, textural, and motion information, making subsequent data alignment and Gaussian mapping more accurate and stable. For example, when a UAV rapidly dives to scan a material pile, the IMU can compensate for drift caused by sudden attitude changes, ensuring accurate point cloud projection positions.
[0078] S102. Project the point cloud data onto the image data to obtain depth map data.
[0079] It should be noted that before projecting point cloud data onto an image to obtain a depth map, the point cloud data, image data, and motion data need to be time-aligned.
[0080] It should be understood that after data acquisition, the system performs time alignment and coordinate unification on data from different sensors. First, it synchronizes LiDAR point cloud (10Hz), RGB image (30Hz), and IMU data (200Hz) based on timestamps, achieving precise spatiotemporal registration using interpolation and extrinsic parameter calibration. Then, the point cloud data is projected onto the camera plane using a projection function to generate a pseudo-depth map. This depth map reflects the geometric distribution of the scene from the camera's perspective, providing a foundation for subsequent Gaussian primitive sampling.
[0081] Specifically, the projection function is:
[0082] Dlidar(u,v)=Proj(PW,[Q iw ,T iw ])
[0083] Among them, P W For the point cloud in the world coordinate system, [Q iw ,T iw [This refers to the camera's orientation.]
[0084] Understandably, this step achieves tight coupling and fusion of multi-source data, significantly improving the consistency between geometric and visual information. For example, when a drone takes a tilted image above a building, the depth map accurately reflects the depth gradient of the building facade, avoiding errors caused by inferring depth from a simple image.
[0085] S103. Using the 3D Gaussian splashing method, Gaussian primitives are sampled based on the depth map and initial parameters are set to obtain the initial Gaussian map.
[0086] It should be understood that, in this embodiment, a corresponding Gaussian primitive is generated by sampling each pixel based on the depth map. Each Gaussian primitive includes a center position µ, color c, opacity o, scale vector S, and rotation matrix R. The initial parameters are determined by the depth value for position, by the camera color for c, and by the local normal approximation for R. All Gaussian primitives together constitute the initial 3D Gaussian map.
[0087] The Gaussian distribution is defined as follows:
[0088]
[0089] The covariance matrix is defined as:
[0090] in, The scale vector For rotation matrix, As the central location, .
[0091] Understandably, this representation method allows for smoother surfaces and more continuous boundary effects with the same number of points, facilitating subsequent volume rendering and gradient optimization. For example, in areas with uneven material piles, the system automatically generates Gaussian primitives of different directions and scales to realistically reflect surface undulations and improve geometric fidelity.
[0092] S104. Use volume rendering to optimize the initial Gaussian map and obtain the 3D map corresponding to the area to be mapped.
[0093] In one possible approach, the pose parameters of the current RGB camera are first optimized based on the goal of minimizing the loss function to obtain the optimized pose parameters. Then, when the change between the optimized pose parameters and the pose parameters of the previous frame is greater than a preset threshold, the current frame is determined as a key frame. Finally, the Gaussian primitives corresponding to the key frame are added to the initial Gaussian map to update the 3D map.
[0094] It should be understood that, in this embodiment, volume rendering is used to optimize the rendering of the initial Gaussian map to generate a high-precision 3D map corresponding to the region to be mapped. Specifically, firstly, based on the objective of minimizing the loss function, the RGB camera pose parameters of the current frame are optimized. The system compares the simulated image generated by the Gaussian map rendering with the measured RGB image, calculates the photometric consistency loss and depth consistency loss, and constructs a joint optimization objective function:
[0095]
[0096] in, L′ is a photometric loss used to constrain the color consistency between the rendered image and the real image. d This is the depth consistency loss, used to constrain the geometric consistency between the rendered depth and the depth map generated by LiDAR projection; This is the balance coefficient between the two. The camera pose parameters are calculated using the backpropagation algorithm. Iterative optimization is performed to obtain the optimal attitude estimation result.
[0097] When the rotational or translational change between the optimized attitude parameters and the attitude parameters of the previous frame exceeds a preset threshold (denoted as ), When a frame is found to be redundant, the system marks it as a keyframe. Selecting keyframes avoids redundant calculations and improves overall optimization efficiency. Subsequently, the Gaussian primitives corresponding to the keyframes are extracted and integrated into the initial Gaussian map to form an updated global 3D map.
[0098] During map updates, the system will adjust the opacity based on the Gaussian primitives. With visibility A screening process is performed to remove duplicate or low-contribution primitives, ensuring the sparsity and coherence of the map structure. Simultaneously, to address vibrations or minor attitude disturbances caused by the UAV during mapping, a vibration compensation term based on IMU pre-integration is introduced into the loss function. Dynamic correction of depth offset is performed to reduce scale drift caused by carrier vibration.
[0099] Among them, visibility The specific formula is:
[0100]
[0101] in, For opacity, Visibility.
[0102] Understandably, this method can optimize camera pose and Gaussian primitive parameters in real time during multi-view scanning by UAVs, thereby significantly improving the accuracy and consistency of 3D maps. Compared with traditional SLAM methods based on point cloud stitching, this embodiment can effectively reduce scale drift and pose accumulation errors. Simultaneously, due to the use of anisotropic Gaussian primitives to express structural features, the final generated 3D map exhibits smoother and more continuous detail in areas such as edges and curved surfaces. For example, when UAVs map material piles or building facades, the volume rendering optimization in step S104 can automatically adjust the camera pose and Gaussian primitive shape, resulting in 3D reconstruction results with rich surface details and consistent texture, providing a high-precision environmental model for subsequent path planning or construction monitoring.
[0103] The unmanned loader positioning and mapping method provided in this application firstly collects point cloud data of the area to be mapped using a lidar mounted on the UAV to obtain high-precision 3D geometric information. Simultaneously, an RGB camera collects image data of the same area to provide texture and color information. Next, the collected point cloud data is projected onto the camera plane to generate depth map data, achieving spatial alignment between the 3D geometry and the 2D image, ensuring synchronous fusion of multimodal information. Then, a 3D Gaussian splashing method is used to sample Gaussian primitives in the depth map, and corresponding parameters are set for each Gaussian primitive to construct an initial Gaussian map. Anisotropic Gaussians represent complex terrain and uneven ground, improving the map's modeling accuracy for structures such as slopes and material piles. Finally, volume rendering is used to optimize the initial Gaussian map. By iteratively calculating the rendered image and depth map and minimizing photometric and depth consistency losses, the position, attitude, and color of the Gaussian primitives are optimized and updated, generating a high-precision, continuous, and differentiable 3D map. This significantly improves the positioning accuracy and map integrity of the UAV in dynamic and complex environments, providing a reliable data foundation for subsequent path planning and obstacle avoidance.
[0104] Figure 2 A flowchart illustrating the unmanned loader positioning and mapping method provided in this application embodiment. Figure 2 ,like Figure 2 As shown, in this embodiment... Figure 1 Based on the examples, a detailed explanation is provided on optimizing the rendering of the initial Gaussian map using volume rendering. This method includes:
[0105] S201. While the UAV is collecting point cloud data, motion data is also collected via IMU.
[0106] It should be understood that, in this embodiment, while the UAV is collecting point cloud data of the area to be mapped, the inertial measurement unit (IMU) collects the UAV's linear acceleration and angular velocity data in real time to reflect the UAV's motion state during the mapping process. The IMU can continuously output data at high frequencies (e.g., 200Hz), thus enabling the complete recording of the UAV's attitude change trajectory even when the LiDAR and camera frame rates are low (e.g., 10Hz to 30Hz).
[0107] Understandably, by synchronously acquiring LiDAR point clouds and RGB images, IMU data can effectively compensate for timing deviations caused by differences in the sampling frequencies of multiple sensors, thereby improving the fusion accuracy of multimodal data.
[0108] S202. Perform relative attitude transformation processing based on motion data to obtain relative attitude data.
[0109] It should be understood that after acquiring motion data, the system uses the acceleration and angular velocity information output by the IMU to calculate the relative attitude transformation between consecutive time points using the IMU pre-integration algorithm, thus obtaining relative attitude data. .in, This represents the rotation quaternion of the drone relative to the world coordinate system at the current moment. This represents the translation vector.
[0110] The specific formula for calculating the relative attitude change between consecutive time steps using the IMU pre-integration algorithm is as follows:
[0111]
[0112] in, For IMU acceleration and angular velocity, For time intervals.
[0113] Understandably, this processing achieves smoothing of attitude estimation and drift suppression by integrating and denoising the raw IMU data within a time window. The system can also correct the IMU integration results based on absolute pose information fed back from external sensors (such as LiDAR or cameras), thereby obtaining more accurate relative attitude data.
[0114] S203. A volume rendering method is adopted, based on the Gaussian primitives in the initial Gaussian map through tile rasterization iteration, and motion constraints are performed based on relative pose data.
[0115] In one possible approach, based on a pre-built loss function and a loss minimization objective, a volume rendering method and a pyramid training method are used, and Gaussian primitives in the initial Gaussian map are iterated based on tile rasterization.
[0116] The loss function includes a photometric loss function and a depth consistency loss function. The value of the depth consistency loss function is determined based on the vibration offset in the relative attitude data.
[0117] It should be noted that the photometric loss function is:
[0118]
[0119] in, This is the weighting factor for photometric loss; This is a photometric error term based on pixel differences; Based on Gaussian primitive G and current pose The rendered image; I represents the image consistency error term based on structural similarity; I represents the image data.
[0120] The deep consistency loss function is:
[0121]
[0122] Where D is the rendered depth map; Depth map generated from point cloud data; This is a compensation amount determined based on the vibration offset in the relative attitude data, used to correct the depth error caused by vibration.
[0123] It should be understood that after obtaining the relative pose data, a volumetric rendering approach is adopted, using a tile-based rasterization strategy to iteratively update the Gaussian primitives in the initial Gaussian map tile by tile. During the rendering process, motion constraints are applied based on the relative pose data to ensure that the projection of the Gaussian primitives maintains spatiotemporal consistency under pose changes. Simultaneously, the system employs a pyramid multi-scale optimization strategy during training, i.e., progressively optimizing the rendering results from low resolution to high resolution. Lower-level stages focus on global structural consistency, while higher-level stages refine local textures and geometric details.
[0124] Understandably, this step enables dynamic optimization and high-fidelity reconstruction of the initial Gaussian map from multiple perspectives. Compared to traditional SLAM systems based on sparse feature matching, this method, through joint optimization of volume rendering and pose constraints, not only reduces reconstruction errors but also improves the system's real-time performance and convergence speed.
[0125] Figure 3 This is a schematic diagram of the unmanned loader positioning and mapping device provided in the embodiments of this application; as shown below. Figure 3 As shown, the device includes:
[0126] The acquisition module 301 is used to acquire point cloud data of the area to be mapped by the laser point cloud of the UAV, and image data of the area to be mapped by the RGB camera.
[0127] The acquisition module 302 is used to project point cloud data onto image data to obtain depth map data;
[0128] Processing module 303 is used to obtain an initial Gaussian map by sampling Gaussian primitives based on the depth map and setting initial parameters using a 3D Gaussian splashing method.
[0129] The rendering module 304 is used to optimize the initial Gaussian map using volume rendering to obtain a 3D map corresponding to the area to be mapped.
[0130] In one possible implementation, the unmanned loader positioning and mapping device further includes an optimization module for:
[0131] While the drone collects point cloud data, motion data is collected via IMU;
[0132] Relative attitude data is obtained by performing relative attitude transformation processing on motion data.
[0133] Accordingly, volume rendering is used to optimize the initial Gaussian map, including:
[0134] A volume rendering approach is adopted, which uses Gaussian primitives in the initial Gaussian map based on tile rasterization iteration and performs motion constraints based on relative pose data.
[0135] In one possible implementation, the optimization module is used for:
[0136] Based on the pre-built loss function and the objective of minimizing the loss value, a volume rendering method and a pyramid training method are used, and Gaussian primitives in the initial Gaussian map are iterated based on tile rasterization.
[0137] The loss function includes a photometric loss function and a depth consistency loss function. The value of the depth consistency loss function is determined based on the vibration offset in the relative attitude data.
[0138] In one possible implementation, the photometric loss function is:
[0139]
[0140] in, This is the weighting factor for photometric loss; This is a photometric error term based on pixel differences; Based on Gaussian primitive G and current pose The rendered image; I represents the image consistency error term based on structural similarity; I represents the image data.
[0141] In one possible implementation, the deep consistency loss function is:
[0142]
[0143] Where D is the rendered depth map; Depth map generated from point cloud data; This is a compensation amount determined based on the vibration offset in the relative attitude data, used to correct the depth error caused by vibration.
[0144] In one possible implementation, the processing device 303 is further configured to:
[0145] Based on the minimization objective of the loss function, the pose parameters of the current RGB camera are optimized to obtain the optimized pose parameters;
[0146] When the change between the optimized pose parameters and the pose parameters of the previous frame is greater than a preset threshold, the current frame is determined as a key frame.
[0147] Add the Gaussian primitives corresponding to the keyframes to the initial Gaussian map to update the 3D map.
[0148] In one possible implementation, before projecting the point cloud data onto the image and obtaining the depth map, the acquisition module 302 is further configured to:
[0149] Time alignment is performed on point cloud data, image data, and motion data.
[0150] The unmanned loader positioning and mapping device provided in this application embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.
[0151] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 4 As shown, the electronic device 40 provided in this embodiment includes at least one processor 401 and a memory 402. Optionally, the device 40 further includes a communication component 403. The processor 401, memory 402, and communication component 403 are connected via a bus 404.
[0152] In a specific implementation, at least one processor 401 executes computer execution instructions stored in memory 402, causing at least one processor 401 to perform the above-described method.
[0153] The specific implementation process of processor 401 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.
[0154] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.
[0155] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.
[0156] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.
[0157] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.
[0158] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.
[0159] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.
[0160] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.
[0161] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0162] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0163] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0164] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.
[0165] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.
Claims
1. A method for positioning and mapping an unmanned loader, characterized in that, include: Point cloud data of the area to be mapped, collected by the laser point cloud of the UAV, and image data of the area to be mapped, collected by the RGB camera; The point cloud data is projected onto the image data to obtain depth map data; An initial Gaussian map is obtained by sampling Gaussian primitives based on the depth map and setting initial parameters using a 3D Gaussian splashing method. The initial Gaussian map is rendered and optimized using volumetric rendering to obtain a 3D map corresponding to the region to be mapped.
2. The method according to claim 1, characterized in that, The method further includes: While the drone collects point cloud data, motion data is collected via IMU; The relative attitude transformation is performed based on the motion data to obtain relative attitude data; Accordingly, the rendering optimization of the initial Gaussian map using volume rendering includes: A volume rendering approach is adopted, based on the Gaussian primitives in the initial Gaussian map through tile rasterization iteration, and motion constraints are performed based on the relative pose data.
3. The method according to claim 2, characterized in that, The method employs volume rendering, iterating through Gaussian primitives in the initial Gaussian map based on tiles, and applying motion constraints based on the relative pose data, including: Based on the pre-constructed loss function and the objective of minimizing the loss value, a volume rendering method and a pyramid training method are used to iterate the Gaussian primitives in the initial Gaussian map based on tile rasterization. The loss function includes a photometric loss function and a depth consistency loss function, the value of which is determined based on the vibration offset in the relative attitude data.
4. The method according to claim 3, characterized in that, The photometric loss function is: in, This is the weighting factor for photometric loss; This is a photometric error term based on pixel differences; Based on Gaussian primitive G and current pose The rendered image; I represents the image consistency error term based on structural similarity; I represents the image data.
5. The method according to claim 3, characterized in that, The depth consistency loss function is: Where D is the rendered depth map; A depth map generated from the point cloud data; The compensation amount, determined based on the vibration offset in the relative attitude data, is used to correct the depth error caused by vibration.
6. The method according to any one of claims 1 to 5, characterized in that, After optimizing the initial Gaussian map using volume rendering, the method further includes: Based on the minimization objective of the loss function, the pose parameters of the current RGB camera are optimized to obtain the optimized pose parameters; When the change between the optimized attitude parameters and the attitude parameters of the previous frame is greater than a preset threshold, the current frame is determined as a key frame. The Gaussian primitives corresponding to the keyframes are added to the initial Gaussian map to update the 3D map.
7. The method according to claim 2, characterized in that, Before projecting the point cloud data onto the image to obtain the depth map, the method further includes: The point cloud data, the image data, and the motion data are time-aligned.
8. A positioning and mapping device for an unmanned loader, characterized in that, include: The acquisition module is used to acquire point cloud data of the area to be mapped by the laser point cloud of the UAV, and image data of the area to be mapped by the RGB camera. The acquisition module is used to project the point cloud data onto the image data to acquire depth map data; The processing module is used to obtain an initial Gaussian map by sampling Gaussian primitives based on the depth map and setting initial parameters using a 3D Gaussian splashing method. The rendering module is used to optimize the initial Gaussian map using volume rendering to obtain a 3D map corresponding to the area to be mapped.
9. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-7.