Unmanned aerial vehicle positioning method, apparatus, device, medium and program product

Abstract

This application relates to the field of UAV positioning technology, and provides a UAV positioning method, device, equipment, medium, and program product. The method includes: synchronously triggering a LiDAR, an RGB camera, and an IMU to acquire LiDAR point cloud sequences, RGB image sequences, dense depth maps, effective projection masks, and IMU pre-integration measurements, and constructing point cloud geometric feature vectors, visual feature vectors, and pixel point cloud spatial pairing indexes; performing attention analysis on the point cloud geometric feature vectors and visual feature vectors based on the pixel point cloud spatial pairing indexes to obtain fused feature descriptors and fused feature matching pairs; solving a six-degree-of-freedom pose sequence based on the fused feature matching pairs and the IMU pre-integration measurements, and generating a semantic 3D map based on the fused feature descriptors and the six-degree-of-freedom pose sequence. This invention ensures map quality and the reliability of path planning.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) positioning technology, and in particular to a UAV positioning method, device, equipment, medium, and program product. Background Technology

[0002] Autonomous localization and mapping of unmanned aerial vehicles (UAVs) in complex environments is a core technological challenge in the field of intelligent flight. Existing single-LiDAR SLAM solutions suffer from sparse edge and planar point features in open areas or scenes with repetitive geometric structures, leading to degraded point-to-surface matching constraints and a sharp increase in accumulated odometry drift. Meanwhile, single-vision SLAM solutions experience high failure rates in feature point detection and matching in environments with weak textures, low light, or dense smoke and dust, and suffer from inherent scale uncertainties, making it difficult to maintain long-term positioning accuracy in the absence of GPS. Current technologies lack real-time recognition and removal mechanisms for dynamic obstacles, resulting in numerous artifacts generated during point cloud stitching, severely interfering with map quality and the reliability of subsequent path planning. Summary of the Invention

[0003] This invention provides a method, apparatus, device, medium, and program product for UAV positioning. This invention ensures strict temporal and spatial correspondence between laser and visual data from the source; it introduces a bidirectional cross-attention mechanism based on pixel point cloud spatial pairing index, enabling visual features and geometric features to achieve bidirectional complementary perception under spatial correspondence constraints, outputting a semantic 3D map without dynamic artifacts, and ensuring map quality and path planning reliability.

[0004] In a first aspect, embodiments of this application provide a method for locating a drone, including: Simultaneously trigger the acquisition of laser point cloud sequences, RGB image sequences, dense depth maps, effective projection masks, and IMU pre-integrated measurements by lidar, RGB camera, and IMU, and construct point cloud geometric feature vectors, visual feature vectors, and pixel point cloud spatial pairing indexes; Based on the pixel point cloud spatial pairing index, attention analysis is performed on the point cloud geometric feature vector and the visual feature vector to obtain fused feature descriptors and fused feature matching pairs. The six-DOF pose sequence is solved based on the fused feature matching pairs and the IMU pre-integrated measurements, and a semantic 3D map is generated based on the fused feature descriptors and the six-DOF pose sequence.

[0005] Optionally, in a first implementation of the first aspect of the present invention, the simultaneous triggering of the lidar, RGB camera, and IMU to acquire the lidar point cloud sequence, RGB image sequence, dense depth map, effective projection mask, and IMU pre-integrated measurement values, and constructing point cloud geometric feature vectors, visual feature vectors, and pixel point cloud spatial pairing indexes, includes: Simultaneously trigger the LiDAR, RGB camera, and IMU to acquire laser point cloud sequences, RGB image sequences, and raw IMU measurements; The laser point cloud sequence is projected onto the pixel plane to generate a dense depth map and an effective projection mask, and the original IMU measurement value is pre-integrated to obtain the IMU pre-integrated measurement value. Based on the laser point cloud sequence, the RGB image sequence, the dense depth map, the effective projection mask, and the IMU pre-integration measurement values, point cloud geometric feature vectors, visual feature vectors, and pixel point cloud spatial pairing indexes are constructed.

[0006] Optionally, in a second implementation of the first aspect of the present invention, the step of projecting the laser point cloud sequence onto a pixel plane to generate a dense depth map and an effective projection mask, and pre-integrating the original IMU measurements to obtain pre-integrated IMU measurements, includes: The laser point cloud sequence is transformed into a three-dimensional point set in the camera coordinate system; The three-dimensional point set of the camera coordinate system is projected onto the pixel plane, and the pixel position is filled with the Z-axis component to generate a dense depth map. The position where the projection falls within the image range and the Z-axis is positive is set to 1 to generate an effective projection mask. The raw IMU measurements are pre-integrated to obtain the pre-integrated IMU measurements.

[0007] Optionally, in a third implementation of the first aspect of the present invention, the step of constructing point cloud geometric feature vectors, visual feature vectors, and pixel point cloud spatial pairing indexes based on the laser point cloud sequence, the RGB image sequence, the dense depth map, the effective projection mask, and the IMU pre-integration measurement values ​​includes: Calculate the unit normal vector, edge point descriptor, and planar point descriptor based on the laser point cloud sequence, and construct the point cloud geometric feature vector; Visual feature descriptors are extracted from the RGB image sequence and a visual feature vector is constructed. The target 3D points in the camera coordinate system are obtained by backprojecting from the dense depth map based on the effective projection mask, and a spatial pairing index of the target 3D points in the camera coordinate system and the pixel cloud of the visual feature descriptor is established.

[0008] Optionally, in a fourth implementation of the first aspect of the present invention, the step of back-projecting the target 3D points in the camera coordinate system from the dense depth map based on the effective projection mask, and establishing a pixel point cloud spatial pairing index between the target 3D points in the camera coordinate system and the visual feature descriptor, includes: Traverse the pixel positions with a value of 1 in the effective projection mask, read the depth value from the dense depth map, and back-project the pixel position based on the depth value to obtain the target 3D point in the camera coordinate system; Establish a spatial pairing index between the target 3D points in the camera coordinate system and the pixel cloud of the visual feature descriptor.

[0009] Optionally, in a fifth implementation of the first aspect of the present invention, the step of performing attention analysis on the point cloud geometric feature vector and the visual feature vector based on the pixel point cloud spatial pairing index to obtain a fused feature descriptor and a fused feature matching pair includes: The point cloud geometric feature vector is feature-encoded to obtain a first encoded feature vector, and the visual feature vector is feature-encoded to obtain a second encoded feature vector; Based on the pixel point cloud spatial pairing index, bidirectional cross-attention calculation is performed on the first encoded feature vector and the second encoded feature vector to obtain the first attention output vector for visual guidance geometric direction and the second attention output vector for geometric constraint visual direction. Calculate the fused feature descriptor and the fused feature matching pair based on the first attention output vector and the second attention output vector.

[0010] Optionally, in a sixth implementation of the first aspect of the present invention, the step of solving the six-DOF pose sequence based on the fused feature matching pair and the IMU pre-integration measurement, and generating a semantic 3D map based on the fused feature descriptor and the six-DOF pose sequence, includes: A joint cost function is constructed based on the visual reprojection residual term, the laser point-surface matching residual term, and the inertial weighted residual term in the IMU pre-integration measurement value in the fusion feature matching pair, and the six-degree-of-freedom pose sequence is solved. Based on the fused feature descriptor and the six-degree-of-freedom pose sequence, the laser point cloud sequence is transformed and stitched to the global coordinate system. The semantic segmentation network is applied to the RGB image sequence to infer the semantic category label of each pixel. The point cloud region corresponding to the dynamic target category is removed to obtain a semantic 3D map.

[0011] Optionally, in a seventh implementation of the first aspect of the present invention, the step of constructing a joint cost function and solving for the six-degree-of-freedom pose sequence based on the visual reprojection residual term, the laser point-to-surface matching residual term, and the inertial weighted residual term in the IMU pre-integration measurement value of the fused feature matching pair includes: The visual reprojection residual term and the laser point-to-surface matching residual term are determined based on the fusion feature matching pair, and the inertial weighted residual term is determined based on the IMU pre-integration measurement value. A joint cost function is constructed based on the visual reprojection residual, the laser point-to-surface matching residual, and the inertial weighted residual. The Jacobian matrix is ​​calculated on the sliding window factor graph using the joint cost function, and a six-degree-of-freedom pose sequence is calculated based on the Jacobian matrix.

[0012] Secondly, embodiments of this application provide a drone positioning device, including: The data acquisition module is used to synchronously trigger the LiDAR, RGB camera and IMU to acquire laser point cloud sequences, RGB image sequences, dense depth maps, effective projection masks and IMU pre-integrated measurements, and to construct point cloud geometric feature vectors, visual feature vectors and pixel point cloud spatial pairing indexes; The attention analysis module is used to perform attention analysis on the point cloud geometric feature vector and the visual feature vector according to the pixel point cloud spatial pairing index, so as to obtain the fused feature descriptor and the fused feature matching pair. The map generation module is used to solve for a six-degree-of-freedom pose sequence based on the fused feature matching pair and the IMU pre-integration measurement value, and to generate a semantic three-dimensional map based on the fused feature descriptor and the six-degree-of-freedom pose sequence.

[0013] Optionally, in a first implementation of the second aspect of the present invention, the map generation module is specifically used for: A joint cost function is constructed based on the visual reprojection residual term, the laser point-surface matching residual term, and the inertial weighted residual term in the IMU pre-integration measurement value in the fusion feature matching pair, and the six-degree-of-freedom pose sequence is solved. Based on the fused feature descriptor and the six-degree-of-freedom pose sequence, the laser point cloud sequence is transformed and stitched to the global coordinate system. The semantic segmentation network is applied to the RGB image sequence to infer the semantic category label of each pixel. The point cloud region corresponding to the dynamic target category is removed to obtain a semantic 3D map.

[0014] Thirdly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described UAV positioning method.

[0015] Fourthly, embodiments of this application provide a readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described UAV positioning method.

[0016] Fifthly, embodiments of this application provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the above-described UAV positioning method.

[0017] In one of the solutions provided by the aforementioned UAV positioning methods, devices, equipment, media, and program products, hardware-level time synchronization is achieved using GPIO trigger pulses or PTP timing signals. Combined with IMU motion compensation to eliminate laser point cloud scanning distortion, and with spatial registration of the extrinsic parameter matrix, strict temporal and spatial correspondence between laser and visual data is ensured from the source. A bidirectional cross-attention mechanism based on pixel point cloud spatial pairing index is introduced, enabling bidirectional complementary perception of visual and geometric features under spatial correspondence constraints. The fusion weights of the two modes are adaptively adjusted by dynamically calculating the gating coefficient based on the point cloud density dispersion and image gradient density. In weak texture scenes, the contribution of the laser mode is automatically increased, and in open and degraded scenes, the contribution of the visual mode is automatically increased. The visual reprojection residual, laser point-surface matching residual, and IMU inertial residual are uniformly incorporated into the same cost function for joint optimization, realizing mutual constraint and compensation of multi-source errors, eliminating the problem of pre-accumulation of independent estimation errors of each sensor, and further eliminating long-term drift by combining bag-of-words loop closure detection and global BA optimization. In terms of semantic mapping, the semantic segmentation network identifies and removes the point cloud regions corresponding to dynamic targets in real time, outputting a semantic 3D map without dynamic artifacts, ensuring map quality and the reliability of path planning. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the structure of an unmanned aerial vehicle (UAV) positioning system according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating a drone positioning method according to an embodiment of the present invention; Figure 3 yes Figure 2 A schematic diagram of the implementation process of step S10; Figure 4 yes Figure 2 A schematic diagram of the implementation process of step S20; Figure 5 yes Figure 2 A schematic diagram of the implementation process of step S30; Figure 6 yes Figure 5 A schematic diagram of the implementation process of step S31; Figure 7 This is a schematic diagram of a UAV positioning device in one embodiment of the present invention; Figure 8This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof. It should also be understood that, as used in this specification and the appended claims, the term "and / or" refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0022] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0023] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0024] It should be understood that the sequence number of each step in the following embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0025] To illustrate the technical solution of the present invention, specific embodiments are described below.

[0026] To address the problems mentioned above in the background art, this application provides a method, apparatus, device, medium, and program product for locating unmanned aerial vehicles (UAVs). The UAV positioning method provided by this invention can be applied to applications such as... Figure 1 The drone positioning system shown includes a client and a server.

[0027] In one embodiment, such as Figure 2 As shown, a method for locating a drone is provided, which can be applied to... Figure 1 Taking the drone positioning system as an example, the following steps are included: S10: Simultaneously triggers the acquisition of laser point cloud sequences, RGB image sequences, dense depth maps, effective projection masks, and IMU pre-integrated measurements by the LiDAR, RGB camera, and IMU, and constructs point cloud geometric feature vectors, visual feature vectors, and pixel point cloud spatial pairing indexes. S20: Based on the pixel point cloud spatial pairing index, perform attention analysis on the point cloud geometric feature vector and visual feature vector to obtain the fused feature descriptor and fused feature matching pair; S30: Solve the six-DOF pose sequence based on the fused feature matching pairs and IMU pre-integrated measurements, and generate a semantic 3D map based on the fused feature descriptor and the six-DOF pose sequence.

[0028] In this embodiment, hardware-level time synchronization is achieved using GPIO trigger pulses or PTP timing signals. Combined with IMU motion compensation to eliminate laser point cloud scanning distortion, and with spatial registration of the extrinsic parameter matrix, strict temporal and spatial correspondence between laser and visual data is ensured from the source. A bidirectional cross-attention mechanism based on pixel point cloud spatial pairing index is introduced, enabling bidirectional complementary perception of visual and geometric features under spatial correspondence constraints. The fusion weights of the two modes are adaptively adjusted by dynamically calculating the gating coefficients based on the point cloud density dispersion and image gradient density. The laser mode contribution is automatically increased in weak texture scenes, and the visual mode contribution is automatically increased in open and degraded scenes. The visual reprojection residual, laser point-surface matching residual, and IMU inertial residual are uniformly incorporated into the same cost function for joint optimization, realizing mutual constraint and compensation of multi-source errors and eliminating the problem of pre-accumulation of independent estimation errors of each sensor. Combined with bag-of-words loop closure detection and global BA optimization, long-term drift is further eliminated. In terms of semantic mapping, the semantic segmentation network identifies and removes the point cloud regions corresponding to dynamic targets in real time, outputting a semantic 3D map without dynamic artifacts, ensuring map quality and the reliability of path planning.

[0029] In one embodiment, such as Figure 3 As shown, step S10 specifically includes the following steps: S11: Synchronously triggers the acquisition of laser point cloud sequences, RGB image sequences, and raw IMU measurements by lidar, RGB camera, and IMU; S12: Project the laser point cloud sequence onto the pixel plane to generate a dense depth map and an effective projection mask, and pre-integrate the raw IMU measurements to obtain the pre-integrated IMU measurements; S13: Construct point cloud geometric feature vectors, visual feature vectors, and pixel point cloud spatial pairing indexes based on laser point cloud sequences, RGB image sequences, dense depth maps, effective projection masks, and IMU pre-integration measurements.

[0030] In this embodiment, a unified clock source or synchronization controller synchronously triggers the lidar, RGB camera, and IMU on the UAV's onboard sensor platform, enabling the acquisition of the laser point cloud sequence, RGB image sequence, and raw IMU measurements under the same time reference. Synchronization can be achieved using GPIO trigger pulses or PTP timing signals, with the timestamp alignment error preset to less than 1ms to reduce spatial correspondence deviations caused by asynchronous sampling from multiple sensors during UAV flight. The synchronously acquired laser point cloud sequence is transformed to the camera coordinate system according to the pre-calibrated extrinsic parameter relationship between the camera and lidar, and projected onto the pixel plane corresponding to the RGB image. A dense depth map is generated based on the depth information of the projected points. Simultaneously, a valid projection mask is generated based on whether the projected points fall within the effective image range and whether the depth direction is valid, thus establishing a correspondence between the three-dimensional spatial points in the point cloud and the pixel positions in the image. Simultaneously, the raw IMU measurements are pre-integrated according to the time interval between adjacent keyframes or adjacent point cloud frames, converting high-frequency angular velocity and linear acceleration data into attitude, position, and velocity increment constraints between two frames, resulting in pre-integrated IMU measurements. Based on the laser point cloud sequence, unit normal vectors, edge point descriptors, and planar point descriptors are extracted to construct point cloud geometric feature vectors. Based on the RGB image sequence, texture, edge, and semantically related visual descriptors are extracted to construct visual feature vectors. Combined with dense depth maps and effective projection masks, effective pixel positions, corresponding depth values, 3D points in camera coordinate system, and visual feature descriptors are associated to form a pixel point cloud spatial pairing index.

[0031] In one embodiment, step S12 specifically includes the following steps: Transform the laser point cloud sequence into a 3D point set in the camera coordinate system; Project the 3D point set of the camera coordinate system onto the pixel plane, fill the pixel position with the Z-axis component to generate a dense depth map, and set the position where the projection falls within the image range and the Z-axis is positive to 1 to generate an effective projection mask. Pre-integrate the raw IMU measurements to obtain the pre-integrated IMU measurements.

[0032] In this embodiment, the time-synchronized laser point cloud sequence is used as input. Based on the pre-calibrated external parameter rigid body transformation relationship between the camera and the lidar, each three-dimensional point in the lidar coordinate system is transformed to the camera coordinate system, forming a three-dimensional point set in the camera coordinate system. The coordinate transformation can be expressed as follows: ,in Represents a three-dimensional point in the lidar coordinate system. This represents a 3D point in the transformed camera coordinate system. This represents the extrinsic rigid body transformation matrix between the camera and the lidar.

[0033] In this embodiment, after obtaining the 3D point set in the camera coordinate system, the 3D points are projected onto the pixel plane of the RGB image according to the pinhole camera projection model. The projection relationship is as follows: , ,in Represents pixel coordinates, This represents the coordinate components of a 3D point in the camera coordinate system. This refers to the focal length parameter in the camera's intrinsic parameters. The principal point coordinates are represented; then, it is determined whether the pixel position obtained by projection is within the width and height range of the image, and the corresponding 3D point is determined. Whether it is positive or negative depends on whether the pixel position is valid and... Only the points that are specified are written into the depth map, and the points are then... The component serves as the depth value for the corresponding pixel location. If multiple 3D points are projected onto the same pixel location, the depth value closer to the camera can be retained to reduce depth conflicts caused by occlusion, thereby generating a dense depth map with the same size as the RGB image. Simultaneously, pixel locations satisfying the above effective conditions are set to 1 in the effective projection mask, while locations not covered by effective point projections, out-of-bounds locations, and... The position is kept at 0, thus obtaining an effective projection mask.

[0034] Pre-integrate the raw IMU measurements between adjacent keyframes or adjacent point cloud frames, continuously accumulating the angular velocity and linear acceleration within this time interval into attitude increment, position increment, and velocity increment, to obtain the IMU pre-integrated measurements. ,in This represents the attitude increment rotation matrix. Represents the position increment vector. This represents the velocity increment vector, which transforms the IMU data from discrete high-frequency sampled values ​​into an overall inertial motion constraint between two frames.

[0035] In one embodiment, step S13 specifically includes the following steps: Calculate the unit normal vector, edge point descriptor, and planar point descriptor based on the laser point cloud sequence, and construct the point cloud geometric feature vector; Visual feature descriptors are extracted from RGB image sequences and visual feature vectors are constructed. Based on the effective projection mask, the target 3D points in the camera coordinate system are obtained by backprojection from the dense depth map, and a spatial pairing index of the target 3D points in the camera coordinate system and the pixel cloud of the visual feature descriptor is established.

[0036] In this embodiment, using the laser point cloud sequence as input, the local neighborhood of each spatial point in the current frame point cloud is searched, and the unit normal vector of the point is calculated based on the coordinate distribution of the neighborhood points, so that each point not only has three-dimensional coordinate information, but also local surface orientation information; according to the local comprehensive curvature calculation method ,in As the current center point, For its neighborhood set, Let be the unit normal vector at the center point. The number of neighboring points. This is used to characterize the degree of geometric change near the point; the point cloud is distinguished according to the preset curvature classification threshold. Points with larger curvature are identified as edge points, and information such as local direction vectors and neighborhood curvature changes are extracted to form edge point descriptors. Points with smaller curvature and gentler local normal vector changes are identified as planar points, and information such as normal vectors, local center point coordinates, and neighborhood plane stability are extracted to form planar point descriptors. Then, the unit normal vector, edge point descriptors, and planar point descriptors are concatenated, normalized, and dimension-mapped to construct the point cloud geometric feature vector, which can reflect structural features such as edges, corners, walls, and ground.

[0037] In this embodiment, the RGB image sequence is subjected to distortion correction and illumination normalization, and then input into a lightweight CNN or Transformer backbone network to extract multi-scale visual features. Visual feature extraction can be represented as follows: ,in The current frame's RGB image. This is the output visual feature vector. This backbone network is used to extract texture, edge, and semantic information. The process compresses texture gradients, object boundaries, and semantic category information from the image into visual feature descriptors that match the geometric dimensions of the point cloud. Pixel locations with reliable depth values ​​in the dense depth map are selected based on an effective projection mask. Depth information is then read from these locations to reconstruct the target 3D point in the camera coordinate system. The target 3D point is then associated with visual feature descriptors at the same or neighboring pixel locations, forming a pixel-point cloud spatial pairing index. This ensures that the point cloud geometric features and image visual features form a callable, matchable, and input-compatible cross-attention fusion module correspondence in the same spatial location.

[0038] In one embodiment, the steps of backprojecting the target 3D points in the camera coordinate system from the dense depth map based on the effective projection mask, and establishing a pixel cloud spatial pairing index between the target 3D points in the camera coordinate system and the visual feature descriptors specifically include the following steps: Traverse the pixel positions with a value of 1 in the effective projection mask, read the depth value from the dense depth map, and backproject the pixel position based on the depth value to obtain the target 3D point in the camera coordinate system. Establish a spatial pairing index between the target 3D points in the camera coordinate system and the pixel cloud of the visual feature descriptor.

[0039] In this embodiment, the effective projection mask is used as the pixel selection condition. It is traversed row by row or block by block. When the mask value of a certain pixel position is 1, it means that the position has been verified as a valid area by the previous point cloud projection process. That is, the pixel falls within the image range and the corresponding depth direction is valid. Therefore, the depth value can be read from the same pixel position in the dense depth map, and the two-dimensional pixel position can be restored to the target three-dimensional point in the camera coordinate system by combining the camera intrinsic parameters. During the restoration process, positions with empty depth, abnormal depth, or excessive depth jump with adjacent pixels are excluded to avoid introducing erroneous three-dimensional points by occlusion edges, sparse point cloud holes, or reflection noise. Then, a set of target three-dimensional points is generated according to the correspondence between the effective pixel position and the depth value, so that each target three-dimensional point retains its source pixel position, depth value, and the image frame identifier to which it belongs. The feature vector corresponding to the same pixel position is read from the visual feature map or visual feature descriptor set. If the visual feature map is scaled relative to the original RGB image, the corresponding feature position is determined according to the mapping relationship between the feature map scale and the image scale. Then, the visual feature descriptor at that position is obtained through neighbor sampling or interpolation. The target 3D point in the camera coordinate system, the source pixel position, the visual feature descriptor, and the corresponding point cloud geometric feature identifier are written into the same index record to form a pixel point cloud spatial pairing index, so that the index entries can simultaneously point to the visual features on the image side and the spatial point information on the point cloud side. 3D points with the same source or too close distance in adjacent pixels are merged. For the position of the same 3D point corresponding to multiple visual features, the descriptor with high response intensity or high semantic confidence is retained, and a low confidence mark is set for effective depth points without reliable visual features.

[0040] In one embodiment, such as Figure 4 As shown, step S20 specifically includes the following steps: S21: Perform feature encoding on the geometric feature vector of the point cloud to obtain the first encoded feature vector, and perform feature encoding on the visual feature vector to obtain the second encoded feature vector; S22: Based on the pixel point cloud spatial pairing index, perform bidirectional cross-attention calculation on the first encoded feature vector and the second encoded feature vector to obtain the first attention output vector for visually guided geometric direction and the second attention output vector for geometrically constrained visual direction. S23: Calculate the fused feature descriptor and the fused feature matching pair based on the first attention output vector and the second attention output vector.

[0041] In this embodiment, the point cloud geometric feature vectors and visual feature vectors, which have already established spatial correspondence, are used as dual-modal fusion inputs. Modal-independent encoding is performed on the two types of features. Since point cloud geometric features mainly reflect local normal vectors, edge points, planar points, and spatial structure changes, while visual features mainly reflect texture, edges, color, and semantic information, their numerical scales, distribution ranges, and feature meanings are not consistent. Therefore, independent linear mapping, layer normalization, and dimension alignment are used to encode the point cloud geometric feature vectors into a first encoded feature vector and the visual feature vectors into a second encoded feature vector. The encoding form is as follows: , ,in Represents the geometric eigenvectors of a point cloud. Represents visual feature vectors. This represents the first encoded feature vector. This represents the second encoded feature vector. This represents the learnable weight matrix for the two features. This represents the bias vector. The representation layer undergoes normalization; after encoding, geometric and visual features are mapped to the same dimensional space. The interaction range of the two types of encoded features is limited by the pixel point cloud spatial pairing index, enabling attention calculation to be performed between locations with spatial correspondence confirmed through projection, backprojection, and masking. In the visually guided geometric direction, using the second encoded feature vector as the query term and the first encoded feature vector as the key and value, the attention weight of visual semantic information on geometric features is calculated. This attention calculation can be written as... ,in This represents the first attention output vector that guides the geometric direction of vision. This represents the query item obtained by mapping visual encoded features. This represents the key and value obtained from the geometrically encoded feature mapping. Representing feature dimension, This is used to scale the dot product result, avoiding excessive concentration of attention weights when the dimension is too large; simultaneously, in the geometrically constrained visual direction, the second attention output vector in the geometrically constrained visual direction is obtained by using the first encoded feature vector as the query term and the second encoded feature vector as the key and value, following the same attention calculation structure. This allows the geometric structure of point clouds to inversely constrain visual scale and spatial location judgment.

[0042] In this embodiment, the first attention output vector and the second attention output vector are adaptively fused. To adapt to different scenarios such as weak texture, low light, dust, empty space, or repetitive structures, a gating coefficient can be calculated based on the current frame's visual quality score, laser quality score, and environmental context encoding. The gating calculation is as follows: ,in Indicates the gating coefficient for the current frame. Indicates visual quality score. Indicates the laser quality score. Represents the environment context encoding vector. Represents the gate weight matrix. Indicates gating bias. Represent the Sigmoid function with the gating coefficient between 0 and 1; according to The fusion intermediate features are obtained, where Indicates the fusion of intermediate features, This indicates that element-weighted methods can increase the contribution of geometric information in weak textures or low light conditions, and increase the contribution of visual semantic information in open or repetitive geometric structures; through... Output fused feature descriptors, where Indicates the fusion feature descriptor, This represents a one-to-one convolution mapping. Residual connection terms are represented, and similarity matching, spatial consistency verification, and index backfilling are performed between fused feature descriptors. Pixels that meet the requirements of descriptor similarity, spatial proximity, and geometric constraint consistency are combined with point cloud points to form fused feature matching pairs.

[0043] In one embodiment, such as Figure 5 As shown, step S30 specifically includes the following steps: S31: Construct a joint cost function based on the visual reprojection residual term, laser point-surface matching residual term, and inertial weighted residual term in the IMU pre-integration measurement value from the fused feature matching pair, and solve the six-degree-of-freedom pose sequence; S32: Based on the fused feature descriptor and the six-degree-of-freedom pose sequence, the laser point cloud sequence is transformed and stitched to the global coordinate system. The semantic segmentation network is applied to the RGB image sequence to infer the semantic category label of each pixel. The point cloud region corresponding to the dynamic target category is removed to obtain a semantic 3D map.

[0044] In this embodiment, the fused feature descriptor and fused feature matching pair are used as the front-end observation results, and the IMU pre-integrated measurement value is used as the inertial constraint between adjacent keyframes and input to the back-end pose calculation module. A unified optimization objective is established around the six-degree-of-freedom pose sequence of the UAV to be estimated. The visual side uses the correspondence between image features and spatial points in the fused feature matching pair to form the visual reprojection residual. The laser side uses the correspondence between feature points in the point cloud and local planes to form the laser point-plane matching residual. The inertial side uses the IMU pre-integrated measurement value to constrain the attitude, velocity and position changes between adjacent keyframes. Since the three types of residuals have different sources and different noise characteristics, weights or covariances need to be configured separately when constructing the joint cost function so that the visual constraint, laser constraint and inertial constraint participate in the optimization together in the same sliding window factor graph. The pose, velocity and related state variables of each keyframe in the window are iteratively updated by nonlinear least squares solution until the residual decreases to the preset convergence condition or reaches the preset number of iterations, thereby outputting a continuous six-degree-of-freedom pose sequence.

[0045] The obtained six-DOF pose sequence is used as the basis for spatial transformation of point cloud stitching. Each frame of laser point cloud is transformed from the coordinate system of the machine or camera to the global coordinate system according to its pose at the corresponding time. Multiple frames of point cloud are accumulated and stitched in chronological order. During the stitching process, the consistency of the same structure in adjacent frames or loop frames can be checked by combining the fusion feature descriptor to reduce ghosting and drift caused by single frame pose error. At the same time, the semantic segmentation network is applied to the synchronously acquired RGB image sequence to infer the semantic category label of each pixel, such as road, wall, ground, building, tree, pedestrian, vehicle or other dynamic obstacle categories. The semantic category of the image pixel is backfilled into the corresponding three-dimensional point cloud region by using dense depth map, effective projection mask and pixel point cloud spatial pairing index. For pixel regions identified as dynamic targets, relevant point sets in the global point cloud can be located based on their corresponding depth values ​​and spatial indices. These point sets are then removed before or after point cloud fusion to prevent moving objects such as pedestrians and vehicles from creating ghosting, double images, or false obstacles during multi-frame stitching. For point cloud regions corresponding to static categories, they are retained and written into the global semantic map. Simultaneously, repeatedly observed static points undergo voxel merging, semantic voting, and confidence updates to make the geometric structure of the same spatial region more stable and the semantic labels more consistent. Global point cloud stitching is completed using spatial pose constraints provided by a six-degree-of-freedom pose sequence. Cross-frame matching and structural consistency are enhanced by fusing feature descriptors. Point cloud regions corresponding to dynamic targets are removed using semantic segmentation results, thus obtaining a semantic 3D map.

[0046] In one embodiment, such as Figure 6 As shown, step S31 specifically includes the following steps: S311: Determine the visual reprojection residual term and the laser point-to-surface matching residual term based on the fusion feature matching pair, and determine the inertial weighted residual term based on the IMU pre-integration measurement value; S312: Construct a joint cost function based on the visual reprojection residual, the laser point-to-surface matching residual, and the inertial weighted residual; S313: Calculate the Jacobian matrix on the sliding window factor graph using the joint cost function, and calculate the six-DOF pose sequence based on the Jacobian matrix.

[0047] In this embodiment, the fusion feature matching pair is split into image-side observation, point cloud-side observation, and the corresponding 3D landmark point relationship. Three types of residuals are established respectively. The visual reprojection residual term is obtained by subtracting the pixel observation position in the fusion matching pair from the predicted pixel position after the 3D landmark point is reprojected onto the image plane in the current pose. The laser point-plane matching residual term is obtained by subtracting the laser 3D point in the fusion matching pair from the predicted projection position on the nearest plane or edge feature in the current local map. The inertial weighted residual term is obtained by subtracting the IMU pre-integrated measurement value from the pre-integrated value predicted based on the pose of the current adjacent keyframe, gyroscope zero bias, and accelerometer zero bias.

[0048] In this embodiment, the three types of residuals are uniformly incorporated into the same nonlinear least squares optimization objective, and the covariance matrix is ​​used to weight the residuals of different sensors. This ensures that observations with higher reliability contribute more to pose updates, while observations with lower reliability or higher noise are appropriately suppressed. The joint cost function can be written as follows: ,in Let be the set of states to be optimized. These are visual pixel observations. For the pose of 3D landmark points The reprojection prediction value under the current conditions, These are the observation values ​​of the laser point. For the laser point on the local map Geometric prediction values ​​in For IMU pre-integrated measurements, The inertial quantity is predicted from the current pose and zero bias. , , These are the covariance matrices of the three types of residuals.

[0049] In this embodiment, the poses, 3D landmarks, and IMU factors of the most recent keyframes are maintained as a sliding window factor map. The derivative of the joint cost function is calculated at the current linearization point to obtain the Jacobian matrix of each residual term relative to the state variable. Then, the state increment is calculated using Levenberg-Marquardt nonlinear least squares iteration. The state increment formula can be written as follows: ,in For Jacobian matrices, The residual vector is the result of stacking the visual, laser, and inertial residuals. The residual covariance matrix is... The damping coefficient is... For the identity matrix; we obtain Then, it is added back to the pose, velocity, and zero bias states within the sliding window, and the residuals and Jacobian matrices are recalculated and iterated until the decrease in the joint cost function meets the preset convergence condition or reaches the preset number of iterations. Finally, the six-degree-of-freedom pose sequence of the UAV is output in the key frame time sequence, so that vision, laser and IMU observations are mutually constrained within the same optimization framework, reducing the drift caused by the degradation of a single sensor.

[0050] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0051] In one embodiment, a drone positioning device is provided, which corresponds one-to-one with the drone positioning method described in the above embodiments. For example... Figure 7 As shown, the drone positioning device includes: The data acquisition module 701 is used to synchronously trigger the lidar, RGB camera and IMU to acquire the lidar point cloud sequence, RGB image sequence, dense depth map, effective projection mask and IMU pre-integrated measurement value, and to construct the point cloud geometric feature vector, visual feature vector and pixel point cloud spatial pairing index; The attention analysis module 702 is used to perform attention analysis on the geometric feature vector and visual feature vector of the point cloud based on the pixel point cloud spatial pairing index, so as to obtain the fused feature descriptor and the fused feature matching pair. The map generation module 703 is used to solve the six-degree-of-freedom pose sequence based on the fusion feature matching pairs and IMU pre-integrated measurements, and to generate a semantic 3D map based on the fusion feature descriptor and the six-degree-of-freedom pose sequence.

[0052] Map generation module 703 is specifically used for: A joint cost function is constructed based on the visual reprojection residual term, the laser point-surface matching residual term, and the inertial weighted residual term in the IMU pre-integration measurement value from the fusion feature matching pair, and the six-degree-of-freedom pose sequence is solved. Based on the fused feature descriptor and the six-degree-of-freedom pose sequence, the laser point cloud sequence is transformed and stitched to the global coordinate system. The semantic segmentation network is applied to the RGB image sequence to infer the semantic category label of each pixel. The point cloud region corresponding to the dynamic target category is removed to obtain a semantic 3D map.

[0053] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0054] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0055] This application also provides a computer device, such as... Figure 8 As shown, the computer device includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor. When the processor executes the computer program, it implements the steps in any of the above method embodiments, or when the processor executes the computer program, it implements the functions of each module / unit in the above device embodiments.

[0056] For example, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the computer device.

[0057] Those skilled in the art will understand that Figure 8 The computer device described is merely an example and does not constitute a limitation on the computer device. It may include more or fewer components than shown, or combine certain components, or different components. For example, the computer device may also include input / output devices, network access devices, buses, etc.

[0058] The aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs). Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0059] The memory can be an internal storage unit of the computer device, such as a hard drive or RAM. The memory can also be an external storage device of the computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory can include both internal and external storage units of the computer device.

[0060] This application also provides a readable storage medium storing a computer program, which, when executed by a processor, can implement the steps in the above-described method embodiments.

[0061] This application provides a computer program product that, when run on an electronic device, enables the electronic device to implement the steps described in the various method embodiments above.

[0062] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0063] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0064] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0065] In the embodiments provided in this application, it should be understood that the disclosed apparatus / devices and methods can be implemented in other ways. For example, the apparatus / device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0066] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0067] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for locating unmanned aerial vehicles (UAVs), characterized in that, include: Simultaneously trigger the acquisition of laser point cloud sequences, RGB image sequences, dense depth maps, effective projection masks, and IMU pre-integrated measurements by lidar, RGB camera, and IMU, and construct point cloud geometric feature vectors, visual feature vectors, and pixel point cloud spatial pairing indexes; Based on the pixel point cloud spatial pairing index, attention analysis is performed on the point cloud geometric feature vector and the visual feature vector to obtain fused feature descriptors and fused feature matching pairs. The six-DOF pose sequence is solved based on the fused feature matching pairs and the IMU pre-integrated measurements, and a semantic 3D map is generated based on the fused feature descriptors and the six-DOF pose sequence.

2. The UAV positioning method as described in claim 1, characterized in that, The synchronously triggered lidar, RGB camera, and IMU acquire lidar point cloud sequences, RGB image sequences, dense depth maps, effective projection masks, and IMU pre-integrated measurements, and construct point cloud geometric feature vectors, visual feature vectors, and pixel point cloud spatial pairing indices, including: Simultaneously trigger the LiDAR, RGB camera, and IMU to acquire laser point cloud sequences, RGB image sequences, and raw IMU measurements; The laser point cloud sequence is projected onto the pixel plane to generate a dense depth map and an effective projection mask, and the original IMU measurement value is pre-integrated to obtain the IMU pre-integrated measurement value. Based on the laser point cloud sequence, the RGB image sequence, the dense depth map, the effective projection mask, and the IMU pre-integration measurement values, point cloud geometric feature vectors, visual feature vectors, and pixel point cloud spatial pairing indexes are constructed.

3. The UAV positioning method as described in claim 2, characterized in that, The process of projecting the laser point cloud sequence onto a pixel plane to generate a dense depth map and an effective projection mask, and pre-integrating the raw IMU measurements to obtain pre-integrated IMU measurements, includes: The laser point cloud sequence is transformed into a three-dimensional point set in the camera coordinate system; The three-dimensional point set of the camera coordinate system is projected onto the pixel plane, and the pixel position is filled with the Z-axis component to generate a dense depth map. The position where the projection falls within the image range and the Z-axis is positive is set to 1 to generate an effective projection mask. The raw IMU measurements are pre-integrated to obtain the pre-integrated IMU measurements.

4. The UAV positioning method as described in claim 2, characterized in that, The construction of point cloud geometric feature vectors, visual feature vectors, and pixel point cloud spatial pairing indexes based on the laser point cloud sequence, the RGB image sequence, the dense depth map, the effective projection mask, and the IMU pre-integration measurements includes: Calculate the unit normal vector, edge point descriptor, and planar point descriptor based on the laser point cloud sequence, and construct the point cloud geometric feature vector; Visual feature descriptors are extracted from the RGB image sequence and a visual feature vector is constructed. The target 3D points in the camera coordinate system are obtained by backprojecting from the dense depth map based on the effective projection mask, and a spatial pairing index of the target 3D points in the camera coordinate system and the pixel cloud of the visual feature descriptor is established.

5. The UAV positioning method as described in claim 1, characterized in that, The step of performing attention analysis on the point cloud geometric feature vector and the visual feature vector based on the pixel point cloud spatial pairing index to obtain fused feature descriptors and fused feature matching pairs includes: The point cloud geometric feature vector is feature-encoded to obtain a first encoded feature vector, and the visual feature vector is feature-encoded to obtain a second encoded feature vector; Based on the pixel point cloud spatial pairing index, bidirectional cross-attention calculation is performed on the first encoded feature vector and the second encoded feature vector to obtain the first attention output vector for visual guidance geometric direction and the second attention output vector for geometric constraint visual direction. Calculate the fused feature descriptor and the fused feature matching pair based on the first attention output vector and the second attention output vector.

6. The UAV positioning method as described in claim 1, characterized in that, The step of solving the six-DOF pose sequence based on the fused feature matching pairs and the IMU pre-integrated measurements, and generating a semantic 3D map based on the fused feature descriptors and the six-DOF pose sequence, includes: A joint cost function is constructed based on the visual reprojection residual term, the laser point-surface matching residual term, and the inertial weighted residual term in the IMU pre-integration measurement value in the fusion feature matching pair, and the six-degree-of-freedom pose sequence is solved. Based on the fused feature descriptor and the six-degree-of-freedom pose sequence, the laser point cloud sequence is transformed and stitched to the global coordinate system. The semantic segmentation network is applied to the RGB image sequence to infer the semantic category label of each pixel. The point cloud region corresponding to the dynamic target category is removed to obtain a semantic 3D map.

7. A drone positioning device, characterized in that, The steps for implementing the UAV positioning method as described in any one of claims 1 to 6 include: The data acquisition module is used to synchronously trigger the LiDAR, RGB camera and IMU to acquire laser point cloud sequences, RGB image sequences, dense depth maps, effective projection masks and IMU pre-integrated measurements, and to construct point cloud geometric feature vectors, visual feature vectors and pixel point cloud spatial pairing indexes; The attention analysis module is used to perform attention analysis on the point cloud geometric feature vector and the visual feature vector according to the pixel point cloud spatial pairing index, so as to obtain the fused feature descriptor and the fused feature matching pair. The map generation module is used to solve for a six-degree-of-freedom pose sequence based on the fused feature matching pair and the IMU pre-integration measurement value, and to generate a semantic three-dimensional map based on the fused feature descriptor and the six-degree-of-freedom pose sequence.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the UAV positioning method as described in any one of claims 1 to 6.

9. A readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the UAV positioning method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the UAV positioning method as described in any one of claims 1 to 6.

Images