A method for dynamic target three-dimensional positioning and tracking of ground fitting and dynamic separation applied to a UAV

Abstract

The embodiment of the application provides a kind of dynamic target three-dimensional positioning and tracking of ground surface fitting and dynamic separation applied to unmanned aerial vehicle, the method comprises: collecting the image data of target scene, and based on the dynamic-static object detection and classification of semantics;Then the ground features in static features are reconstructed and fitted by SLAM three-dimensionally;Subsequently, using ground height constraint, dynamic target is calculated by single-frame three-dimensional positioning;Finally, fusion multi-frame observation and motion model are optimized jointly.The core problems of the prior art, such as difficult depth information acquisition, poor environmental adaptability and high cost, are solved.

Description

Technical Field

[0001] This invention relates to the field of drone tracking technology, and in particular to a method for dynamic target three-dimensional positioning and tracking applied to drones by ground surface fitting and dynamic separation. Background Technology

[0002] As drones become increasingly popular, they are being used in a growing number of applications, such as inspection and security monitoring, smart transportation and traffic flow analysis, moving target interaction, geographic information acquisition, and other fields, to monitor the movement of targets and locate them in real time.

[0003] However, the following problems exist when using monocular vision from UAVs for 3D localization of moving targets:

[0004] If we adopt the purely dynamic SLAM and target tracking schemes commonly used in existing technologies, such schemes (e.g., derivatives based on ORB-SLAM3) first run visual SLAM to estimate the UAV's own motion and reconstruct the static environment, while simultaneously detecting and tracking dynamic targets. Their core drawback is that the depth information of dynamic targets is highly dependent on multi-frame triangulation. When there is relative motion between the target and the UAV, triangulation requires precise cross-frame matching of target feature points with the UAV's pose. Any matching error or pose drift will lead to highly unstable depth calculations, especially when the target moves along the camera's optical axis, resulting in noisy and unreliable calculations.

[0005] If we assume a flat surface: Some simplified schemes assume the target is located on a level surface at a known height, and calculate the target's position using image coordinates. The drawback of this scheme is that it cannot handle non-flat, undulating real-world surfaces (such as slopes, grass, or roads with curbs), leading to incorrect height estimation and consequently, significant positional errors.

[0006] There are also some solutions that rely on high-cost sensors: using LiDAR or binocular / multi-view cameras can directly acquire depth information, but this significantly increases the hardware cost, weight and power consumption of the system, making it unsuitable for small commercial drone platforms that are sensitive to cost and payload. Summary of the Invention

[0007] To address the problems existing in the prior art, embodiments of the present invention provide a method for dynamic target 3D localization and tracking applied to UAVs for surface fitting and dynamic separation.

[0008] In a first aspect, embodiments of this specification provide a method for dynamic target 3D localization and tracking applied to ground surface fitting and dynamic separation of unmanned aerial vehicles (UAVs), the method comprising:

[0009] Continuously acquire image data of the target scene, extract image features from the image data and perform feature type segmentation, output the segmented feature types and image mask, the feature types include static regions and dynamic targets, the static regions include planar static regions and vertical static regions;

[0010] Based on the mask of the static region, static feature points are extracted. Combined with the position of the dynamic target on the image, the ROI of the dynamic target is determined. Based on the ROI, the feature point set of the static region is traversed. After matching with the feature points of the same semantic region in the historical frame, a 3D point cloud corresponding to the region where the dynamic target is located is generated.

[0011] Based on the 3D point cloud, a corresponding 3D map is established;

[0012] Using the contact point of the dynamic target in the image as the observation point, the target is projected onto the three-dimensional map to solve for the three-dimensional coordinates of the dynamic target. Combined with the historical trajectory observations of the dynamic target, the motion trajectory prediction result of the dynamic target is generated.

[0013] Secondly, embodiments of this specification provide a system for dynamic target 3D localization and tracking applied to ground surface fitting and dynamic separation of unmanned aerial vehicles (UAVs), the system comprising:

[0014] The acquisition module is used to continuously acquire image data of the target scene, extract image features from the image data and perform feature type segmentation, and output the segmented feature types and image mask. The feature types include static regions and dynamic targets. The static regions include planar static regions and vertical static regions.

[0015] The feature extraction module is used to extract static feature points based on the mask of the static region, determine the ROI of the dynamic target by combining the position of the dynamic target on the image, and traverse the feature point set of the static region based on the ROI. After matching with the feature points of the same semantic region in the historical frame, a 3D point cloud corresponding to the region where the dynamic target is located is generated.

[0016] A 3D map module is used to create a corresponding 3D map based on the 3D point cloud;

[0017] The trajectory module is used to calculate the three-dimensional coordinates of the dynamic target by taking the contact point of the dynamic target in the image as the observation point and combining it with the three-dimensional map, and to generate the motion trajectory prediction result of the dynamic target by combining it with the historical trajectory observation value of the dynamic target.

[0018] Thirdly, embodiments of this specification provide an electronic device, including a processor and a memory;

[0019] The processor is connected to the memory;

[0020] The memory is used to store executable program code;

[0021] The processor runs a program corresponding to the executable program code stored in the memory to perform the methods described in one or more embodiments.

[0022] Fourthly, embodiments of this specification provide a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described method for dynamic target three-dimensional positioning and tracking applied to UAVs for surface fitting and dynamic separation.

[0023] In view of the above, the beneficial effects of the technical solutions provided by some embodiments of this specification include at least the following:

[0024] In one or more embodiments of this specification, the monocular vision dynamic target 3D localization process is explicitly decomposed into four interdependent, data-closed-loop steps: "semantic-based dynamic-static object detection and classification," "SLAM reconstruction and surface modeling based on purely static features," "single-frame 3D solution of dynamic targets based on surface height constraints," and "joint optimization by fusing multi-frame observations and motion models." This addresses the core problems of existing technologies, such as difficulty in acquiring depth information, poor environmental adaptability, and high cost. It provides a method that does not rely on expensive sensors, can adapt to non-flat surfaces, and stably estimates the 3D coordinates of moving targets. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart illustrating a method for dynamic target 3D localization and tracking applied to UAVs for ground surface fitting and dynamic separation, provided by one embodiment of this specification.

[0027] Figure 2 This is a flowchart of another embodiment of the present specification, which provides a method for dynamic target three-dimensional localization and tracking applied to UAVs for surface fitting and dynamic separation.

[0028] Figure 3 This is a schematic diagram of the structure of a system for dynamic target 3D localization and tracking applied to UAVs for surface fitting and dynamic separation, provided by one embodiment of this specification.

[0029] Figure 4 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this specification. Detailed Implementation

[0030] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed merely to enable those skilled in the art to better understand and implement the subject matter described herein, and are not intended to limit the scope, applicability, or examples set forth in the claims. The function and arrangement of the elements discussed may be changed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the various examples. For example, the described methods may be performed in a different order than described, and steps may be added, omitted, or combined. Furthermore, features described in some examples may be combined in other examples.

[0031] As used herein, the term "comprising" and its variations are open terms meaning "including but not limited to". The term "based on" means "at least partially based on". The terms "one embodiment" and "an embodiment" mean "at least one embodiment". The term "another embodiment" means "at least one other embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other definitions, whether explicit or implicit, may be included below. Unless explicitly indicated by the context, the definition of a term shall remain consistent throughout the specification.

[0032] Please see Figure 1 , Figure 1 This document illustrates an overall flowchart of a method for dynamic target 3D localization and tracking applied to UAVs for ground surface fitting and dynamic separation, as provided in an embodiment of this specification.

[0033] like Figure 1 As shown, the method for dynamic target 3D localization and tracking applied to UAVs for surface fitting and dynamic separation includes at least the following steps:

[0034] Step S102: Continuously acquire image data of the target scene, extract image features from the image data and perform feature type segmentation, output the segmented feature types and image mask, the feature types include static regions and dynamic targets, and the static regions include planar static regions and vertical static regions.

[0035] Image data of the current scene is acquired. Specifically, when acquiring image data of the target scene, entire frames of the scene are captured at fixed time intervals. The image signals are then preprocessed. This preprocessing step includes two channels to simultaneously meet the requirements of SLAM (Simultaneous Localization and Mapping) geometric accuracy and the input size of the segmentation network. These two preprocessing channels are: SLAM geometry channel: First, distortion correction is performed on the original image (using pre-calibrated camera intrinsics and distortion coefficients), then it is converted to a single-channel grayscale image, maintaining the original image resolution (or downsampled to a fixed size, such as 640×480, according to the SLAM configuration). The output of this channel is used for feature extraction and pose estimation. Segmentation network channel: The original image is white-balanced, normalized, and scaled to the network's required input size (e.g., 640×640), maintaining the RGB three channels. The output of this channel is used for object detection and instance segmentation.

[0036] This method distinguishes between dynamic targets and static regions in image data by extracting image features. Feature discrimination can include: inputting the image into a pre-trained neural network, extracting features through multiple convolutional layers and attention mechanisms, performing bounding box regression and classification on the feature map (to determine the location and category of objects in the image), and mask generation (to generate accurate pixel-level segmentation for each instance). Specifically, the MMOT dataset is used as the core dataset for classifying targets in the neural network model, and its annotation results are used to differentiate between dynamic and static targets. Depending on the computing resources, different neural network models can be selected; for example, a lightweight model like YOLOv8 can be directly trained for bounding box object detection. For high-precision pixel-level detection (with higher tracking and localization requirements), a "DINOv3 backbone network + segmentation head" structure is used, loading a pre-trained DINOv3 model (pre-trained from satellite imagery) and an FPN-structured decoder to output the category of each pixel.

[0037] Then, based on the distinction between static and dynamic targets, further subdivisions are made according to the predefined geometric attributes of semantic categories. For example, static regions can be divided into planar static regions (such as roads, sidewalks, parking lots, grasslands, and bare soil (which can be used for height fitting of subsequent dynamic targets)), vertical static regions (such as building walls, fences, and tree trunks (which provide SLAM features but cannot be used as ground), and fixed detail regions (shrubs, rocks, and fixed facilities). Dynamic targets can be divided into rigid dynamics (cars, trucks, and motorcycles (shape stable)), non-rigid dynamics (pedestrians and animals (shape variable)), and special dynamics (rivers (texture changes but geometry may be static)). Each subclass distinction has its own subsequent use. For example, planar statics can be used for both surface modeling and SLAM feature point extraction in subsequent steps, vertical statics and fixed details are used for SLAM feature point extraction, and rigid and non-rigid dynamics are used for tracking and localizing dynamic targets. Special dynamics are completely excluded from subsequent SLAM static feature extraction and are not used for feature extraction. In surface modeling, if the riverbed is known, it can be considered a static surface but requires special processing.

[0038] Furthermore, after SLAM completes initialization and provides pose, a consistency check can be performed on the dynamic and static differentiation results of multi-frame image data to determine the differentiation error between dynamic targets and static regions. For example, classifying a "moving vehicle" as a static region. Between consecutive frames, using the inter-frame camera motion estimated by SLAM, feature points in the static region of the previous frame are projected onto the current frame to obtain the theoretical projection position. The actual corresponding position is obtained through feature matching or optical flow, and then the position residual is calculated. If the residual exceeds a threshold and shows consistency (most feature points in the region are moving), then the region is determined to be dynamic. For each region classified as "static," the difference between its actual motion and expected motion is calculated. If the region shows consistent residual motion, it is considered "dynamic," and relabeling is performed. That is, if a region labeled as static undergoes a change in its 3D position exceeding the noise range across multiple frames, the system will reclassify it as dynamic and update the mask.

[0039] Furthermore, in subsequent steps, the binary mask for each static region is decoded from the output data of the neural network. Mask instances with confidence scores below a confidence threshold are then discarded and classified as "uncertain masks." Morphological optimization is then performed on each instance mask, including but not limited to smoothing edges and filling small holes through morphological closing operations (dilation followed by erosion) to obtain the complete region. Noise is then removed through opening operations, and finally, boundary smoothing is performed through median filtering (kernel size 3×3) or conditional dilation and erosion.

[0040] The final generated mask can include four types:

[0041] 1. A global static binary mask is the union of all static category regions. The binary mask is an image containing only 0s and 1s, where 1 represents all static regions and 0 represents others. This mask can be directly used in subsequent SLAM operations. SLAM will extract ORB feature points in regions where the static mask is 1, fundamentally avoiding feature extraction from objects such as cars and pedestrians.

[0042] 2. The static planar region, which contains only planar categories such as "road" and "sidewalk", serves as the basis for surface modeling (its format is a set of outline points and category labels). It is the main data source for fitting the local ground plane and guides the selection of ground points in subsequent steps.

[0043] 3. Dynamic target instances, including an independent mask, bounding box, category, and confidence score for each dynamic target (e.g., vehicle number 1, pedestrian number 2). This serves as input to the dynamic target tracking and localization module, providing an initial state for each target to be tracked, enabling association with the corresponding dynamic target data in subsequent steps.

[0044] 4. Masks for uncertain regions, such as low-confidence regions, shadows, and specular reflection regions, serve as prohibited regions for subsequent SLAM static feature extraction and as ignored ground points for terrain modeling.

[0045] Step S104: Based on the mask of the static region, extract static feature points, combine the position of the dynamic target on the image to determine the ROI of the dynamic target, and traverse the feature point set of the static region based on the ROI. After matching with the feature points of the same semantic region in the historical frame, generate the 3D point cloud corresponding to the region where the dynamic target is located.

[0046] Specifically, this addresses the issues of pose estimation contamination (feature points on dynamic objects lead to incorrect camera motion estimation) and map contamination in traditional visual SLAM in dynamic scenes.

[0047] The input image data for SLAM static feature extraction is obtained. This input image data can include the grayscale image of the current frame (after preprocessing steps, including distortion correction and pyramid construction), a globally static binary mask (aligned with the current frame's timestamp), and the pose and local map of the previous frame (for projection matching). An ORB feature detector is run on pixel regions with a static mask value of 1. This ensures that all feature points originate from the static background.

[0048] Then, a constant-velocity motion model is used to predict the pose, projecting local map points onto the current frame. Matches are searched near the projected location, and PnP is used to solve for the pose. A preliminary pose estimate is performed, and after obtaining the preliminary pose, the reprojection error of all matching points is checked. Points with abnormally large errors (e.g., >3 pixels) are considered potentially dynamic points and removed from the current frame's matching. Then, the remaining static points are used for local BA optimization or single-frame pose optimization (optimizing the current frame pose while fixing the map point coordinates) to further refine the current frame pose and map point positions, minimizing the reprojection error. This ultimately determines the position of the dynamic target on the image.

[0049] Furthermore, while SLAM typically reconstructs the entire scene, this embodiment performs selective local reconstruction because reconstructing the entire scene consumes significant resources and computing power. The focus is placed on planar structures that dynamic targets may come into contact with or depend on. A mask for each dynamic target is obtained, and the pixel coordinates of its bottom center point in the image are determined. Based on the dynamic target's position in the image, its ROI (Region of Interest) is determined. The ROI can be a preset range surrounding the dynamic target, and this range is not limited to a single fixed value; it can be adjusted based on the target type and the drone's altitude. For example, the ROI is relatively smaller when the dynamic target is a pedestrian, and relatively larger when the dynamic target is a vehicle. Furthermore, by defining the ROI around the target and using the ROI as the range, SLAM ground reconstruction of the target area is performed. This saves resources and computing power by reconstructing the entire scene.

[0050] Within the Region of Interest (ROI) of a dynamic target, planar static regions (corresponding contour point sets) are retrieved. For each ROI, all planar static regions are traversed, and the intersection area between its contour and the ROI is calculated. Then, according to priority, the region with the first intersection area greater than a threshold (e.g., 50 pixels) is selected as the "supporting plane candidate" for the current target. The priority order can be dynamically adjusted based on the type of dynamic target; different types of dynamic targets have different priorities in their planar static region relationships. For example, for the dynamic target "vehicle," the planar static regions "road," "parking lot surface," etc., are searched first; for the dynamic target "pedestrian," the planar static regions "sidewalk," "road," "lawn," etc., are searched first; and for dynamic targets on rooftops, the planar static regions "building roof," etc., are searched first.

[0051] Then, from the ORB feature points extracted from the current frame image data, feature points located inside the planar static region are retained, and the matched points are matched with points of the same semantic region in previous image frames. During matching, to ensure that two planar static regions in different frames are "the same region," each planar static region needs to be assigned a globally unique identifier (ID). When a new frame arrives, the contour of the planar static region detected in the current frame is projected onto the world coordinate system (using the current frame pose) and matched with existing regions. If a match is successful (based on positional consistency and salient feature points), it is considered the same region and inherits the original ID.

[0052] In addition, for successfully matched points, condition filtering and quality checks are required. Once the conditions are met, the points are triangulated to generate new 3D point clouds. Constraints for condition filtering can include disparity angle (the angle between the optical center of two frames and the 3D point > 2° (to avoid depth uncertainty)), baseline length (the distance between the optical centers of two frames > 0.1 meters (to avoid degradation)), reprojection error (the reprojection error of the triangulated 3D point in both frames < 1 pixel), etc. After generating the 3D point cloud, it is relatively sparse in the overall image data but highly concentrated near dynamic targets, on planes that directly support target localization, making it extremely valuable. Each map point includes its corresponding (x, y, z), semantic category (static / dynamic target classification), confidence level, and timestamp. The output of newly added reliable static map points (3D point clouds with semantic labels) and planar region ID association information serves as the main input for subsequent surface modeling steps.

[0053] Step S106: Based on the 3D point cloud, establish a corresponding 3D map.

[0054] Specifically, 3D map points are discrete spatial points near dynamic targets. To locate the dynamic target in subsequent steps, it's necessary to continuously determine "the height of a certain point on the ground (the ground where the dynamic target is located)," achieving intelligent conversion from discrete point clouds to continuous height maps. This includes:

[0055] From the 3D map points in each frame, ground points are filtered. The methods for filtering ground points may include:

[0056] First, based on the feature type of step S102, 3D map points with the feature type of (planar) static (for surface modeling in subsequent steps) are selected. Then, ground points from these points can be further determined using methods including:

[0057] A simple threshold method based on altitude: The approximate altitude of the drone is estimated by SLAM, and the altitude range of ground points is determined. For example, the ground is no more than 100 meters or more than 10 meters below the drone's altitude, and ground points are selected within this range.

[0058] The residual method based on the reference plane (relatively the most accurate): calculate the distance of each point to the global reference plane (if the global reference plane has not yet been established, in the initial frame, the ground points determined by the simple height-based threshold method can be fitted as the initial reference plane for calculation). If the distance is less than the preset distance threshold, the point is considered to be a ground point.

[0059] Normal vector consistency check: Calculate the local normal vector for each point and its neighboring points, and then check the angle between the normal vector and the normal vector of the reference plane. Points with excessively large angles are identified, thus eliminating points that may be walls or other debris.

[0060] Based on the selected ground points, a global reference plane is established. That is, generating a 2D height map from a 3D point cloud requires a reference plane. When a sufficient number of ground points have been accumulated, the RANSAC plane fitting algorithm can be used to find the plane equation that brings the most points closer together, resulting in the robust global reference plane equation ax + by + cz + d = 0. This plane may not perfectly fit the actual undulating terrain, but it provides a stable, global 2D coordinate framework. Its core function is to reduce the dimensionality of all subsequent 3D point processing: the height z of a 3D point (x, y, z) can be seen as its "attribute value" at its corresponding position on the reference 2D plane.

[0061] Then, all ground points are vertically projected onto the reference plane, resulting in a series of two-dimensional points, labeled with their original height z. These two-dimensional points are then triangulated, forming an irregular mesh covering them, composed of countless triangles. The advantage of triangulation is that it automatically generates triangles that minimize acute angles, resulting in greater stability during numerical interpolation. A three-dimensional height field (TIN) is then formed, where each vertex of a triangle has not only two-dimensional coordinates but also its actual three-dimensional height. This creates a three-dimensional, continuous, irregular triangular mesh surface. For any point within the mesh, once it falls into a triangle, its height can be estimated by linear interpolation using the heights of the three vertices of that triangle. The TIN surface model (two-dimensional index + triangular mesh) will be used in subsequent steps for locating the footholds of dynamic targets. Furthermore, for new ground points, it is not necessary to reconstruct the entire triangular mesh from scratch. Only the connection relationships of nearby triangles need to be locally modified after projecting the new point, resulting in high computational efficiency.

[0062] Step S108: Using the contact point of the dynamic target in the image as the observation point, and combining it with the three-dimensional map, solve for the three-dimensional coordinates of the dynamic target, and combine it with the historical trajectory observation values ​​of the dynamic target to generate the motion trajectory prediction result of the dynamic target.

[0063] Specifically, in the image data, each dynamic target is located, and the foot point of each dynamic target is found, that is, the contact point between the target and the ground in the image. For rigid bodies such as vehicles, the pixel coordinates of the bottom center point of its 2D detection box are usually taken.

[0064] Then, a height constraint query is performed using a 3D map (TIN). By assuming the "footpoints" of the dynamic target are located on the digital terrain model, it's necessary to determine the z-value in the world coordinates of these "footpoints." However, (X, Y) is required to query the 3D height field and obtain Z, while solving for (X, Y) requires Z. Methods to break this cycle include iterative solutions or ray intersection, including:

[0065] Iterative solution: First, assume an initial Z (such as the average ground height) and calculate a rough (X,Y); use this (X,Y) to look up a height map to get a more accurate Zground; then use the new Zground to recalculate (X,Y), iterating until convergence.

[0066] Ray intersection:

[0067] a. Constructing a spatial ray: Using the camera's intrinsic parameters K and the UAV's current precise pose [R|t], backproject the pixel (u,v) into a three-dimensional spatial ray originating from the camera's optical center and passing through the pixel. All points on this ray can potentially be displayed as (u,v) in the image.

[0068] b. To accelerate queries, a two-dimensional spatial index (such as a quadtree or raster) is constructed for the TIN, projecting the triangle onto a reference plane and recording the coverage area. When calculating the intersection of rays, the intersection points with the reference plane are calculated first to locate candidate triangles.

[0069] c. If the ray has no intersection with any triangle (e.g., the target is outside the TIN coverage area), then revert to the iterative solution method, using the global reference plane height as the initial Z. When an intersection exists, it represents the three-dimensional coordinates of the dynamic target. This method completely avoids triangulation of the target depth and directly utilizes the static environment as geometric constraints.

[0070] Based on the above localization process, the single-frame localization process can be determined. For each tracked target, the pixel coordinates (u, v) of the bottom center point of its 2D bounding box (usually in contact with the ground) can be taken. Then, the Z-ground is determined by querying the ground elevation map. Using the camera intrinsic parameter K and the UAV's current precise pose [R|t], the equation s*[u,v,1] is solved. T =K*[R|t]*[X,Y,Zground,1] T Solve for the target's current (X,Y) to determine the complete (X,Y,Z ground).

[0071] Furthermore, the current frame's dynamic target instance list (including bounding boxes, individual masks, and categories) output in step S102 is received and associated with the dynamic target trajectories tracked in the previous frame. DeepSORT or a Hungarian matching algorithm based on IoU and position prediction can be used to assign an existing trajectory ID or create a new ID for each detected instance. The association result ensures that each target has a continuous observation sequence.

[0072] For each associated dynamic target, its foot pixel coordinates (u,v) are used in conjunction with the TIN constructed in step S106 to calculate the world coordinates (X,Y,Z) of the target's foot in the current frame using iterative solution or ray intersection method. The confidence level of the localization result is quantified by factors such as reprojection error and ray intersection stability, and used as the observation covariance for subsequent optimization.

[0073] Then, a sliding window of length N (e.g., 20 frames) is maintained for each dynamic target, containing the 3D localization observations of the target in each frame within the window and the UAV pose (from SLAM) of the corresponding frame. A graph optimization problem is constructed, with the target position in each frame as the vertex, the localization observations and inter-frame motion model as the edges, and the UAV pose as the fixed parameter. Then, the problem is solved using the g2o or Ceres algorithm, outputting the optimized target position and velocity for the current frame. The optimized trajectory can be used for motion prediction and obstacle avoidance decisions. This invention provides a method for 3D localization and tracking of dynamic targets applied to UAVs for surface fitting and dynamic separation. It explicitly decomposes the 3D localization process of dynamic targets using monocular vision into four interdependent, data-closed-loop steps: "semantic-based dynamic-static object detection and classification," "SLAM reconstruction and surface modeling based on pure static features," "single-frame 3D solution of dynamic targets based on surface height constraints," and "joint optimization of multi-frame observations and motion models." This solves the core problems of existing technologies, such as difficulty in acquiring depth information, poor environmental adaptability, and high cost. By fully utilizing the dominant and reconstructable static environmental information in dynamic scenes, a reliable height constraint benchmark is provided for dynamic targets that are difficult to triangulate directly. This method provides a way to stably estimate the three-dimensional coordinates of moving targets without relying on expensive sensors, adapting to non-flat terrain.

[0074] Furthermore, based on the flowchart of the previous embodiment, a detailed implementation flowchart of a method for dynamic target 3D localization and tracking applied to UAVs for surface fitting and dynamic separation can be as follows: Figure 2 As shown, in Figure 2The system acquires monocular video stream input and uses the YOLO / Dinov3 model for object detection and semantic segmentation to perform feature type segmentation on the image data, achieving separation and classification of static and dynamic targets. On one hand, static background features, such as stones, zebra crossings, and buildings, are extracted. The scene is reconstructed using SLAMORB-SLAM3 based on sparse features to determine the 3D coordinates of these static feature points. A RANSAC or triangulation mesh is then fitted to the ground model to generate a continuous ground elevation map. On the other hand, dynamic foreground targets (vehicles, pedestrians, etc.) are tracked in 2D using Kalman filtering / optical flow. By combining the coordinate calculations from both aspects, and integrating the target's 2D position and ground elevation, the 3D coordinates of the dynamic target are determined. Finally, multi-frame trajectory optimization / filtering is used to output the optimized 3D trajectory and velocity of the dynamic target.

[0075] Please refer to the following. Figure 3 , Figure 3 This diagram illustrates the structure of a system for dynamic target 3D localization and tracking applied to UAVs for ground surface fitting and dynamic separation, as provided in an embodiment of this specification. It should be noted that... Figure 3 The system shown for dynamic target 3D localization and tracking, applied to UAV surface fitting and dynamic separation, is used to perform the functions described in this specification. Figure 1 The methods shown in the embodiments are illustrated for ease of explanation, showing only the parts related to the embodiments of this specification. For specific technical details not disclosed, please refer to this specification. Figure 1 The example shown.

[0076] like Figure 3 As shown, the system for dynamic target 3D localization and tracking applied to UAVs for surface fitting and dynamic separation may include at least:

[0077] The acquisition module S302 is used to continuously acquire image data of the target scene, extract image features from the image data and perform feature type segmentation, and output the segmented feature types and image mask. The feature types include static regions and dynamic targets. The static regions include planar static regions and vertical static regions.

[0078] The feature extraction module S304 is used to extract static feature points based on the mask of the static region, determine the ROI of the dynamic target by combining the position of the dynamic target on the image, and traverse the feature point set of the static region based on the ROI. After matching with the feature points of the same semantic region in the historical frame, a 3D point cloud corresponding to the region where the dynamic target is located is generated.

[0079] The 3D map module S306 is used to build a corresponding 3D map based on the 3D point cloud;

[0080] The trajectory module S308 is used to calculate the three-dimensional coordinates of the dynamic target by taking the contact point of the dynamic target in the image as the observation point and combining it with the three-dimensional map, and to generate the motion trajectory prediction result of the dynamic target by combining it with the historical trajectory observation value of the dynamic target.

[0081] In another embodiment, a system for dynamic target 3D localization and tracking applied to surface fitting and dynamic separation of unmanned aerial vehicles further includes:

[0082] The pre-training module is used to input the image data into a pre-trained neural network and output the pixel categories in the image data;

[0083] The mask module is used to generate a corresponding mask based on the pixel category, wherein the mask includes a binary mask for static regions, a planar static region mask, a dynamic target instance mask, and an uncertain region mask.

[0084] In another embodiment, a system for dynamic target 3D localization and tracking applied to UAVs for surface fitting and dynamic separation further includes: an acquisition module for acquiring the pixel coordinates of the bottom center point of the dynamic target to define the ROI around the target, and performing SLAM surface reconstruction of the target area within the ROI.

[0085] In another embodiment, a system for dynamic target 3D localization and tracking applied to surface fitting and dynamic separation of unmanned aerial vehicles further includes:

[0086] The depth module is used to back-project the bottom center point of the dynamic target and estimate the depth by intersecting the back-projection direction with the ground surface of the image.

[0087] Those skilled in the art will clearly understand that the technical solutions of the embodiments in this specification can be implemented by means of software and / or hardware. In this specification, "unit" and "module" refer to software and / or hardware capable of independently performing or cooperating with other components to perform a specific function. The hardware may be, for example, a Field-Programmable Gate Array (FPGA), an Integrated Circuit (IC), etc.

[0088] Each processing unit and / or module in the embodiments of this specification can be implemented by an analog circuit that implements the functions described in the embodiments of this specification, or by software that executes the functions described in the embodiments of this specification.

[0089] See Figure 4 It shows a schematic diagram of the structure of an electronic device according to an embodiment of this specification, which can be used to implement... Figure 1 The method in the illustrated embodiment. (As shown) Figure 4As shown, the electronic device 400 may include: at least one central processing unit 401, at least one network interface 404, user interface 403, memory 405, and at least one communication bus 402.

[0090] The communication bus 402 is used to enable communication between these components.

[0091] The user interface 403 may include a display screen and a camera. Optionally, the user interface 403 may also include a standard wired interface and a wireless interface.

[0092] The network interface 404 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0093] The processor 401 may include one or more processing cores. The processor 401 connects to various parts within the electronic device 400 using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 405, and by calling data stored in the memory 405. Optionally, the processor 401 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 401 may integrate one or a combination of several of the following: a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and a modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display; and the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip, without being integrated into the processor 401.

[0094] The memory 405 may include random access memory (RAM) or read-only memory. Optionally, the memory 405 may include a non-transitory computer-readable storage medium. The memory 405 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 405 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 405 may also be at least one storage system located remotely from the aforementioned processor 401. Figure 4 As shown, the memory 405, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and program instructions.

[0095] exist Figure 4 In the illustrated electronic device 400, the user interface 403 is mainly used to provide an input interface for the user and to acquire user input data; while the processor 401 can be used to call the image-based interactive application stored in the memory 405 and specifically perform the following operations:

[0096] Continuously acquire image data of the target scene, extract image features from the image data and perform feature type segmentation, output the segmented feature types and image mask, the feature types include static regions and dynamic targets, the static regions include planar static regions and vertical static regions;

[0097] Based on the mask of the static region, static feature points are extracted. Combined with the position of the dynamic target on the image, the ROI of the dynamic target is determined. Based on the ROI, the feature point set of the static region is traversed. After matching with the feature points of the same semantic region in the historical frame, a 3D point cloud corresponding to the region where the dynamic target is located is generated.

[0098] Based on the 3D point cloud, a corresponding 3D map is established;

[0099] Using the contact point of the dynamic target in the image as the observation point, the target is projected onto the three-dimensional map to solve for the three-dimensional coordinates of the dynamic target. Combined with the historical trajectory observations of the dynamic target, the motion trajectory prediction result of the dynamic target is generated.

[0100] As an optional embodiment of this specification, after acquiring the image data of the target scene, the method further includes:

[0101] The image data is input into a pre-trained neural network, which outputs the pixel categories in the image data.

[0102] Based on the pixel category, a corresponding mask is generated, wherein the mask includes a binary mask for static regions, a planar static region mask, a dynamic target instance mask, and an uncertain region mask.

[0103] As an optional embodiment of this specification, determining the ROI of the dynamic target by defining its position on the image includes:

[0104] The bottom center point pixel coordinates of the dynamic target are obtained to define the ROI around the target. SLAM surface reconstruction of the target area is then performed within the ROI.

[0105] As an optional embodiment of this specification, the depth calculation step of the dynamic target includes:

[0106] The depth is estimated by back-projecting the bottom center point of the dynamic target and intersecting it with the ground surface of the image along the back-projection direction.

[0107] This specification also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method. The computer-readable storage medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, DVDs, CD-ROMs, microdrives, as well as magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic cards or optical cards, nanosystems (including molecular memory ICs), or any type of medium or device suitable for storing instructions and / or data.

[0108] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this specification is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this specification. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this specification.

[0109] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0110] In the embodiments provided in this specification, it should be understood that the disclosed system can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some service interface; the indirect coupling or communication connection between systems or units may be electrical or other forms.

[0111] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0112] Furthermore, the functional units in the various embodiments of this specification can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0113] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this specification, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this specification. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0114] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0115] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

Claims

1. A method for dynamic target three-dimensional localization and tracking applied to unmanned aerial vehicles (UAVs), the method comprising: Continuously acquire image data of the target scene, extract image features from the image data and distinguish feature types, output feature types and segmented image masks, wherein the feature types include static regions and dynamic targets; Based on the mask of the static region, the feature point set of the static region is extracted. Combined with the position of the dynamic target on the image, the dynamic target region SLAM surface reconstruction is performed to determine the ROI of the dynamic target. Based on the ROI, the feature point set of the static region is traversed. After matching with the feature points of the same semantic region in the historical frame, a 3D point cloud corresponding to the static region where the dynamic target is located is generated. Based on the 3D point cloud, a corresponding 3D map is established; Using the contact point of the dynamic target in the image as the 2D observation point, it is projected onto the 3D map to obtain the corresponding ground elevation. The 2D position of the dynamic target is calculated using the ground elevation constraint. Combined with the ground elevation, the 3D coordinates of the dynamic target are solved. Combined with the historical trajectory observation values ​​of the dynamic target, the motion trajectory and prediction results of the dynamic target are generated.

2. The method according to claim 1, characterized in that, After acquiring the image data of the target scene, the method further includes: The image data is input into a pre-trained neural network, which outputs the pixel categories in the image data. Based on the pixel category, a corresponding mask is generated, wherein the mask includes a binary mask for static regions, a planar static region mask, a dynamic target mask, and an uncertain region mask.

3. The method according to claim 1, characterized in that, Determining the ROI of the dynamic target by combining its position on the image includes: The bottom center point pixel coordinates of the dynamic target are obtained to define the ROI around the target. SLAM surface reconstruction of the dynamic target area is then performed within the ROI.

4. A system for dynamic target three-dimensional positioning and tracking applied to unmanned aerial vehicles (UAVs), characterized in that, The system includes; The acquisition module is used to continuously acquire image data of the target scene, extract image features from the image data and distinguish feature types, and output the feature types and the segmented image mask. The feature types include static regions and dynamic targets. The feature extraction module is used to extract the feature point set of the static region based on the mask of the static region, combine the position of the dynamic target on the image, perform SLAM surface reconstruction of the dynamic target region, determine the ROI of the dynamic target, and traverse the feature point set of the static region based on the ROI. After matching with the feature points of the same semantic region in the historical frame, a 3D point cloud corresponding to the static region where the dynamic target is located is generated. A 3D map module is used to create a corresponding 3D map based on the 3D point cloud; The trajectory module is used to project the contact point of the dynamic target in the image as a 2D observation point onto the 3D map to obtain the corresponding ground elevation. The 2D position of the dynamic target is calculated using the ground elevation constraint. The 3D coordinates of the dynamic target are solved by combining the ground elevation and the historical trajectory observation values ​​of the dynamic target. The motion trajectory and prediction results of the dynamic target are generated by combining the historical trajectory observation values ​​of the dynamic target.

5. The system according to claim 4, characterized in that, The system also includes: The pre-training module is used to input the image data into a pre-trained neural network and output the pixel categories in the image data; The mask module is used to generate a corresponding mask based on the pixel category, wherein the mask includes a binary mask for static regions, a planar static region mask, a dynamic target mask, and an uncertain region mask.

6. The system according to claim 4, characterized in that, The system also includes: The acquisition module is used to acquire the pixel coordinates of the bottom center point of the dynamic target, define the ROI around the target, and perform SLAM surface reconstruction of the dynamic target area within the ROI.

7. An electronic device, comprising a processor and a memory; The processor is connected to the memory; The memory is used to store executable program code; The processor runs a program corresponding to the executable program code stored in the memory to perform the method as described in any one of claims 1-3.

8. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of claims 1-3.

Images