Dynamic obstacle detection and tracking method in weak light environment

Abstract

The invention discloses a method for detecting and tracking a dynamic obstacle in a weak light environment. The method comprises the following steps: 1) acquiring an image; (2) a U-depth detector and a DBSCAN detector are used for conducting rapid detection on a near obstacle, and after weak light images of a far obstacle are enhanced through a lightweight image enhancement network LPF, a lightweight FastDet model is input to conduct deep learning auxiliary detection; 3) calculating the intersection-to-union ratio of the U-depth detector, the DBSCAN detector and the LPF-FastDet detector, fusing the intersection-to-union ratio results of the U-depth detector, the DBSCAN detector and the LPF-FastDet detector, and generating a set bounding box; 4) performing data association on the detected obstacles through a feature association method, and performing continuous tracking by using Kalman filtering; and 5) identifying a dynamic obstacle from all detected obstacles, and outputting the size of a three-dimensional bounding box of the dynamic obstacle and the center position coordinates of the dynamic obstacle in the unmanned aerial vehicle coordinate system.The method provides a high-precision and low-delay solution for obstacle perception of the unmanned aerial vehicle in a weak light environment, and is suitable for different scene requirements.

Description

Technical Field

[0001] The present invention relates to the field of intelligent robot perception technology, and in particular to a method for detecting and tracking dynamic obstacles in a low-light environment. Background Art

[0002] In recent years, drones, particularly multi-rotor aircraft, have been widely used in civilian industries due to their high maneuverability and cost-effectiveness. They play a significant role in various fields, including forest fire monitoring, factory warehouse inspections, express delivery, and light shows. For autonomous drones, the ability to safely perform various missions and possess basic decision-making skills are crucial. In low-light mission environments, the quality of image data sources is often affected by insufficient lighting, which severely impairs the drone's ability to perceive surrounding obstacles and significantly reduces the success rate of drone missions. To ensure drone flight safety, drones must be able to identify and perceive dynamic obstacles in low-light environments, ensuring their safety and reliability in low-light environments.

[0003] The following defects exist in drone perception scenarios:

[0004] (1) Insufficient adaptability of a single detection method: Traditional methods rely solely on non-learning detectors or learning detectors. In low-light environments, it is difficult to balance the detection accuracy and real-time requirements of near- and long-distance obstacles, resulting in insufficient close-range detection speed or decreased long-range detection accuracy.

[0005] (2) Separation of low-light image enhancement and detection: Existing solutions usually treat image enhancement and obstacle detection as independent modules, and fail to optimize the collaborative mechanism between the lightweight enhancement network and the detection model for UAV scenarios, resulting in a waste of computing resources and limited real-time performance.

[0006] (3) Lack of multi-detector fusion strategy: Traditional methods lack a dynamic fusion mechanism (such as the ensemble detection of U-depth and YOLO), and are unable to adaptively select the optimal detection strategy based on the distance to the obstacle. In complex scenes with low light, it is easy to miss detections at close range or misdetect at far distances. Summary of the Invention

[0007] This invention addresses the challenges of existing obstacle perception methods in low-light environments, such as the difficulty of using a single detector to detect both near and far distances, high rates of missed and false detections, and poor tracking consistency. This method proposes a method for dynamic obstacle detection and tracking in low-light environments. This method combines the advantages of both non-learning and learning detectors, employing an ensemble detection strategy to improve accuracy and real-time performance. Nearby obstacles are rapidly detected using a U-depth detector and the DBSCAN clustering algorithm. For distant obstacles, the low-light image is enhanced using a lightweight image enhancement network (LPF) and then fed into a lightweight FastestDet model for deep learning-assisted detection. By dynamically fusing the results of different detectors, high-precision, real-time obstacle detection is achieved in low-light environments. Furthermore, the candidate bounding boxes returned by each detector, along with their geometric, depth, and point cloud statistical features, are normalized. Inter-frame matching is performed based on a multivariate feature correlation metric to construct a tracking object. Finally, a Kalman filter with a constant acceleration model is used to smoothly estimate and predict the obstacle state (position, velocity, and acceleration). This allows the drone to continuously and robustly track dynamic targets even when the object is temporarily absent, obstructed, or undergoes drastic changes in illumination.

[0008] The technical solution for achieving the purpose of the present invention is:

[0009] A method for detecting and tracking dynamic obstacles in a low-light environment, the method comprising the following steps:

[0010] 1) After acquiring the depth image and RGB image of the environment in front of the drone, an ensemble detection strategy is used to improve the accuracy and real-time performance of the drone's detection of dynamic obstacles;

[0011] 2) For obstacles within 5 meters of the drone, two non-learning detectors, the U-depth detector and the DBSCAN detector, are used for rapid detection. For obstacles farther away than 5 meters, the low-light image is enhanced using the lightweight image enhancement network LPF (Learn Periodic Features) and then input into the lightweight FastestDet model for deep learning-assisted detection.

[0012] 3) By calculating the intersection-over-union (IOU) of the U-depth detector, DBSCAN detector, and LPF-FastestDet detector, and fusing the IOU results of the three, a collective bounding box is generated to achieve high-precision and real-time obstacle detection in low-light environments;

[0013] 4) Data association is performed on detected obstacles using feature association methods, and continuous tracking is performed using Kalman filtering;

[0014] 5) Identify dynamic obstacles from all detected obstacles and output the 3D bounding box size (length ,Width ,high ) and its center position coordinates in the drone coordinate system ( ).

[0015] In step 2), the U-depth detector converts the depth image into a U-depth map and combines depth continuity and linear grouping methods to locate the center coordinates and three-dimensional bounding box size of the obstacle. However, the detection effect will be significantly reduced when the depth data is noisy. The DBSCAN detector divides the point cloud into multiple clusters by analyzing the local density of the point cloud data and generates a three-dimensional bounding box of the obstacle based on these clusters. However, it is sensitive to parameter settings and has a large amount of computation when the scene is complex. By fully leveraging the advantages of both methods and compensating for their disadvantages, more robust and accurate obstacle detection and positioning can be achieved in low-light dynamic environments.

[0016] Before converting the depth image into a U-depth map in step 2), the U-depth detector preprocesses the collected RGB image and depth image of the environment in front of the drone and calculates a depth value histogram for each column of pixels in the depth image. The U-depth detector then converts the preprocessed depth image into a U-depth map. Each pixel value in the U-depth map represents the height or density of obstacles at that location, providing a basis for subsequent detection. Next, a linear grouping algorithm is used to identify obstacle regions in the U-depth map and generate a two-dimensional bounding box. The width of this two-dimensional bounding box is determined by the obstacle width in the U-depth map, and its length is determined by the depth range in the depth value histogram. After generating the two-dimensional bounding box, a depth continuity search is performed on the original depth image to determine the height information of the obstacle. By checking the depth value continuity and combining it with the width information in the U-depth map, a three-dimensional bounding box of the obstacle is generated. Finally, the two-dimensional bounding box and the depth continuity information are converted to the drone coordinate system to generate the U-depth bounding box.

[0017] In the process of converting the depth image into the U-depth map by the U-depth detector in step 2), the coordinate transformation method cited in formula (1) is adopted, where and The horizontal position of the obstacle in the collective coordinate system; and are the length and width of the obstacle in the body coordinate system; is the focal length of the camera; , is the upper left corner position of the obstacle in the body coordinate system; , is the lower right corner position of the obstacle in the body coordinate system;

[0018]

[0019] The conversion of point clouds in the DBSCAN detector uses the same internal parameters as the RGB-D camera for inverse transformation. The fixed correspondence between the 3D camera point [x, y, z] and its image pixel coordinates [u, v, d] is shown in Equation (2), where fx and fy are the focal lengths of the camera on the x and y axes; cx and cy are the center of the camera aperture; and s represents the scaling factor of the depth map. Since the RGB image used for color images has been aligned in the process of converting the depth image to a point cloud by the DBSCAN detector, no separate coordinate conversion is required.

[0020]

[0021] In step 2), the DBSCAN detector detects obstacles and converts the depth map into a point cloud. A voxel filter is then used to denoise the point cloud, removing noise points and retaining valid point cloud data. Next, the DBSCAN algorithm is applied to cluster the point cloud data. DBSCAN automatically divides the point cloud into multiple clusters by identifying dense areas within the point cloud. Each cluster represents a potential obstacle. After clustering, a 3D bounding box is generated for each cluster. The center coordinates and size of the DBSCAN bounding box are determined by the distribution of the point cloud within the cluster.

[0022] In step 2), after enhancing the low-light image of distant obstacles greater than or equal to 5 meters from the drone using the lightweight image enhancement network LPF (Learn Periodic Features), the image is then input into the lightweight FastestDet model for deep learning-assisted detection.

[0023] 2.1) The LPF-FastestDet detector acquires RGB images from an RGB-D camera and uses an LPF network to enhance low-light RGB images. The LPF network effectively improves image quality in low-light environments by learning periodic features, providing clearer input for subsequent object detection.

[0024] 2.2) The enhanced RGB image is input into the FastestDet model for object detection. FastestDet is a lightweight object detection algorithm that uses a 5×5 grouped convolutional parallel network structure to integrate features from different receptive fields. It adaptively detects objects of varying sizes using a single lightweight detection head. Designed for the computationally scarce ARM platform commonly used by drones, it effectively reduces network parameter size and runtime while ensuring high detection accuracy.

[0025] 2.3) LPF-FastestDet uses a depth-wise separable convolutional module to simultaneously predict the target classification and bounding box regression, further improving the detection efficiency.

[0026] In the deep learning detector, a lightweight LPF network is used to enhance low-light RGB images in real time to improve texture details. The LPF network extracts frequency domain features through a periodic residual module and performs supervised training on low-light and normal-light image pairs (such as the LOL dataset). The SSIM and L1 loss functions are used to optimize the network parameters. The enhanced image significantly restores edge and texture information. The enhanced image is then input into the FastestDet model for deep learning target detection. The FastestDet model is based on the MobileNetV3 lightweight backbone network, uses 5×5 grouped convolutions to extract multi-scale features in parallel, predicts target classification and bounding box regression through a depth-separable convolution module, and performs model quantization (INT8) and layer fusion when deployed on ARM platforms (such as NVIDIA Jetson Nano), achieving real-time inference as low as 15ms.

[0027] Before fusing the IoU results of the three in step 3), the detection results of the non-learning detector are first integrated. If the obstacle is greater than or equal to 5m away from the drone, the learning-based detection module is activated. Subsequently, the results of the U-depth, DBSCAN, and LPF-FastestDet detectors are combined to improve the overall detection accuracy.

[0028]

[0029] in is the scale of the integrated detection bounding box (w, l, h); 、 、 They are the 3D bounding box scales from U-depth detector, DBSCAN detector, and LPF-FastestDet detector respectively; 、 、 are the predetermined weights of the three bounding boxes.

[0030] The specific steps for generating the set bounding box in step 3) are as follows:

[0031] 3.1) Obtain the 3D bounding boxes of the U-depth detector, DBSCAN detector, and lightweight object detection model FastestDet respectively;

[0032] 3.2) Calculate the intersection over union (IOU) ratio for each pair of 3D bounding boxes. When the IOU exceeds a predetermined threshold, the detection result is confirmed to be the same obstacle.

[0033] 3.3) Fuse the detection boxes that meet the IOU condition, take the average value of the center position and the maximum value of the size of the 3D bounding boxes of the U-depth detector, DBSCAN detector, and LPF-FastestDet detector, and generate a collective 3D bounding box.

[0034] The specific steps of the feature association method in step 4) are as follows:

[0035] 4.1) Defining a feature vector for each obstacle, wherein the feature vector includes the obstacle's center position, size, and point cloud statistical features;

[0036] 4.2) Calculate the similarity score between the obstacle detected at the current moment and the obstacle detected at the previous moment;

[0037] 4.3) Associate the current obstacle with the obstacle with the highest similarity score at the previous moment.

[0038] The specific steps for identifying dynamic obstacles in step 5) are as follows:

[0039] 5.1) Calculate the center velocity based on the change in the obstacle center position between two consecutive frames, i.e., time t-1 and time t :

[0040] The three-dimensional coordinates of the obstacle center at time t are ( ),exist The three-dimensional coordinates of the moment are ( ),in: Respectively represent the obstacle center at Time along The coordinates of the axes, Respectively represent the obstacle center at Time along Axis coordinates, center speed By calculating the Euclidean distance between the obstacle center and the two frames Divide by the time interval Get, that is:

[0041]

[0042] in , is the time interval between two consecutive frames. Compare with the predetermined threshold to preliminarily screen out dynamic obstacles;

[0043] 5.2) For the dynamic obstacle candidate targets initially screened in step 5.1), calculate the displacement vector of each point on the obstacle surface by aligning the adjacent frame point clouds, and combine the time interval Calculate the point speed; vote based on the speed distribution of each point, if the speed exceeds the threshold The percentage of points is greater than the set ratio , it is confirmed as a dynamic obstacle, further confirming the dynamic nature of the obstacle, which complements the central velocity screening in step 5.1) and provides more robust dynamic obstacle recognition;

[0044] 5.3) Output the 3D bounding box size of the dynamic obstacle (length ,Width ,high ) and its center position coordinates in the drone coordinate system ( ), used for drone navigation and obstacle avoidance.

[0045] The depth image and RGB image obtained in step 1) are used in a collective detection strategy to detect dynamic and static obstacles in the surrounding environment information obtained by the perception module in real time.

[0046] In step 2), the lightweight image enhancement network (LPF) and the lightweight object detection model FastestDet enhance low-light images using the LPF, improving detail before inputting them into FastestDet for accurate detection. While maintaining low computational complexity, the LPF-FastestDet detector significantly improves the accuracy of long-range obstacle detection, overcoming the limitations of traditional RGB cameras in low-light environments and providing drones with more reliable environmental perception capabilities.

[0047] In step 3), the high-precision depth information processing capability of the U-depth detector and the robust noise suppression characteristics of the DBSCAN detector are integrated, while the image quality improvement capability and efficient target detection performance in low-light environments are combined. This compensates for the insufficient accuracy of non-deep learning detectors in long-distance obstacle detection, and achieves high-precision, low-complexity long-distance obstacle detection in low-light environments.

[0048] In step 4), the Kalman filter uses a constant acceleration motion model to perform state estimation on the position, velocity, and acceleration of the obstacle to achieve continuous tracking.

[0049] Beneficial effects of this technical solution:

[0050] This technical solution addresses the shortcomings of existing technologies in drone perception scenes in low-light environments. Through an innovative collective detection strategy, it significantly improves the accuracy, real-time performance, and environmental adaptability of obstacle detection and tracking. To address the problem of insufficient adaptability of a single detection method, this technical solution combines the advantages of non-learning detectors and learning detectors. In the detection of obstacles within a distance of less than 5m from the drone, it combines the depth histogram analysis of the U-depth map and the DBSCAN point cloud density clustering to achieve fast positioning and noise suppression. In the detection of obstacles at a distance of 5m or more from the drone, the details of the low-light image are enhanced through a lightweight LPF network and then input into the FastestDet model for efficient recognition, thereby taking into account both short-range detection speed and long-range detection accuracy, avoiding the performance limitations of traditional single detection methods in low-light environments.

[0051] To address the resource waste caused by the separation of low-light image enhancement and detection, this technical solution utilizes a collaborative design of a lightweight LPF network and the FastestDet model. The LPF network is specifically optimized for low-light UAV scenarios, restoring image texture through periodic feature learning, directly improving the input quality of the detection model. The FastestDet model, through its lightweight structure and computational optimization, achieves low-latency inference on embedded platforms, overcoming the computational redundancy of traditional separate processing and significantly improving the real-time performance of UAVs. This effectively facilitates real-time dynamic obstacle detection on UAVs with limited computing resources.

[0052] In addition, to address the lack of multi-detector fusion strategies, this technical solution proposes an adaptive weight allocation mechanism based on obstacle distance to dynamically fuse the detection results of U-depth, DBSCAN, and LPF-FastestDet. The same target is determined by the intersection-over-union (IOU) threshold, and the weight of each detector is automatically adjusted according to the obstacle distance. For example, geometric feature detectors are prioritized for close detections, while deep learning models are relied upon for far detections. This effectively reduces missed detections in the near distance and false detections in the far distance, and improves the robustness of detection in complex low-light scenes. In dynamic target tracking, feature association and Kalman filter prediction are combined with speed threshold screening and point cloud speed voting mechanisms to ensure stable tracking of dynamic obstacles even when occluded or with changing illumination. Compared with traditional color feature-based methods, this significantly reduces false detection rates and enhances trajectory continuity.

[0053] Ultimately, this technical solution provides a high-precision, low-latency solution for drone obstacle perception in low-light environments through multi-detector collaboration and adaptive fusion strategies, adapting to the needs of different scenarios and significantly improving the drone's obstacle avoidance efficiency and exploration capabilities in unknown and complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 is a system block diagram of an embodiment;

[0055] Figure 2 A comparison chart of detection accuracy based on learning and non-learning detectors in an embodiment;

[0056] Figure 3 The obstacle detection map based on U-depth in the embodiment, where (a) RGB image at time t1 (b) RGB image with light at time t1 (c) u-depth map (d) depth map;

[0057] Figure 4 This is an obstacle detection map based on DBSCAN in the embodiment;

[0058] Figure 5 The obstacle detection image based on LPF-FastestDet in the embodiment, including (a) actual RGB image (b) LPF enhanced RGB image (c) aligned depth image (d) obstacle 3D box;

[0059] Figure 6 This is the effect diagram of the embodiment deployed on the drone, including (a) actual RGB image (b) U-depth image (c) aligned depth map (d) obstacle bird's-eye view (e) obstacle occupancy map. DETAILED DESCRIPTION

[0060] The present invention will be further described below with reference to the accompanying drawings and embodiments, but the present invention is not limited thereto.

[0061] Example:

[0062] Reference Figure 1 A method for detecting and tracking dynamic obstacles in a low-light environment comprises the following steps:

[0063] 1) After acquiring the depth image and RGB image of the environment in front of the drone, an ensemble detection strategy is used to improve the accuracy and real-time performance of the drone's detection of dynamic obstacles;

[0064] 2) For obstacles that are closer than 5 meters from the drone, two non-learning detectors, U-depth and DBSCAN, are used for rapid detection. For obstacles that are farther away than 5 meters from the drone, the low-light image is enhanced by the lightweight image enhancement network LPF and then input into the lightweight FastestDet model for deep learning-assisted detection. Figure 2 As shown;

[0065] 3) By calculating the intersection-over-union (IoU) of the U-depth detector, the DBSCAN detector, and the LPF-FastestDet detector, and fusing the IoU results, a collective bounding box is generated to achieve high-precision, real-time obstacle detection in low-light environments.

[0066] 4) Data association is performed on detected obstacles using feature association methods, and continuous tracking is performed using Kalman filtering;

[0067] 5) Identify dynamic obstacles from all detected obstacles and output the 3D bounding box size (length ,Width ,high ) and its center position coordinates in the drone coordinate system ( ).

[0068] In step 2), the U-depth detector converts the depth image into a U-depth map, and combines the depth continuity and linear grouping method to locate the center coordinates of the obstacle and the size of the three-dimensional bounding box. Figure 3 As shown in Figure 2; the DBSCAN detector divides the point cloud into multiple clusters by analyzing the local density of the point cloud data and generates a three-dimensional bounding box of the obstacle based on these clusters, as shown in Figure 2. Figure 4 shown.

[0069] In the step 2), before the U-depth detector converts the depth image into a U-depth map, the collected RGB image and depth image of the environment in front of the drone are preprocessed, and the depth value histogram of each column of pixels in the depth image is calculated. Then, the U-depth detector converts the preprocessed depth image into a U-depth map. Each pixel value of the U-depth map represents the height or density of the obstacle at that position, providing a basis for subsequent detection. Next, the obstacle area is identified on the U-depth map through a linear grouping algorithm to generate a two-dimensional bounding box. The width of the two-dimensional bounding box is determined by the width of the obstacle in the U-depth map, and the length is determined by the depth range in the depth value histogram. After generating the two-dimensional bounding box, a depth continuity search is required on the original depth image to determine the height information of the obstacle. By checking the continuity of the depth value and combining the width information in the U-depth map, a three-dimensional bounding box of the obstacle is generated. Finally, the two-dimensional bounding box and the depth continuity information are converted into the drone coordinate system to generate a three-dimensional bounding box, as shown in FIG. Figure 3 shown.

[0070] In the process of converting the depth image into the U-depth map by the U-depth detector in step 2), the coordinate transformation method cited in formula (1) is adopted, where and The horizontal position of the obstacle in the collective coordinate system; and are the length and width of the obstacle in the body coordinate system; is the focal length of the camera; , is the upper left corner position of the obstacle in the body coordinate system; , is the lower right corner position of the obstacle in the body coordinate system;

[0071]

[0072] The conversion of point clouds in the DBSCAN detector uses the same internal parameters as the RGB-D camera for inverse transformation. The fixed correspondence between the 3D camera point [x, y, z] and its image pixel coordinates [u, v, d] is shown in Equation (2), where fx and fy are the focal lengths of the camera on the x and y axes; cx and cy are the center of the camera aperture; and s represents the scaling factor of the depth map. Since the RGB image used for color images has been aligned in the process of converting the depth image to a point cloud by the DBSCAN detector, no separate coordinate conversion is required.

[0073]

[0074] In step 2), the DBSCAN detector detects obstacles and converts the depth map into a point cloud. The voxel filter is used to denoise the point cloud, remove noise points, and retain valid point cloud data. Next, the DBSCAN algorithm is applied to cluster the point cloud data. DBSCAN automatically divides the point cloud into multiple clusters by identifying dense areas in the point cloud. Each cluster represents a potential obstacle. After clustering, a three-dimensional bounding box is generated for each cluster, as shown in the following example. Figure 4 As shown in Figure 2, the center position coordinates and 3D bounding box size of the DBSCAN bounding box are determined by the distribution of the point cloud in the cluster.

[0075] In step 2), for distant obstacles greater than or equal to 5 meters from the drone, the steps of enhancing the low-light image through the lightweight image enhancement network LPF and then inputting it into the lightweight FastestDet model for deep learning-assisted detection are as follows:

[0076] 2.1) The LPF-FastestDet detector acquires RGB images through an RGB-D camera and uses the LPF network to enhance low-light RGB images;

[0077] 2.2) The enhanced RGB image is input into the FastestDet model for object detection. FastestDet is a lightweight object detection algorithm that uses a 5×5 grouped convolutional parallel network structure to fuse features from different receptive fields and adaptively detect objects of different sizes using a single lightweight detection head.

[0078] 2.3) LPF-FastestDet uses a depth-wise separable convolutional module to simultaneously predict the classification and bounding box regression of the target.

[0079] like Figure 5 As shown in the figure, in the deep learning detector, a lightweight LPF network is used to enhance the low-light RGB image in real time to improve texture details. The LPF network extracts frequency domain features through a periodic residual module, and performs supervised training on low-light and normal light image pairs (such as the LOL dataset). The SSIM and L1 loss functions are used to optimize the network parameters. The enhanced image significantly restores the edge and texture information. The enhanced image is then input into the FastestDet model for deep learning target detection. The FastestDet model is based on the MobileNetV3 lightweight backbone network, uses 5×5 grouped convolution to extract multi-scale features in parallel, predicts target classification and bounding box regression through a depth-separable convolution module, and performs model quantization (INT8) and layer fusion when deployed on an ARM platform (such as NVIDIA JetsonNano), achieving real-time inference as low as 15ms.

[0080] Before fusing the IoU results of the three in step 3), the detection results of the non-learning detector are first integrated. If the obstacle is at a long distance, the learning-based detection module is activated. Subsequently, the results of the U-depth, DBSCAN, and LPF-FastestDet detectors are combined.

[0081]

[0082] in is the scale of the integrated detection bounding box (w, l, h); 、 、 They are the 3D bounding box scales from U-depth detector, DBSCAN detector, and LPF-FastestDet detector respectively; 、 、 are the predetermined weights of the three bounding boxes.

[0083] In this example, for nearby obstacles (<5m), the U-depth detector is assigned a weight of 0.6, the DBSCAN detector is assigned a weight of 0.4, and the LPF-FastestDet detector is assigned a weight of 0. For distant obstacles (>=5m), the three detectors are assigned weights of 0.3, 0.3, and 0.4, respectively. By calculating the intersection-over-union ratio of the bounding boxes of each detector (IOU ≥ 0.5 is considered to be the same target), the detection bounding box is obtained according to formula (3). In the ROS system, the dynamic obstacle detection results are published to the topic / obstacle_detection. The message type is a custom ObstacleArray.msg, which contains the obstacle position, size, and dynamic flag. The dynamic obstacle is circled with a blue cube and its motion trajectory is drawn using the RVIZ visualization tool, as shown in the following example: Figure 6 As shown, it provides real-time obstacle avoidance guidance for the UAV navigation system.

[0084] The specific steps for generating the set bounding box in step 3) are as follows:

[0085] 3.1) Obtain the 3D bounding boxes of the U-depth detector, DBSCAN detector, and lightweight object detection model LPF-FastestDet respectively;

[0086] 3.2) Calculate the intersection over union (IOU) for each pair of 3D bounding boxes. When the IOU exceeds a predetermined threshold, the detection result is confirmed to be the same obstacle.

[0087] 3.3) Fuse the detection boxes that meet the IOU condition, take the average value of the center position and the maximum value of the size of the 3D bounding boxes of the U-depth detector, DBSCAN detector, and LPF-FastestDet detector, and generate a collective 3D bounding box.

[0088] The specific steps of the feature association method in step 4) are as follows:

[0089] 4.1) Defining a feature vector for each obstacle, wherein the feature vector includes the obstacle's center position, size, and point cloud statistical features;

[0090] 4.2) Calculate the similarity score between the obstacle detected at the current moment and the obstacle detected at the previous moment;

[0091] 4.3) Associate the current obstacle with the obstacle with the highest similarity score at the previous moment.

[0092] The specific steps for identifying dynamic obstacles in step 5) are as follows:

[0093] 5.1) Calculate the center velocity based on the change in the obstacle center position between two consecutive frames, i.e., time t-1 and time t :

[0094] The three-dimensional coordinates of the obstacle center at time t are ( ),exist The three-dimensional coordinates of the moment are ( ),in: Respectively represent the obstacle center at Time along The coordinates of the axes, Respectively represent the obstacle center at Time along Axis coordinates, center speed By calculating the Euclidean distance between the obstacle center and the two frames Divide by the time interval Get, that is:

[0095]

[0096] in , is the time interval between two consecutive frames. Compare with the predetermined threshold to preliminarily filter out dynamic obstacles. In this example, the time interval between two frames is 1 / 30s, and the predetermined threshold is 0.6;

[0097] 5.2) For the dynamic obstacle candidate targets initially screened in step 5.1), calculate the displacement vector of each point on the obstacle surface by aligning the adjacent frame point clouds, and combine the time interval Calculate the point speed; vote based on the speed distribution of each point, if the speed exceeds the threshold The percentage of points is greater than the set ratio , it is confirmed to be a dynamic obstacle. In this case, the predetermined speed threshold 0.3m / s, overspeed point proportional threshold is 0.6;

[0098] 5.3) Output the 3D bounding box size of the dynamic obstacle (length ,Width ,high ) and its center position coordinates in the drone coordinate system ( ), used for drone navigation and obstacle avoidance.

Claims

1. A method for detecting and tracking dynamic obstacles in a low-light environment, characterized in that: The method comprises the following steps: 1) After acquiring the depth image and RGB image of the environment in front of the drone, an ensemble detection strategy is used to improve the accuracy and real-time performance of the drone's detection of dynamic obstacles; 2) For obstacles within 5 meters of the drone, two non-learning detectors, the U-depth detector and the DBSCAN detector, are used for rapid detection. For obstacles 5 meters or more away from the drone, the low-light image is enhanced using the lightweight image enhancement network (LPF) and then fed into the lightweight FastestDet model for deep learning-assisted detection. 3) By calculating the intersection-over-union (IoU) of the U-depth detector, the DBSCAN detector, and the LPF-FastestDet detector, and fusing the IoU results, a collective bounding box is generated to achieve high-precision, real-time obstacle detection in low-light environments. 4) Data association is performed on detected obstacles using feature association methods, and continuous tracking is performed using Kalman filtering; 5) Identify dynamic obstacles from all detected obstacles and output the 3D bounding box size of the dynamic obstacle, i.e. length Width high and its center position coordinates in the UAV coordinate system ( ).

2. The method for detecting and tracking dynamic obstacles in a low-light environment according to claim 1, wherein: In step 2), the U-depth detector converts the depth image into a U-depth map and locates the center coordinates and 3D bounding box size of the obstacle by combining depth continuity and linear grouping methods. The DBSCAN detector divides the point cloud into multiple clusters by analyzing the local density of the point cloud data and generates a 3D bounding box for the obstacle based on these clusters.

3. The method for detecting and tracking dynamic obstacles in a low-light environment according to claim 2, wherein: In the process of converting the depth image into the U-depth map by the U-depth detector in step 2), the coordinate transformation method cited in formula (1) is adopted, where and The horizontal position of the obstacle in the collective coordinate system; and are the length and width of the obstacle in the body coordinate system; is the focal length of the camera; , is the upper left corner position of the obstacle in the body coordinate system; , is the lower right corner position of the obstacle in the body coordinate system; The conversion of point clouds in the DBSCAN detector uses the same internal parameters as the RGB-D camera for inverse transformation. The fixed correspondence between the 3D camera point [x, y, z] and its image pixel coordinates [u, v, d] is shown in Equation (2), where fx and fy are the focal lengths of the camera on the x and y axes; cx and cy are the center of the camera aperture; and s represents the scaling factor of the depth map. Since the RGB image used for color images has been aligned in the process of converting the depth image to a point cloud by the DBSCAN detector, no separate coordinate conversion is required.

4. The method for detecting and tracking dynamic obstacles in a low-light environment according to claim 1, wherein: In step 2), for distant obstacles greater than or equal to 5 meters from the drone, the steps of enhancing the low-light image through the lightweight image enhancement network LPF and then inputting it into the lightweight FastestDet model for deep learning-assisted detection are as follows: 2.1) The LPF-FastestDet detector acquires RGB images through an RGB-D camera and uses the LPF network to enhance low-light RGB images; 2.2) The enhanced RGB image is input into the FastestDet model for object detection. FastestDet is a lightweight object detection algorithm that uses a 5×5 grouped convolutional parallel network structure to fuse features from different receptive fields and adaptively detect objects of different sizes using a single lightweight detection head. 2.3) LPF-FastestDet uses a depth-wise separable convolutional module to simultaneously predict the classification and bounding box regression of the target.

5. The method for detecting and tracking dynamic obstacles in a low-light environment according to claim 1, wherein: Before fusing the IoU results of the three in step 3), the detection results of the non-learning detector are first integrated. If the obstacle is at a long distance, the learning-based detection module is activated. Subsequently, the results of the U-depth, DBSCAN, and LPF-FastestDet detectors are combined. in is the scale of the integrated detection bounding box (w, l, h); 、 、 They are the 3D bounding box scales from U-depth detector, DBSCAN detector, and LPF-FastestDet detector respectively; 、 、 are the predetermined weights of the three bounding boxes.

6. The method for detecting and tracking dynamic obstacles in a low-light environment according to claim 1, wherein: The specific steps for generating the set bounding box in step 3) are as follows: 3.1) Obtain the 3D bounding boxes of the U-depth detector, DBSCAN detector, and lightweight object detection model FastestDet respectively; 3.2) Calculate the intersection over union (IOU) ratio for each pair of 3D bounding boxes. When the IOU exceeds a predetermined threshold, the detection result is confirmed to be the same obstacle. 3.3) Fuse the detection boxes that meet the IOU condition, take the average value of the center position and the maximum value of the size of the 3D bounding boxes of the U-depth detector, DBSCAN detector, and LPF-FastestDet detector, and generate a collective 3D bounding box.

7. The method for detecting and tracking dynamic obstacles in a low-light environment according to claim 1, wherein: The specific steps of the feature association method in step 4) are as follows: 4.1) Defining a feature vector for each obstacle, wherein the feature vector includes the obstacle's center position, size, and point cloud statistical features; 4.2) Calculate the similarity score between the obstacle detected at the current moment and the obstacle detected at the previous moment; 4.3) Associate the current obstacle with the obstacle with the highest similarity score at the previous moment.

8. The method for detecting and tracking dynamic obstacles in a low-light environment according to claim 1, wherein: The specific steps for identifying dynamic obstacles in step 5) are as follows: 5.1) Calculate the center velocity based on the change in the obstacle center position between two consecutive frames, i.e., time t-1 and time t : The three-dimensional coordinates of the obstacle center at time t are ( ),exist The three-dimensional coordinates of the moment are ( ),in: Respectively represent the obstacle center at Time along The coordinates of the axes, Respectively represent the obstacle center at Time along Axis coordinates, center speed By calculating the Euclidean distance between the obstacle center and the two frames Divide by the time interval Get, that is: in , The time interval between two consecutive frames is the calculated center speed. Compare with the predetermined threshold to preliminarily screen out dynamic obstacles; 5.2) For the dynamic obstacle candidate targets initially screened in step 5.1), calculate the displacement vector of each point on the obstacle surface by aligning the adjacent frame point clouds, and combine the time interval Calculate the point speed; vote based on the speed distribution of each point, if the speed exceeds the threshold The percentage of points is greater than the set ratio , it is confirmed as a dynamic obstacle; 5.3) Output the 3D bounding box size of the dynamic obstacle, i.e. length ,Width ,high and its center position coordinates in the UAV coordinate system ( ), used for drone navigation and obstacle avoidance.

Images