A method for obstacle detection for highway inspection of unmanned aerial vehicles

Abstract

The application relates to the technical field of unmanned aerial vehicle obstacle avoidance, in particular to an obstacle detection method for highway inspection of an unmanned aerial vehicle, which comprises the following steps: acquiring a plurality of laser detection point sets at different moments; determining a plurality of associated point pairs between adjacent moments based on image coordinates of each laser detection point, local motion consistency deviation and motion source spatial discreteness; respectively performing spatial clustering on the laser detection points at each moment in all associated point pairs to obtain corresponding tracking point clusters at each moment; and performing obstacle verification on the tracking point cluster corresponding to the current moment based on the tracking point clusters corresponding to the plurality of moments, and determining the tracking point cluster that passes the verification as an obstacle. The method can improve the detection accuracy and reliability of small static obstacles in extreme scenes such as dramatic changes in light.

Description

Technical Field

[0001] This application relates to the field of obstacle avoidance technology for unmanned aerial vehicles (UAVs), specifically to an obstacle detection method for UAV highway inspection. Background Technology

[0002] In related technologies, drones can autonomously detect and avoid obstacles when performing highway inspections using visual sensors and lidar.

[0003] However, in scenarios such as tunnels and bridges, drones experience drastic changes in lighting as they rapidly transition from well-lit areas to shadowed areas. At this time, due to uneven lighting on the obstacle surface, the overall image contrast is low, the visual signal is weak, and the visual sensor has difficulty in consistently extracting and utilizing effective feature information. Meanwhile, when radar detects distant or small obstacles, the returned point cloud data is very sparse. This sparse point cloud data is easily identified as random noise and filtered out. Under this dual failure, obstacles may be missed or detected inaccurately. Summary of the Invention

[0004] To address the technical problems of missed obstacle detection and inaccurate obstacle detection during drone inspection, this application aims to provide an obstacle detection method for drone highway inspection. The specific technical solution adopted is as follows:

[0005] This application provides an obstacle detection method for unmanned aerial vehicle (UAV) highway inspection. The method includes: acquiring a set of laser detection points at multiple times, including the current time; determining multiple associated point pairs between adjacent times based on the image coordinates, local motion consistency deviation, and motion source spatial dispersion of each laser detection point; the local motion consistency deviation characterizes the degree of consistency between the actual motion of the local image region where the laser detection point is located and the assumed motion of a stationary rigid body; the motion source spatial dispersion characterizes the degree of dispersion of the motion source of the laser detection point and the motion sources of other laser detection points in its image neighborhood in three-dimensional space; performing spatial clustering on the laser detection points at each time in all associated point pairs to obtain a tracking point cluster corresponding to each time; and verifying the tracking point cluster corresponding to the current time as an obstacle based on the tracking point clusters corresponding to multiple times, and determining the verified tracking point cluster as an obstacle.

[0006] Optionally, the method further includes: determining multiple feature points within the image block corresponding to the first laser detection point, wherein the first laser detection point is any laser detection point in the set of laser detection points at the current moment, and the current moment is a moment included in multiple moments; determining the theoretical motion vector of each feature point based on the motion information of the UAV between the previous moment and the current moment; determining the actual motion vector of each feature point from the previous moment to the current moment based on an optical flow estimation algorithm; and determining the local motion consistency deviation of the first laser detection point based on the theoretical motion vector and the actual motion vector of all feature points.

[0007] Optionally, determining multiple associated point pairs between adjacent time points based on the image coordinates, local motion consistency deviation, and motion source spatial dispersion of each laser detection point includes: determining the initial association cost between the point pairs at the current time and the previous time point based on the image coordinates and local motion consistency deviation of the laser detection points at the current time and the previous time point; adjusting the initial association cost of the point pairs corresponding to each laser detection point at the current time based on the motion source spatial dispersion of each laser detection point at the current time to obtain the final association cost between the point pairs at the current time and the previous time point; and determining the associated point pairs between the current time and the previous time point based on the final association cost.

[0008] Optionally, determining the initial association cost between the point pairs at the current and previous times based on the image coordinates and local motion consistency deviation of the laser detection points at the current and previous times includes: determining the image position prediction deviation between the first and second laser detection points based on the image coordinates of the first and second laser detection points, where the second laser detection point is any laser detection point at the previous time; determining the motion feature change between the first and second laser detection points based on the local motion consistency deviation of the first and second laser detection points; and performing a weighted summation of the image position prediction deviation and the motion feature change to obtain the initial association cost between the first and second laser detection points.

[0009] Optionally, the method further includes: determining M neighboring laser detection points of the first laser detection point on the current camera image, where M is an integer greater than 1, and the current camera image is the camera image at the current moment; determining candidate associated points of the first laser detection point and the candidate associated points of the M neighboring laser detection points based on the initial association cost between the laser detection points at the current moment and the previous moment; determining the three-dimensional spatial centroid of the set formed by the candidate associated points of the M neighboring laser detection points; and determining the spatial dispersion of the motion source of the first laser detection point based on the Euclidean distance between the candidate associated points of the first laser detection point and the three-dimensional spatial centroid.

[0010] Optionally, the method of performing obstacle verification on the tracking point cluster corresponding to the current moment based on the tracking point clusters corresponding to multiple moments, and determining the verified tracking point clusters as obstacles, includes: determining the continuous tracking point clusters of the first tracking point cluster based on the tracking point clusters corresponding to multiple moments, wherein the first tracking point cluster is any tracking point cluster corresponding to the current moment; performing obstacle verification on the first tracking point cluster based on the number of laser detection points in the first tracking point cluster, the spatial dispersion of the motion source of all laser detection points in the first tracking point cluster, and the number of continuous tracking point clusters of the first tracking point cluster; and determining the first tracking point cluster as an obstacle if the verification is successful.

[0011] Optionally, determining the continuous tracking point cluster of the first tracking point cluster based on the tracking point clusters corresponding to multiple time points includes: determining the Euclidean distance between the three-dimensional spatial centroid of the first tracking point cluster and the three-dimensional spatial centroid of the second tracking point cluster, wherein the second tracking point cluster is any tracking point cluster corresponding to the previous time point; and if the Euclidean distance is less than a preset distance threshold, determining the second tracking point cluster as the continuous tracking point cluster of the first tracking point cluster.

[0012] Optionally, the method further includes: determining that the first tracking point cluster has passed verification if the number of laser detection points in the first tracking point cluster is greater than a threshold for the number of laser detection points, the mean of the spatial dispersion of the motion source is less than a threshold for the spatial dispersion of the motion source, and the number of consecutive tracking frames is greater than a threshold for the number of consecutive tracking frames.

[0013] Optionally, acquiring a set of laser detection points at multiple times includes: acquiring a camera image and multiple three-dimensional spatial points acquired by radar at the current time; projecting the multiple three-dimensional spatial points onto the pixel coordinate system of the camera image at the current time to generate a set of laser detection points at the current time.

[0014] Optionally, the above-described method of projecting multiple three-dimensional spatial points onto the pixel coordinate system of the camera image at the current moment to generate the set of laser detection points at the current moment includes: projecting multiple three-dimensional spatial points onto the pixel coordinate system of the camera image at the current moment to generate candidate laser detection points at the current moment; determining the local average gradient energy of the image patch corresponding to the first candidate laser detection point, wherein the first candidate laser detection point is any candidate laser detection point at the current moment; determining the global average gradient energy of the camera image at the current moment; determining the texture saliency of the first candidate laser detection point based on the ratio between the local average gradient energy and the global average gradient energy; and adding the first candidate laser detection point to the set of laser detection points at the current moment if the texture saliency of the first candidate laser detection point is greater than a texture saliency threshold.

[0015] This application has the following beneficial effects:

[0016] In this embodiment, laser detection points with motion patterns conforming to the laws of static rigid bodies and high grouping are selected by considering local motion consistency deviation, spatial dispersion of motion sources, and image coordinates. By associating laser detection points at multiple times, continuously existing laser detection points are obtained, effectively identifying and suppressing random noise points that do not conform to the motion trend of the neighborhood. Finally, through spatial clustering, based on tracking point clusters at multiple times, obstacle verification is performed from the dimension of tracking point clusters. Tracking point clusters that are spatially clustered and temporally continuous are identified as obstacles, improving the accuracy and reliability of detecting small static obstacles in extreme scenarios such as drastic changes in lighting. Attached Figure Description

[0017] To more clearly illustrate the technical solutions and advantages in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A structural diagram of an obstacle detection system for unmanned aerial vehicle (UAV) highway inspection provided in one embodiment of this application;

[0019] Figure 2 A flowchart illustrating an obstacle detection method for unmanned aerial vehicle (UAV) highway inspection, provided as an embodiment of this application;

[0020] Figure 3 A flowchart illustrating another obstacle detection method for unmanned aerial vehicle (UAV) highway inspection, provided as an embodiment of this application;

[0021] Figure 4 This is a flowchart illustrating another obstacle detection method for unmanned aerial vehicle (UAV) highway inspection, provided as an embodiment of this application. Detailed Implementation

[0022] To further illustrate the technical means and effects adopted by this application to achieve the intended inventive objective, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of an obstacle detection method for unmanned aerial vehicle (UAV) highway inspection proposed according to this application. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0024] Drones face unique obstacle detection challenges when performing highway inspections, particularly in structured road sections such as tunnels and under bridges. Drones typically fly along predetermined routes, and autonomously detecting and avoiding obstacles is fundamental to their safe operation. However, in certain highway inspection scenarios, there exists an extreme but high-risk situation: when a drone approaches a tunnel entrance or bridge shadow area at high speed from a well-lit area, it enters an area with drastically changing lighting. Within this area, detecting small, stationary obstacles that pose a safety threat to the drone, such as tire debris or stones left on the road surface, becomes extremely difficult.

[0025] Existing detection methods primarily rely on visual sensors and LiDAR. Deep learning-based visual detection methods, in areas of dramatic lighting changes, suffer from uneven illumination on the target object's surface, resulting in low overall image contrast and difficulty in extracting effective features. This leads to decreased confidence in the detection results and a high risk of missed detections. While radar, as an active sensor, is unaffected by ambient lighting, its inherent principle dictates that when detecting distant or small obstacles, the returned point cloud data is extremely sparse, typically forming only a few discrete points. These sparse points, due to their limited number, cannot form effective point cloud clusters using conventional clustering algorithms, such as density-based spatial clustering of applications with noise (DBSCAN). They are often filtered out as random noise, again leading to missed detections. The current challenge lies in the fact that when a small, stationary obstacle happens to be located in the aforementioned area of ​​dramatic lighting changes, both weak visual signals and sparse radar signals can occur simultaneously. In this situation, neither single-sensor solutions nor conventional fusion methods relying on at least one sensor to provide definitive results can achieve stable detection.

[0026] The following description, in conjunction with the accompanying drawings, details a specific scheme for an obstacle detection method for unmanned aerial vehicle (UAV) highway inspection provided in this application.

[0027] Please see Figure 1 The diagram shows a structural diagram of an obstacle detection system for unmanned aerial vehicle (UAV) highway inspection according to an embodiment of this application.

[0028] like Figure 1 As shown, the obstacle detection system 10 for UAV highway inspection includes an instantaneous feature extraction module 101, a neighborhood consistency association module 102, and a sequential verification and confirmation module 103.

[0029] The instantaneous feature extraction module 101 is used to extract structured data units with clear features (such as laser detection points, texture saliency, and local motion consistency deviation) from the raw, unstructured data of the sensor.

[0030] The neighborhood consistency association module 102 is used to construct an association cost matrix based on the complete set of laser detection point features and establish the association relationship between laser detection points at adjacent time points.

[0031] The sequential verification module 103 is used to cluster discrete laser detection points into tracking point clusters, and to verify the obstacles of the tracking point clusters from three dimensions: spatial scale, temporal stability and motion consistency.

[0032] Please see Figure 2 The diagram illustrates a method flowchart for obstacle detection in unmanned aerial vehicle (UAV) highway inspection, according to an embodiment of this application.

[0033] like Figure 2 As shown, the obstacle detection method for UAV highway inspection includes S201-S204.

[0034] S201. Obtain a set of laser detection points at multiple times.

[0035] Among these multiple moments, the current moment is included.

[0036] It should be understood that these multiple moments include the current moment and the historical moment of the current moment, with the historical moment being at least one moment.

[0037] It should be understood that a set of laser detection points includes multiple laser detection points. A laser detection point can be understood as a data unit, which is the projection of the detected three-dimensional spatial point onto a two-dimensional image. The laser detection point corresponds to a three-dimensional spatial point coordinate and a two-dimensional image point coordinate.

[0038] Alternatively, the two-dimensional image can be a camera image captured by a drone.

[0039] It is understood that the obstacle detection method provided in this application embodiment is performed in real time. In one optional implementation, if obstacle detection has been performed in the previous moment, the set of laser detection points at the previous moment and the obstacle detection results at the previous moment (including local motion consistency deviation, continuous tracking point clusters, etc.) can be directly obtained, and then the set of laser detection points at the current moment can be generated by the sensor data at the current moment.

[0040] In one alternative implementation, the camera image at the current moment and multiple three-dimensional spatial points acquired by the radar can be obtained; the multiple three-dimensional spatial points are projected onto the pixel coordinate system of the camera image at the current moment to generate a set of laser detection points at the current moment.

[0041] It should be understood that the camera images are from the cameras configured on the drone, including images from the left and right cameras.

[0042] Understandably, radar is used to collect point cloud data, which includes multiple three-dimensional spatial points.

[0043] In one alternative implementation, the three-dimensional spatial points can be transformed from the lidar coordinate system to the camera coordinate system using pre-calibrated camera extrinsic and intrinsic parameter matrices, and then projected onto the pixel coordinate system of the left camera image to calculate the corresponding two-dimensional image point coordinates, which are then identified as the laser detection point.

[0044] It should be noted that the camera extrinsic parameter matrix is ​​used to describe the pose relationship between the camera and the radar. The coordinate system transformation is a perspective projection transformation known in the art, which will not be elaborated here.

[0045] In another alternative implementation, the coordinates of the converted two-dimensional image points can be simply filtered out to remove coordinates that are obviously noise or suspended particles in the air.

[0046] Specifically, multiple three-dimensional spatial points can be projected onto the pixel coordinate system of the camera image at the current moment to generate candidate laser detection points at the current moment.

[0047] It should be understood that the candidate laser detection point is the two-dimensional image point corresponding to all three-dimensional spatial points.

[0048] For example, taking the first candidate laser detection point as an example, the first candidate laser detection point is any candidate laser detection point at the current time. First, the local average gradient energy of the image patch corresponding to the first candidate laser detection point is determined; at the same time, the global average gradient energy of the camera image at the current time is determined; then, based on the ratio between the local average gradient energy and the global average gradient energy, the texture saliency of the first candidate laser detection point is determined; if the texture saliency of the first candidate laser detection point is greater than the texture saliency threshold, the first candidate laser detection point is added to the laser detection point set at the current time.

[0049] It should be noted that the image patch corresponding to a candidate laser detection point is an image patch of a preset size extracted from the left camera image with the candidate laser detection point as the center. The local average gradient energy is used to characterize the degree of gray-level change within the image patch. The larger the value, the more drastic the gray-level change and the richer the texture.

[0050] This preset size can be referenced to the expected imaging size of the candidate laser detection point, such as a tire fragment.

[0051] For example, the preset size can be 32×32 pixels.

[0052] It should be understood that the global average gradient energy is used to characterize the average texture level of the entire image.

[0053] Optionally, the grayscale value of each pixel in the camera image can be obtained, and then the grayscale gradient of each pixel can be calculated using the Sobel operator. Then, the square of the magnitude of the grayscale gradient can be determined, and the mean of the squares of the gradient magnitudes of all pixels in the image block corresponding to the first candidate laser detection point can be determined as the local average gradient energy. The mean of the squares of the gradient magnitudes of all pixels in the camera image at the current moment can be determined as the global average gradient energy.

[0054] It should be understood that texture saliency is used to characterize the saliency of local texture relative to global texture. When texture saliency is high, it indicates that the local area (image patch) to which the first candidate laser detection point is attached has a texture richness much higher than the average level of the entire scene. The candidate laser detection point is very likely attached to the surface of an independent object with significant texture features (such as an obstacle), rather than a uniform sky, road surface, or wall. Conversely, it may be noise.

[0055] Optionally, when the global average gradient energy is not zero, the ratio between the local average gradient energy and the global average gradient energy can be determined as the texture saliency of the first candidate laser detection point; when the global average gradient energy is zero, a smoothing coefficient can be set, the sum of the global average gradient energy and the smoothing coefficient can be determined as the denominator, the local average gradient energy can be determined as the numerator, and the ratio of the numerator to the denominator can be determined as the texture saliency of the first candidate laser detection point.

[0056] For example, the smoothing coefficient can be 10. -6 The texture saliency threshold can be 0.

[0057] Understandably, filtering the initially generated candidate laser detection points based on texture saliency can remove some obvious noise points before data analysis, improve data quality, and reduce subsequent computational burden and interference.

[0058] S202. Based on the image coordinates of each laser detection point, the local motion consistency deviation, and the spatial dispersion of the motion source, determine multiple pairs of associated points between adjacent time points.

[0059] Among them, the local motion consistency deviation characterizes the degree of consistency between the actual motion of the local image region where the laser detection point is located and the assumed motion of the stationary rigid body, and the motion source spatial dispersion is used to characterize the degree of dispersion of the motion source of the laser detection point and the motion source of other laser detection points in the image neighborhood in three-dimensional space.

[0060] In this embodiment of the application, a pair of related points between adjacent time points are the observation results of the same real point at different time points.

[0061] It should be understood that the image coordinates of the laser detection point can provide positional information. Based on the coordinates of the laser detection point, laser detection points with small positional deviations from the current laser detection point at adjacent times can be selected. Based on the local motion consistency deviation, laser detection points that conform to rigid body motion can be selected. Based on the local motion consistency deviation and the coordinates, two laser detection points with high individual similarity can be identified.

[0062] It is understandable that the motion source of a laser detection point is actually the position of the corresponding laser detection point at the previous moment. Since the motion sources of neighboring points on the same rigid body are necessarily neighboring in physical space, when the motion source of a laser detection point is significantly dispersed from the motion sources of other laser detection points, it indicates that the point may be noise. Based on the spatial dispersion of motion sources, laser detection points that are incorrectly associated due to accidental motion similarity can be removed, i.e., laser detection points with high group consistency.

[0063] Therefore, based on the image coordinates of the laser detection points, the local motion consistency deviation, and the spatial dispersion of the motion source, a pair of related points with high individual similarity and high group consistency can be accurately identified.

[0064] In one alternative implementation, before executing S202, the local motion consistency deviation of each laser detection point can be determined first.

[0065] Optionally, multiple feature points within the image block corresponding to the first laser detection point can be determined first; the theoretical motion vector of each feature point can be determined based on the motion information of the UAV between the previous and current time; the actual motion vector of each feature point from the previous to the current time can be determined based on the optical flow estimation algorithm; and the local motion consistency deviation of the first laser detection point can be determined based on the theoretical motion vector and the actual motion vector of all feature points.

[0066] The first laser detection point is any laser detection point in the set of laser detection points at the current moment, and the current moment is a moment included in multiple moments.

[0067] Optionally, K stable pixel-level feature points can be identified from the image block corresponding to the first laser detection point based on a feature detection algorithm (such as the Shi-Tomasi corner detection algorithm), where K is a preset positive integer.

[0068] For example, K can be 5.

[0069] Optionally, for each feature point, its three-dimensional coordinates can be recovered, and then its theoretical motion vector can be calculated based on its three-dimensional coordinates. Specifically, pixel matching can be performed on the left and right camera images at the current moment (e.g., matching based on the SAD algorithm), its disparity can be calculated, and then, based on the known focal length and baseline length of the stereo camera, the three-dimensional depth (i.e., z-axis coordinate) of the feature point in the camera coordinate system can be calculated. Combining the camera intrinsic parameter matrix, the complete three-dimensional coordinates of the feature point can be recovered using its pixel coordinates (i.e., x-axis and y-axis coordinates) and its depth.

[0070] Alternatively, pixel matching can be performed based on the image stereo matching primary block matching algorithm (sum of absolute differences, SAD).

[0071] Optionally, the 3D depth of a feature point satisfies the following formula:

[0072]

[0073] in, Representing feature points The three-dimensional depth Indicates the focal length of a binocular camera. Indicates the baseline length of the stereo camera. Representing feature points Parallax.

[0074] It should be understood that this theoretical motion vector is used to characterize the changes in the pixel coordinates of feature points caused by the movement of the UAV itself.

[0075] Optionally, the motion information of the UAV includes the linear velocity and angular velocity in the UAV body coordinate system. Based on the linear velocity and angular velocity and the three-dimensional coordinates of each feature point, the theoretical motion vector of the feature point on the image plane from the previous moment to the current moment is calculated using the rigid body kinematics equations under the assumption of "scene static".

[0076] It should be noted that the method for determining the theoretical motion vector is a well-known technique in the field and will not be elaborated here.

[0077] It should be understood that the actual motion vector is used to characterize the changes in the actual pixel coordinates of the feature point.

[0078] Alternatively, an optical flow estimation algorithm (e.g., Lucas-Kanade optical flow) can be used to track the pixel displacement of these K feature points from the left camera image at the previous time step to the left camera image at the current time step, and the displacement difference can be determined as their actual motion vector.

[0079] It should be understood that, ideally, for a point on a stationary rigid body, its actual motion vector should be approximately equal to its theoretical motion vector. Considering factors such as noise, assuming a local linear transformation relationship exists between them, a 2×2 transformation matrix is ​​obtained by using K pairs of motion vectors within the image patch (each pair consisting of the theoretical and actual motion vectors of a feature point) through linear least squares. This matrix is ​​the deviation matrix used to characterize the local motion consistency deviation of the first laser detection point. Theoretically, the deviation matrix of an ideal stationary rigid body should be close to the identity matrix; therefore, the degree to which this deviation matrix deviates from the identity matrix can characterize the degree of deviation in the local motion of the first laser detection point.

[0080] Alternatively, the change in motion characteristics in scalar form can be obtained by calculating the Frobenius norm of the difference between the two local motion consistency deviation matrices. The larger the norm value, the greater the difference in local motion patterns between the two points.

[0081] It should be understood that the greater the deviation in the consistency of local motion, the further the deviation is from the identity matrix, and the less consistent the motion of the feature point is with the assumption of a stationary rigid body.

[0082] In this embodiment, since the motion patterns of points generated by non-rigid phenomena (such as swaying light and shadow, floating debris) or sensor noise usually do not follow the geometry of rigid body motion, by comparing the theoretical motion vector calculated by the UAV itself with the actual motion vector observed by optical flow, the motion of the area where a laser detection point is attached can be quantified to determine whether it conforms to the expectation of a stationary rigid body, thus effectively distinguishing points on real stationary obstacles.

[0083] S203. Perform spatial clustering on the laser detection points at each time step in all associated point pairs to obtain the tracking point clusters corresponding to each time step.

[0084] It should be understood that a moment can correspond to one or more tracking point clusters. A tracking point cluster is a set of multiple laser detection points that exhibit high consistency in space and time, which may be multiple points observed on a real object.

[0085] Optionally, spatial clustering can be performed on laser detection points at the same time based on the three-dimensional coordinates of each laser detection point.

[0086] Optionally, spatial clustering can be performed based on the DBSCAN algorithm, in which the neighborhood radius can be set based on the expected maximum physical distance between any two points on the expected image (such as a tire fragment) of the laser detection point, for example, 0.2 meters; and the minimum number of neighborhood points of the core point can be set based on the number of radar detection points required to constitute a minimum identifiable obstacle, for example, 3 points.

[0087] It should be understood that the DBSCAN algorithm can aggregate points that are density-reachable into clusters and mark points that do not belong to any cluster as noise.

[0088] S204. Based on the tracking point clusters corresponding to multiple time points, perform obstacle verification on the tracking point cluster corresponding to the current time point, and determine the tracking point cluster that passes the verification as an obstacle.

[0089] In one alternative implementation, tracking point clusters at multiple time points can be associated. If a tracking point cluster exists continuously at consecutive time points, it can be identified as an obstacle.

[0090] For example, if there are two times, the number of consecutive times can be two. When there are three times, the number of consecutive times can be set to three.

[0091] In one alternative implementation, after an obstacle is identified, information about the obstacle, including its three-dimensional position (e.g., obtained by calculating the centroid of the cluster), can be output for the UAV's path planning system to avoid.

[0092] In this embodiment, laser detection points with motion patterns conforming to the laws of static rigid bodies and high grouping are selected by considering local motion consistency deviation, spatial dispersion of motion sources, and image coordinates. By associating laser detection points at multiple times, continuously existing laser detection points are obtained, effectively identifying and suppressing random noise points that do not conform to the motion trend of the neighborhood. Finally, through spatial clustering, based on tracking point clusters at multiple times, obstacle verification is performed from the dimension of tracking point clusters. Tracking point clusters that are spatially clustered and temporally continuous are identified as obstacles, improving the accuracy and reliability of detecting small static obstacles in extreme scenarios such as drastic changes in lighting.

[0093] Combination Figure 2 ,like Figure 3 As shown, in one implementation of this application embodiment, the determination of multiple associated point pairs between adjacent time points based on the image coordinates of each laser detection point, the local motion consistency deviation, and the spatial dispersion of the motion source can specifically include S301-S303.

[0094] S301. Based on the consistency deviation of the laser detection point image coordinates and local motion between the current time and the previous time, determine the initial association cost between the point pairs between the current time and the previous time.

[0095] A point pair includes a laser detection point at the current moment and a laser detection point at the previous moment.

[0096] It should be understood that image coordinates and local motion consistency deviation are data of different dimensions, and there is a difference in the units of measurement between the two, so they cannot be directly compared. Therefore, the embodiments of this application measure the association cost between two laser detection points based on a cost matrix that can fuse multi-dimensional information. An initial cost matrix is ​​constructed based on image coordinates and local motion consistency deviation to measure the initial association cost; a final cost matrix is ​​constructed based on image coordinates, local motion consistency deviation, and spatial dispersion of motion source to measure the final association cost.

[0097] In one alternative implementation, assuming the number of laser detection points at the current moment is X and the number of laser detection points at the previous moment is Y, the size of the cost matrix is ​​X×Y. Each element in the cost matrix represents the initial cost or distance that associates the laser detection points at the previous moment with the laser detection points at the current moment; the smaller the value of the element, the higher the probability of a match.

[0098] In one optional implementation, the image position prediction deviation between the first laser detection point and the second laser detection point can be determined based on the image coordinates of the first laser detection point and the image coordinates of the second laser detection point; the change in motion characteristics between the first laser detection point and the second laser detection point can be determined based on the local motion consistency deviation of the first laser detection point and the local motion consistency deviation of the second laser detection point; and the image position prediction deviation and the change in motion characteristics can be weighted and summed to obtain the initial association cost between the first laser detection point and the second laser detection point.

[0099] The second laser detection point is any laser detection point from the previous moment.

[0100] It should be understood that this position prediction bias is used to characterize the degree of agreement between the theoretical trajectory of a laser detection point from the previous moment to the current moment and the actual observed position.

[0101] First, based on the three-dimensional coordinates of the second laser detection point and the UAV's own motion (linear velocity and angular velocity), the theoretical three-dimensional coordinates of its position at the current moment are determined. Using the camera intrinsic parameter matrix, these theoretical three-dimensional coordinates are projected onto the image plane at the current moment to obtain the two-dimensional image coordinates of its theoretical position. The Euclidean distance between these theoretical two-dimensional coordinates and the first laser detection point is determined. This Euclidean distance is used as the original image position prediction deviation, and the original image position prediction deviation is normalized to obtain the image position prediction deviation between the first and second laser detection points.

[0102] Optionally, the image position prediction deviation between two laser detection points satisfies the following formula:

[0103]

[0104] in, express laser detection point at time and laser detection point at time Image location prediction deviation between express laser detection point at time Two-dimensional coordinates in the pixel coordinate system of the left camera image express laser detection point at time exist Two-dimensional coordinates of the theoretical position at time [time]. The L2 norm (e.g., the Euclidean norm) is used to calculate Euclidean distance. express and The Euclidean distance between them, i.e., the original image location prediction bias. This represents a preset normalization parameter used to perform a bounded transformation on the original image location prediction deviation, compressing the original image location prediction deviation to the range [0, 1). The value of determines the inflection point of the normalization curve; when the deviation value equals When the normalization result is 0.5, its value is related to the range of pixel prediction error allowed by the system.

[0105] For example, if a 5-pixel error is allowed, then It can be set to 5.

[0106] It should be understood that normalization can eliminate the dimension of pixel distance and map the image position prediction bias to the interval [0, 1).

[0107] Based on the description of the above embodiments, it should be understood that the local motion consistency deviation is actually a local motion consistency deviation matrix, and the change in motion feature can be determined based on the difference between the two local motion consistency deviation matrices.

[0108] Specifically, the Frobenius norm of the difference between the two local motion consistency deviation matrices can be determined and used as the original motion feature change. Then, the original motion feature change is normalized using the same normalization method as the image position prediction deviation to obtain the motion feature change.

[0109] Optionally, the change in motion characteristics between the two laser detection points satisfies the following formula:

[0110]

[0111] in, express laser detection point at time and laser detection point at time The change in motion characteristics between them and They represent laser detection point at time and laser detection point at time The corresponding local motion consistency deviation matrix, The Frobenius norm of a matrix is ​​calculated as the square root of the sum of the squares of all its elements. express and The Frobenius norm between, which characterizes the change in the original motion characteristics. This represents the preset normalization parameter, which is used to perform a bounded transformation on the original motion feature changes, compressing the original motion feature changes to the range of [0, 1).

[0112] Optionally, the initial association cost between two laser detection points satisfies the following formula:

[0113]

[0114] in, express laser detection point at time and laser detection point at time The initial association cost between them express laser detection point at time and laser detection point at time Image location prediction deviation between express laser detection point at time and laser detection point at time The change in motion characteristics between them and This represents the preset weighting coefficients used to balance the importance of image location prediction bias and motion feature variation in the cost matrix, and their sum is 1.

[0115] Optionally, adjustments can be made based on the actual situation. and The value of .

[0116] For example, the and The value can be 0.5.

[0117] It should be understood that the greater the initial association cost between two laser detection points, the greater the cost of associating the two laser detection points.

[0118] The method described above for determining the initial association cost ensures the continuity of the associated points on the physical motion trajectory through image position deviation and guarantees the stability of the associated points in the motion pattern through the change in motion features. It overcomes the limitations of relying solely on position information, which is prone to failure during drastic motion or appearance changes, or relying solely on motion features, which is unstable in areas with missing textures, making the initial association cost more robust.

[0119] S302. Based on the spatial dispersion of the motion source of each laser detection point at the current moment, adjust the initial association cost of the point pair corresponding to each laser detection point at the current moment to obtain the final association cost between the point pair at the current moment and the previous moment.

[0120] In one optional implementation, M neighboring laser detection points of the first laser detection point on the current camera image can be determined. Based on the initial association cost between the laser detection points at the current time and the previous time, candidate associated points of the first laser detection point and the candidate associated points of the M neighboring laser detection points are determined. The three-dimensional spatial centroid of the set formed by the candidate associated points of the M neighboring laser detection points is determined. Based on the Euclidean distance between the candidate associated points of the first laser detection point and the three-dimensional spatial centroid, the spatial dispersion of the motion source of the first laser detection point is determined.

[0121] Where M is an integer greater than 1, and the current camera image is the camera image at the current moment, specifically the left camera image at the current moment.

[0122] First, in the pixel coordinate system of the current camera image, taking the image coordinates of the first laser detection point as the center, based on the nearest neighbor search algorithm (e.g., kd-tree algorithm), find the M laser detection points that are closest to it in Euclidean distance, and determine these M laser detection points as the neighboring laser detection points of the first laser detection point.

[0123] For example, the value of M can be 5.

[0124] Secondly, the laser detection point with the lowest initial association cost with the first laser detection point can be determined as the candidate association point of the first laser detection point. Similarly, the candidate association points of each neighboring laser detection point can be determined in turn.

[0125] It is understandable that, based on the candidate association points determined by this method, there may be multiple laser detection points corresponding to the same candidate association point. Since the candidate association points determined in this case are used to determine the spatial dispersion of the motion source, rather than the final association points, it is only necessary to determine the optimal choice for a single laser detection point.

[0126] Alternatively, candidate associated points of each laser detection point at the current time can be determined based on a globally optimal algorithm (such as the Hungarian algorithm) from the previous time step.

[0127] Next, determine the three-dimensional coordinates of the candidate associated points of M neighboring laser detection points, and then determine the three-dimensional centroid of these three-dimensional coordinate points.

[0128] It should be understood that the centroid of this three-dimensional space represents the center of the motion source of the neighboring laser detection point at the previous moment.

[0129] Optionally, the centroid in three-dimensional space satisfies the following formula:

[0130]

[0131] in, Indicates laser detection point The nearest laser detection point in The three-dimensional centroids between candidate correlation points at time t, where the laser detection point... for Laser detection point at any moment Indicates laser detection point The number of neighboring laser detection points, Indicates laser detection point Nearby laser detection points exist The three-dimensional coordinates of the candidate correlation points at time 1. Indicates the laser detection point All neighboring laser detection points in Sum the three-dimensional coordinates of the candidate associated points at time t.

[0132] Finally, the Euclidean distance between the three-dimensional coordinates of the candidate associated points of the first laser detection point and the centroid of that three-dimensional space is determined. This distance is then defined as the spatial dispersion of the motion source of the first laser detection point.

[0133] Optionally, the spatial discreteness of the motion source of a laser detection point satisfies the following formula:

[0134]

[0135] in, Indicates laser detection point The motion originates from the spatial discreteness. Indicates laser detection point In The three-dimensional coordinates of the candidate correlation points at time 1. Indicates laser detection point The nearest laser detection point in The three-dimensional centroids between candidate correlation points at time t. The L2 norm (Euclidean norm) operator is used to calculate Euclidean distance. express and The Euclidean distance between them.

[0136] It is understandable that by evaluating the three-dimensional centroid of candidate associated points of a laser detection point in three-dimensional space, and the distance between the three-dimensional coordinates of the candidate associated points and the three-dimensional centroid, the clustering degree between the laser detection point and the candidate motion source of the laser detection point can be accurately reflected.

[0137] After obtaining the spatial dispersion of motion source for each laser detection point, the initial correlation cost can be adjusted based on this spatial dispersion, i.e., the initial correlation cost is penalized to obtain the final correlation cost.

[0138] Optionally, the final association cost between the two laser detection points satisfies the following formula:

[0139]

[0140] in, express laser detection point at time and laser detection point at time The ultimate cost of association between them express laser detection point at time and laser detection point at time The initial association cost between them Indicates laser detection point The motion originates from the spatial discreteness. This is a preset weighting coefficient used to control the strength of the spatial dispersion penalty term. This represents the preset positive adjustment parameter, in meters (m), used to control the shape of the normalization function, and... The value is mapped to the interval [0, 1).

[0141] For example, the value can be 0.5.

[0142] Based on this formula, it should be understood that The larger the value, the more likely it is to be detected by the laser. This indicates that the more "uncooperative" it is, the less consistent its motion pattern is with the overall motion pattern of other points in its image neighborhood. Regardless of the laser detection point... Matching with any point from the previous time step will incur additional costs. Weighting coefficients. This determines the strength of the penalty. A larger penalty... This significantly increases the association cost of disjoint points, making the system more inclined to associate laser detection points with consistent motion patterns; smaller This makes the system more dependent on the initial association cost.

[0143] This cost adjustment mechanism ensures that even locally optimal matches (i.e., (small value) but abnormal movement pattern (i.e.) The point with the largest value has the final associated cost. This will significantly increase, thus reducing its matching priority in subsequent global optimizations. Final association cost This also reflects the individual continuity and group consistency of point pairs, providing a more reliable decision-making basis for subsequent global optimal allocation.

[0144] S303. Based on the final association cost, determine the association point pair between the current time and the previous time.

[0145] Optionally, an optimal association scheme, i.e. the association scheme with the minimum total association cost, can be obtained based on a global optimal algorithm.

[0146] Alternatively, the globally optimal algorithm can be the Hungarian algorithm.

[0147] Optionally, before performing the global optimal allocation, obviously unreasonable matching combinations can be filtered out to improve the efficiency of subsequent calculations.

[0148] Specifically, a maximum association cost is set, each element in the final association cost matrix is ​​traversed, elements whose final association cost is greater than the maximum association cost are removed, and the global optimal allocation is performed based on the remaining elements.

[0149] Based on the methods provided in S301-S303 above, the spatial dispersion of the motion source is directly embedded as an adjustment factor into the calculation of the association cost. This ensures that the association process considers both the point-to-point matching degree and the structural consistency of the point in its neighborhood. Therefore, even a point with a low individual association cost (i.e., initial association cost) but which is not part of the group will be downgraded in the global optimal allocation due to the applied cost penalty. This method reduces the probability of associating random noise with the true trajectory, improving the rationality of the associated point pairs and the overall confidence level.

[0150] Combination Figure 2 ,like Figure 4 As shown, in one implementation of this application embodiment, the above-mentioned tracking point clusters corresponding to multiple time moments are used to perform obstacle verification on the tracking point clusters corresponding to the current time moment, and the tracking point clusters that pass the verification are determined as obstacles, specifically including S401-S403.

[0151] S401. Determine the continuous tracking point cluster of the first tracking point cluster based on the tracking point cluster corresponding to multiple time points.

[0152] The first tracking point cluster is any tracking point cluster corresponding to the current moment.

[0153] It should be understood that since the laser detection points in different tracking point clusters are related, the tracking point clusters are also related to each other. The continuous tracking point clusters of a tracking point cluster are used to characterize the continuity relationship between the tracking point cluster and the historical cluster.

[0154] In one alternative implementation, the number of associations between any two tracking point clusters at adjacent time points can be determined, and the two tracking point clusters with the largest number of associations can be identified as consecutive tracking point clusters.

[0155] In another alternative implementation, the Euclidean distance between the three-dimensional centroid of the first tracking point cluster and the three-dimensional centroid of the second tracking point cluster is determined. If the Euclidean distance is less than a preset distance threshold, the second tracking point cluster is determined as a continuous tracking point cluster of the first tracking point cluster.

[0156] The second tracking point cluster is any tracking point cluster corresponding to the previous time step.

[0157] It should be understood that the method for determining the three-dimensional centroid of the tracking point cluster is similar to the method for determining the three-dimensional centroid of the candidate associated points, and will not be repeated here.

[0158] It is understandable that if the Euclidean distance is less than the preset distance threshold, it means that the first tracking point cluster and the second tracking point cluster are similar in position. In this case, the second tracking point cluster can be identified as a continuous tracking point cluster of the first tracking point cluster.

[0159] Optionally, when multiple tracking point clusters are less than a preset distance threshold from the first tracking point cluster, the tracking point cluster with the smallest distance can be determined as a continuous tracking point cluster of the first tracking point cluster.

[0160] In one alternative implementation, when the first tracking point cluster successfully matches the second tracking point cluster, the first tracking point cluster can inherit the unique tracking document (ID) of the second tracking point cluster, and the number of consecutive tracking frames of the unique tracking ID can be incremented by one.

[0161] Optionally, if the Euclidean distance between the first tracking point cluster and the centroids of all the previous tracking point clusters in three-dimensional space is greater than or equal to a preset distance threshold, the first tracking point cluster can be determined to be a newly emerging cluster. In this case, a new unique tracking ID can be assigned to it, and its continuous tracking frame number T can be initialized to 1.

[0162] Optionally, the preset distance threshold can be set based on the maximum possible displacement between two adjacent frames of the drone. Its value must ensure that it can cover the distance that the drone can move within the time interval between two adjacent frames at its maximum flight speed, while avoiding mismatches.

[0163] For example, the preset distance threshold can be 1.5 meters.

[0164] S402. Based on the number of laser detection points in the first tracking point cluster, the spatial dispersion of the motion source of all laser detection points in the first tracking point cluster, and the number of continuous tracking point clusters in the first tracking point cluster, perform obstacle verification on the first tracking point cluster.

[0165] It should be understood that, based on the above determination of the continuous tracking point clusters of the first tracking point cluster, the number of continuous tracking point clusters of the first tracking point cluster is obtained, and based on the spatial dispersion of the motion source of all laser detection points in the first tracking point cluster, the mean value of the spatial dispersion of the motion source of the first tracking point cluster can be determined.

[0166] Understandably, verification based on the number of laser detection points can ensure that the tracking point cluster has a sufficient scale to represent an entity with a stable physical form; verification based on the spatial dispersion of the motion source can distinguish between a real single object and a false cluster accidentally formed by multiple different motion source points; and verification based on the number of this continuous tracking point cluster can ensure that the tracking point cluster is not a transient phenomenon, but an entity that can exist stably in the time dimension.

[0167] In one alternative implementation, the first tracking point cluster is determined to have passed verification if the number of laser detection points in the first tracking point cluster is greater than a threshold for the number of laser detection points, the mean of the spatial dispersion of the motion source is less than a threshold for the spatial dispersion of the motion source, and the number of consecutive tracking frames is greater than a threshold for the number of consecutive tracking frames.

[0168] Optionally, the threshold for the number of laser detection points is set based on the expected number of sampling points for a minimum-sized obstacle (e.g., a tire fragment or half a brick) at a typical flight altitude.

[0169] For example, the threshold for the number of laser detection points can be set to 3 to ensure that false clusters consisting of one or two isolated points are filtered out.

[0170] Optionally, the threshold for the spatial dispersion of motion sources is set based on the statistical distribution of the spatial dispersion of motion sources between real obstacles and spurious aggregates. For a real stationary rigid body, the motion sources at each point on its surface should be highly clustered in three-dimensional space. This threshold is used to distinguish between a real single object and a spurious cluster formed by multiple random motion source points.

[0171] For example, the spatial dispersion of the motion source can be 0.1 meters.

[0172] Optionally, the continuous tracking frame rate threshold is set in relation to the drone's flight speed and sensor data frame rate. The purpose is to ensure that obstacles remain in the field of view for a sufficiently long time to eliminate accidental noise aggregates.

[0173] For example, the continuous tracking frame threshold can be 3 frames.

[0174] S403. If the verification is successful, the first tracking point cluster is identified as an obstacle.

[0175] Understandably, if the verification passes, it means that the first cluster of points is a real entity, and at this point, the first tracking point cluster can be identified as an obstacle.

[0176] Based on the methods provided by S401-S403, the object of obstacle identification is elevated from discrete laser detection points or associated point pairs to the level of "tracking point clusters". By comprehensively evaluating the size of the point cluster (number of laser detection points), the number of continuous tracking point clusters, and the spatial dispersion of motion sources, a more comprehensive and reliable evaluation of potential obstacles is achieved.

[0177] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0178] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

Claims

1. An obstacle detection method for unmanned aerial vehicle (UAV) highway inspection, characterized in that, The method includes: Obtain a set of laser detection points at multiple times, including the current time; Based on the image coordinates of each laser detection point, local motion consistency deviation, and spatial dispersion of the motion source, multiple pairs of related points between adjacent time points are determined, including: Based on the image coordinates and local motion consistency deviation of the laser detection points at the current time and the previous time, the initial association cost between the point pairs at the current time and the previous time is determined. Determine the M neighboring laser detection points of the first laser detection point on the current camera image, where M is an integer greater than 1, and the current camera image is the camera image at the current moment; Based on the initial association cost between the laser detection points at the current time and the previous time, candidate association points for the first laser detection point and candidate association points for the M neighboring laser detection points are determined. Determine the three-dimensional centroid of the set of candidate associated points of the M neighboring laser detection points; Based on the Euclidean distance between the candidate associated points of the first laser detection point and the centroid of the three-dimensional space, the spatial dispersion of the motion source of the first laser detection point is determined. Based on the spatial dispersion of the motion source of each laser detection point at the current moment, the initial association cost of the point pair corresponding to each laser detection point at the current moment is adjusted to obtain the final association cost between the point pair at the current moment and the previous moment. Based on the final association cost, the associated point pairs between the current time and the previous time are determined; the local motion consistency deviation characterizes the degree of consistency between the actual motion of the local image region where the laser detection point is located and the relative motion of the stationary rigid body caused by the motion of the UAV itself; the motion source spatial dispersion is used to characterize the degree of dispersion of the motion source of the laser detection point and the motion sources of other laser detection points in its image neighborhood in three-dimensional space; the motion source is the candidate associated point of the laser detection point at the previous time. Spatial clustering is performed on the laser detection points at each time step in all associated point pairs to obtain the tracking point clusters corresponding to each time step; Based on the tracking point clusters corresponding to multiple time points, obstacle verification is performed on the tracking point cluster corresponding to the current time point, and the tracking point clusters that pass the verification are identified as obstacles.

2. The obstacle detection method for unmanned aerial vehicle (UAV) highway inspection according to claim 1, characterized in that, The method further includes: Multiple feature points within the image block corresponding to the first laser detection point are determined. The first laser detection point is any laser detection point in the set of laser detection points at the current moment, and the current moment is a moment included among multiple moments. Based on the motion information of the UAV between the previous moment and the current moment, the theoretical motion vector of each feature point is determined; The actual motion vector of each feature point from the previous time step to the current time step is determined based on the optical flow estimation algorithm; Based on the theoretical motion vectors and actual motion vectors of all feature points, the local motion consistency deviation of the first laser detection point is determined.

3. The obstacle detection method for unmanned aerial vehicle (UAV) highway inspection according to claim 1, characterized in that, The determination of the initial association cost between point pairs at the current and previous times, based on the image coordinates and local motion consistency deviation of the laser detection points at the current and previous times, includes: Based on the image coordinates of the first laser detection point and the image coordinates of the second laser detection point, the image position prediction deviation between the first laser detection point and the second laser detection point is determined, where the second laser detection point is any laser detection point at the previous moment. Based on the local motion consistency deviation of the first laser detection point and the local motion consistency deviation of the second laser detection point, the change in motion characteristics between the first laser detection point and the second laser detection point is determined. The image position prediction deviation and motion feature change are weighted and summed to obtain the initial association cost between the first laser detection point and the second laser detection point.

4. The obstacle detection method for unmanned aerial vehicle (UAV) highway inspection according to claim 1, characterized in that, The step of verifying the tracking point clusters corresponding to multiple time points, and determining the verified tracking point clusters as obstacles, includes: The first tracking point cluster is determined based on the tracking point clusters corresponding to multiple time points, where the first tracking point cluster is any tracking point cluster corresponding to the current time point; Based on the number of laser detection points in the first tracking point cluster, the spatial dispersion of the motion source of all laser detection points in the first tracking point cluster, and the number of consecutive tracking point clusters in the first tracking point cluster, obstacle verification is performed on the first tracking point cluster. If the verification is successful, the first cluster of tracking points will be identified as an obstacle.

5. The obstacle detection method for unmanned aerial vehicle (UAV) highway inspection according to claim 4, characterized in that, The step of determining the continuous tracking point cluster of the first tracking point cluster based on tracking point clusters corresponding to multiple time points includes: Determine the Euclidean distance between the three-dimensional centroid of the first tracking point cluster and the three-dimensional centroid of the second tracking point cluster, where the second tracking point cluster is any tracking point cluster corresponding to the previous time step; If the Euclidean distance is less than a preset distance threshold, the second tracking point cluster is determined as a continuous tracking point cluster of the first tracking point cluster.

6. The obstacle detection method for unmanned aerial vehicle (UAV) highway inspection according to claim 4, characterized in that, The method further includes: If the number of laser detection points in the first tracking point cluster is greater than the threshold for the number of laser detection points, the mean of the spatial dispersion of the motion source is less than the threshold for the spatial dispersion of the motion source, and the number of consecutive tracking frames is greater than the threshold for the number of consecutive tracking frames, then the first tracking point cluster is determined to have passed verification.

7. The obstacle detection method for unmanned aerial vehicle (UAV) highway inspection according to claim 1, characterized in that, Obtain a set of laser detection points at multiple times, including: Acquire camera images and multiple 3D spatial points acquired by radar at the current moment; Multiple three-dimensional spatial points are projected onto the pixel coordinate system of the camera image at the current moment to generate the set of laser detection points at the current moment.

8. The obstacle detection method for unmanned aerial vehicle (UAV) highway inspection according to claim 7, characterized in that, The step of projecting multiple three-dimensional spatial points onto the pixel coordinate system of the camera image at the current moment to generate the laser detection point set at the current moment includes: Multiple three-dimensional spatial points are projected onto the pixel coordinate system of the camera image at the current moment to generate candidate laser detection points at the current moment; Determine the local average gradient energy of the image block corresponding to the first candidate laser detection point, where the first candidate laser detection point is any candidate laser detection point at the current moment; Determine the global average gradient energy of the camera image at the current moment; The texture saliency of the first candidate laser detection point is determined based on the ratio between the local average gradient energy and the global average gradient energy. If the texture saliency of the first candidate laser detection point is greater than the texture saliency threshold, the first candidate laser detection point is added to the laser detection point set at the current time.

Images