Automatic detection and tracking method for air targets

Abstract

This application relates to the field of image measurement technology and proposes an automatic detection, tracking, and positioning method for aerial targets, applicable to high-precision perception and stable locking of low-altitude and near-altitude moving targets in complex environments. This invention employs a system redundancy and fusion detection approach, utilizing both a visible light camera and a lidar sensor to construct two sets of detection sensors with different modes for target detection. The data from these different modes are then fused, enabling automatic detection, tracking, and positioning of small aerial targets at distances greater than 500 meters in the far field. This provides target positioning information for subsequent precise detection, focusing, and real-time tracking display.

Description

Technical Field

[0001] This application relates to the field of image measurement technology and proposes an automatic detection and tracking method for aerial targets, which is suitable for high-precision perception and stable locking of low-altitude and near-altitude moving targets in complex environments. Background Technology

[0002] In scenarios such as low-altitude security and near-air target guidance, portable near-air target detection equipment needs to achieve high-precision, real-time tracking and measurement of small targets at long distances. For example, for precise guidance of takeoff and landing of runway-landing drones, it needs to achieve automatic tracking and measurement of small targets at long distances within 300m-500m and beyond 500m, such as precise automatic tracking and locking of runway-landing drones during flight. However, traditional detection technologies have many limitations:

[0003] (1) Shortcomings of single sensor performance. A single visible light camera or infrared camera can capture the high-density contours of a target, but the far-field depth measurement error is large. For example, the error can be greater than 1.5m at a distance of 100m and greater than 8m at a distance of 500m. A single lidar has high depth accuracy, but the point cloud is sparse when detecting air targets, making it difficult to fully characterize the target shape. The radar reflective area of ​​small targets at long distances is small and the motion characteristics are complex. Single lidar detection is easily affected by birds, dust, etc., and the pixel ratio of far-field targets is extremely low, resulting in a high rate of missed detection. In addition, the data coordinate systems of the two types of equipment are not unified and cannot be directly fused.

[0004] (2) The traditional processing flow lacks real-time performance. When performing target recognition and detection based on high-precision cameras, the computational load of target feature extraction is extremely large, which does not meet the real-time requirements of rapid tracking and locking;

[0005] (3) Complex environment and lack of redundancy guarantee: Sand and dust in the field and weak light will cause the camera imaging quality to decline. Furthermore, traditional systems have no redundancy fault tolerance mechanism, and the failure of a single sensor will cause the detection to be interrupted. At the same time, portable devices need to balance lightweight and detection performance. Traditional heavy detection equipment cannot be adapted to individual / portable deployment scenarios. Summary of the Invention

[0006] To overcome at least one problem or deficiency in the prior art, this application proposes an automatic detection and tracking method for air targets.

[0007] An automatic detection and tracking method for aerial targets, the tracking method mainly includes steps S100-S600, wherein: S100, acquiring images of aerial targets in the same scene by a visible light camera group and a lidar group respectively, obtaining 2D images from multiple cameras through the visible light camera group, and a second point cloud image through the lidar group; S200, performing long-range small target detection and preliminary localization on the 2D images, collecting the first feature of the target in real time, if the first feature of the target does not contain a preset cooperative identifier feature, then performing small target feature detection S210 on the 2D image, otherwise performing cooperative identifier point detection S220 on the 2D image, and outputting the first target detection result; S300, when the target is at a long distance, treating the small target as a... The system processes targets at single points for visual intersection measurement and localization. When a target enters the mid-range range, it switches to cooperative marker point detection, measuring the target pose at multiple points to locate the target marker point model. In step S400, when a target enters the monitoring range of the lidar group, it performs fast point cloud target detection based on a grid map. By rasterizing the second point cloud image, a grid map is established. The target size and shape are extracted based on the spatial distribution characteristics of the target in the grid map, and the target is detected and located, outputting the second target detection result. In step S500, the second target detection result is combined with the first target detection result to perform multi-sensor data fusion detection, resulting in the third target detection result. In step S600, the target is locked and continuously tracked, completing automatic tracking and measurement of small targets at long distances.

[0008] Based on some of the above embodiments, the further small target feature detection S210 mainly includes steps S211 to S214, wherein: S211, based on block matching inverse synthesis search, the current frame image of the 2D image is divided into non-overlapping image blocks of fixed size, each image block is used as a reference block to be matched; according to each reference block, a certain range of search window is set in the next frame image, the similarity between the reference block and all candidate blocks in the search window is calculated, and the candidate block with the highest similarity is selected as the matching block; the displacement vector of the matching block relative to the reference block is recorded to construct a sparse optical flow field; S212, multi-scale pyramid reconstruction is performed on the sparse optical flow field, and the current frame image and the next frame image are downsampled at multiple scales respectively to form a pyramid. The system generates image sequences at different scales; at each scale layer, it interpolates and supplements missing optical flow points, upsamples layer by layer, and fuses the optical flow information of adjacent scale layers to obtain a first dense optical flow field; in step S213, it performs variational optimization on the first dense optical flow field, using global smoothness constraints and local brightness constant constraints, and corrects discontinuous regions in the optical flow field by minimizing the energy function, continuously adjusting the optical flow vector of each pixel until the energy function converges to the minimum value, thereby obtaining an optimized second dense optical flow field to capture the motion trajectory of small targets; in step S214, if there is a significant motion trajectory in the second dense optical flow field output in step S223 and it meets the preset small target feature template matching threshold, then it is determined that a small target exists, and the first target detection result is output; otherwise, it is determined that there is no valid target, and the system returns to step S210 to continue detection.

[0009] Based on some of the above embodiments, the further cooperative marker detection S220 mainly includes steps S221 to S224, wherein: S221, performing marker point ROI extraction, binarizing the 2D image, extracting areas with brightness higher than a threshold as marker point candidate areas, matching the marker point candidate areas with a preset marker point template, and filtering out marker point ROI areas that conform to geometric shape and size characteristics; S222, performing anisotropic diffusion filtering on the marker point ROI areas, and calculating the gray-level gradient value of the 8-neighbor pixels of each pixel in the marker point ROI area. S223: The diffusion coefficient is dynamically adjusted according to the magnitude of the gray-level gradient value, and the marker point filtered image is output; S224: The gradient magnitude and direction of the marker point filtered image are calculated using a gradient operator, and edge points with gradient magnitudes greater than the preset gradient threshold are selected according to the preset gradient threshold. Adjacent discrete edge points are connected to form continuous edges through 8-neighborhood connectivity analysis, and the pixel-level contour of the marker points is extracted; S225: A preset local window is set with each pixel-level edge point in the pixel-level contour of the marker points as the center, the Zernike moment of the image within the local window is calculated, the gray-level distribution features of the edge points are extracted, and based on Zernike... The quantitative relationship between the IKE moment and the edge position is calculated to obtain the precise coordinates of the edge points within the pixel, thus obtaining the sub-pixel edge point coordinates. A set fitting is performed on all sub-pixel edge points to calculate the sub-pixel level contour or sub-pixel level center coordinates of the marker points. S225: The sub-pixel level contour or sub-pixel level center coordinates of the marker points are transformed with a preset three-dimensional spatial distribution coordinate system for the marker points to obtain the true coordinate system of the target marker points. The true marker point model of the target is reconstructed, and the Euclidean distance between the centroids of each marker point is calculated. S226: The two marker points with the smallest centroid distance are identified, and the distance between the centroids of these two marker points is calculated. One point is used as the initial starting point, and another point is used as the second point to form the initial chain code. The remaining marker points are connected in sequence according to the minimum distance principle until the last point and the starting point form the final chain code, thus generating a closed chain code. In S227, the first difference of the closed chain code of the real marker model is calculated to obtain the differential chain code. The differential chain code of the real marker model is matched and compared with the differential chain code of the preset marker model. If the two differential chain codes are the same, it is determined that the measured marker point and the marker model are successfully matched, and the first target detection result is output. Otherwise, the initial starting point is changed and S226 is re-executed to rebuild the chain code until the differential chain code is successfully matched.

[0010] Based on some of the above embodiments, the further single-point target visual intersection measurement and positioning includes the following steps: S311, synchronously imaging the target using each camera in the visible light camera group; after the target is detected in the first target detection result output by each camera in the visible light camera group, calculating the target sub-pixel level center point in each 2D image using the gray-scale centroid method; S312, constructing a visual intersection equation system based on the target sub-pixel level center point in each 2D image, combined with the intrinsic and extrinsic parameter matrices of each camera, and solving to obtain the single-point target three-dimensional coordinates in the world coordinate system.

[0011] Based on some of the above embodiments, the target marker model pose measurement and positioning further includes the following steps: S321, based on the successfully matched marker model in S227, the measured marker points of each camera in the visible light camera group are matched one-to-one, the marker point coordinates of the measured target are calculated, and the measured marker point cloud is obtained; S322, the measured marker point cloud is registered with the preset marker model by spatial rigid body transformation, the rotation matrix and translation matrix are solved by singular value decomposition, the target pose estimation is completed, and the three-dimensional attitude angle and position parameters of the target in the world coordinate system are output, thus completing the pose calculation of the target marker model.

[0012] Based on some of the above embodiments, the target marker model pose measurement and localization further includes a feature point-based tracking acceleration measurement process. When performing cooperative marker detection S220 on the 2D image, global feature point detection is first performed on the 2D image acquired by one camera in the visible light camera group. The measured marker points and the target marker model are extracted and matched. After determining the correspondence of the successfully matched marker points, subsequent time series images and images from other cameras in the visible light camera group only perform local feature search and marker point relationship matching in the corresponding areas.

[0013] Based on some of the above embodiments, the further rapid point cloud target detection based on grid map includes the following steps: S401, establishing a grid map, dividing the lidar detection area into multiple uniform three-dimensional grid units, and discretizing the point cloud data of the second point cloud image; S402, transforming the point cloud coordinate system of the discretized point cloud data to the grid map coordinate system, traversing point by point, determining the grid unit to which each point belongs, and counting the number and distribution characteristics of point clouds in each grid, and outputting a grid grayscale map; S403, denoising the grid grayscale map according to the distribution characteristics and grayscale information of the point clouds, and matching it according to the distribution feature template of the grid image of the preset target model to filter out candidate grid areas where the suspected target is located; S404, clustering and segmenting the point cloud data in the candidate grid areas, extracting target point cloud clusters, calculating the centroid coordinates of all point clouds in the target point cloud clusters, obtaining target location information, completing target detection and localization, and outputting the second target detection result.

[0014] Based on some of the above embodiments, the further multi-sensor data fusion detection includes the following steps: S501, simultaneously acquiring the first target detection result output by the visible light camera group and the second target detection result output by the lidar group; S502, projecting the lidar point cloud of the second target detection result onto the camera image plane through a calibrated transformation matrix, calculating the image pixel coordinates corresponding to each lidar point cloud; extracting the sub-pixel contour and grayscale features of the marker points in the first target detection result, and back-projecting the image pixel coordinates onto the three-dimensional space of the lidar point cloud through the extrinsic parameter matrix of the visible light camera group to obtain the initial visual ray; using grayscale... Similarity and spatial distance thresholds are used as constraints to match the intersections of the laser point cloud and the initial visual ray, determining the one-to-one correspondence between the laser point cloud and image pixels. Successfully matched point clouds are fused to output a high-precision target 3D feature set. In S503, the high-precision target 3D feature set is input into a spatiotemporal consistency filter to verify the consistency of matching results across multiple consecutive frames, eliminating mismatched point clouds and outputting a high-density initial visual point cloud covering the complete outline of the target. In S504, the high-density initial visual point cloud is corrected by selecting effective seed points based on the distribution pattern of the difference between laser depth and visual depth. The laser point cloud confidence and image point cloud confidence are used to correct the effective seed points to obtain corrected seed points; the normal vector of the neighborhood of the corrected seed point is calculated. If the normal vector deviation is greater than a preset deviation value, the weights of the laser point cloud confidence and image point cloud confidence are readjusted, and iterative correction continues to output a high-density corrected point cloud; S505, the high-density corrected point cloud is interpolated and completed, and the high-density corrected point cloud is divided into flat regions and edge regions according to the curvature of the target surface; in the flat regions, the depth of non-seed points in the flat regions is completed by bilinear interpolation based on the depth of adjacent corrected seed points; in the edge regions, the depth of non-seed points in the flat regions is completed by bilinear interpolation based on the depth of adjacent corrected seed points; in the edge regions, the depth of non-seed points in the flat regions is completed by bilinear interpolation based on the depth of adjacent corrected seed points. At the sub-point center, a continuous radial basis function is constructed to fit a continuous surface covering the entire edge region. The interpolation density is adaptively adjusted in combination with curvature changes to complete the high-precision interpolation of the edge region. A complete preliminary fused point cloud covering the target surface is generated. In S506, the preliminary fused point cloud is spatiotemporally optimized. The motion consistency constraint of the point cloud between consecutive frames is used to compensate for the target's dynamic displacement. If the depth difference of the point cloud at the same position in adjacent frames exceeds a preset threshold, the interpolation parameters of S505 are adjusted based on the depth of the corrected seed point in the current frame to re-optimize the surface fitting result, obtain the optimized fused point cloud, and output the third target detection result.

[0015] Based on some of the above embodiments, the further continuous target tracking S600 includes redundancy measurement fault-tolerant processing S610, wherein: S611, the point cloud coordinates and depth data in the first target detection result are used as first redundancy information, and the point cloud coordinates and depth data in the second target detection result are used as second redundancy information, to confirm whether the visible light camera group and the lidar group have completed satellite positioning and timing synchronization and three-dimensional coordinate system fusion calibration; S612, the visible light camera group and the lidar group perform device self-testing, and perform self-testing on the camera's frame rate, exposure parameters, and interface communication, and perform self-testing on the lidar's echo intensity, point cloud density, and other parameters. The system performs a self-test on the voltage. If any hardware parameter exceeds the standard threshold, the redundancy is marked as an equipment malfunction. In step S613, the visible light camera group and the lidar group perform signal self-tests, including detection range verification, frame rate timing verification, and noise verification. If any signal parameter exceeds the standard threshold, the redundancy is marked as a signal malfunction. In step S614, the visible light camera group and the lidar group perform dual-redundancy cross-consistency verification, including spatial deviation calculation and consistency determination. By comparing the first and second redundancy information at the same time, the Euclidean distance deviation of the same target is calculated. If the Euclidean distance deviation is less than the preset dynamic adjustment based on the target distance... If the deviation threshold is met, the spatial consistency requirement is satisfied; otherwise, a secondary verification is initiated, extracting the grayscale features of the marker points of the first redundancy information and matching them with the depth contour gradient features of the second redundancy information. If the feature matching degree is greater than the preset matching threshold, the data drift is determined to be correctable; otherwise, it is determined to be a single redundancy fault. S615: When any redundancy is determined to be an equipment malfunction, signal malfunction, or single redundancy fault, its data output is immediately cut off, retaining only the data channels of the normal redundancy and the correctable redundancy as effective redundancy. S616: The retained normal redundancy and correctable redundancy undergo fault-tolerant reconstruction processing, using real-time self-test pass rate and mutual test consistency. The system calculates the confidence weight of each effective redundancy based on the environmental adaptation coefficient, and normalizes the redundancy confidence weights of the first and second redundancy information in the same time sequence to obtain the confidence level of each redundancy. In step S617, signal voting is performed based on the states of the first and second redundancy information in the same time sequence. When both are effective redundancies, the final third target detection result is output through weighted fusion. When only one is an effective redundancy, its corresponding target detection data is directly output. If both redundancies fail, an emergency mode is activated, calling the most recent historical effective fusion data and combining it with the inertial navigation calculation results to output the predicted third target detection result.

[0016] Based on some of the above embodiments, the target locking and continuous tracking S600 further includes target confirmation processing S620, wherein: S621, a target confirmation request is initiated to confirm the target in the third target detection result; S622, after receiving the target confirmation instruction, the target is locked, the multi-sensor collaborative tracking mode is started, the target is tracked in real time, and the automatic tracking and locking measurement of long-distance small targets is completed.

[0017] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention include:

[0018] (1) The present invention adopts a dual-mode sensor architecture of visible light camera group + lidar group. It adopts the idea of ​​system redundancy and fusion detection. It uses two kinds of sensors, visible light camera and lidar, to form two different modes of detection sensors to perform target detection. The two can independently complete the detection and positioning, and at the same time, they can achieve a unified coordinate system through fusion calibration. The high-precision depth data of lidar can correct the sparse depth measurement of visible light camera group, and the high-density contour data of visible light camera group can make up for the deficiencies of lidar point cloud. It significantly reduces the positioning error of far-field targets above 500 meters, and provides target positioning information for subsequent accurate detection, focusing and real-time tracking display, meeting the requirements of precise guidance for UAV gliding.

[0019] (2) The present invention designs a two-stage detection scheme for near and far ranges. In the far field (greater than 500 meters), the reverse synthesis fast optical flow method is used to detect small targets. In the near field, sub-pixel marker point extraction and model matching are used to achieve high-precision positioning. Combined with the feature point tracking acceleration algorithm, the global marker point is extracted only for the first frame, and the local tracking method is used for subsequent frames to avoid repeated calculations. Through experiments, it was found that the attitude calculation time is mainly proportional to the number of cameras and the number of marker points, and is independent of the camera resolution. The average attitude calculation time for 50 marker points is 12ms, i.e. 83Hz, which is far better than the expected 15Hz requirement.

[0020] (3) The present invention can also add a redundant measurement fault-tolerant module, which can eliminate faulty equipment or abnormal data. At the same time, it improves the anti-interference capability of data by using a fusion strategy of correcting camera depth values ​​through lidar. It solves the problem that detection interruption will occur when the imaging quality deteriorates due to the failure of a single sensor or complex environment (dust, low light). It is also suitable for lightweight design, and the portability of the device meets the needs of mobile scenarios such as low-altitude security and drone guidance.

[0021] It should be noted that different embodiments of this application may produce different beneficial effects. In different embodiments, the beneficial effects that may be produced may be any one or more combinations mentioned in this application, or any other beneficial effects that may be obtained that are not exhaustively described. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. For those skilled in the art, without creative effort, the solutions shown in these drawings can be replaced, adjusted, combined, etc., to create different technical solutions; and this application can also be applied to other similar scenarios based on these drawings to obtain application solutions for other scenarios.

[0023] in:

[0024] Figure 1 A schematic diagram of the tracking and locking method workflow is shown according to some embodiments of this application;

[0025] Figure 2 A schematic diagram of the workflow of the visible light camera group target detection method in this application;

[0026] Figure 3 According to the schematic diagram of small target detection based on optical flow in two consecutive frames in this application;

[0027] Figure 4 Based on the target marker detection effect diagram in this application;

[0028] Figure 5 Based on the schematic diagram of the sub-pixel identifier extraction process in this application;

[0029] Figure 6 Based on the schematic diagram of the target marker modeling process in this application;

[0030] Figure 7 Based on the schematic diagram of the target marker modeling results in this application;

[0031] Figure 8 Based on the target centroid extraction diagram of the visible light camera group in this application;

[0032] Figure 9 A schematic diagram of the workflow of the accelerated measurement method based on feature point tracking in this application;

[0033] Figure 10 According to the schematic diagram of the two stacked lidar units in this application;

[0034] Figure 11 A schematic diagram of the workflow of the raster image method in this application;

[0035] Figure 12 A schematic diagram of the workflow of the raster image method in this application;

[0036] Figure 13 A schematic diagram of the workflow of the lidar group target detection method in this application;

[0037] Figure 14 A schematic diagram of the workflow of the multi-sensor data fusion processing method in this application;

[0038] Figure 15 A schematic diagram of the workflow of the redundancy measurement fault tolerance processing method in this application;

[0039] It should be noted that, due to the large number of figures and reference numerals in this application, if there are any discrepancies between the description of the figures and the illustrations in the specification, those skilled in the art should understand them based on the logic of the technical principles described in this application. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and it is not possible to exhaustively describe all embodiments.

[0041] like Figure 1 As shown, in some embodiments, the tracking and positioning method proposed in this invention mainly includes steps S100-S600. It employs a system redundancy and fusion detection approach, utilizing a visible light camera and a lidar sensor to form two sets of detection sensors with different modes for target detection. The data from these different modes are then fused to achieve automatic detection, tracking, and positioning of small aerial targets in the far field greater than 500 meters. This provides target positioning information for subsequent precise detection, focusing, and real-time tracking display. The system corresponding to this method can consist of a long-range, small-field-of-view high-resolution visible light camera group, a long-range, small-field-of-view lidar group, a data processor, a portable power supply, a data fusion detection module, and a redundancy measurement fault-tolerant module. In some embodiments, the visible light camera group and the lidar group in this system can also independently perform aerial target detection, identification, positioning, and guidance capabilities, and can autonomously select to enable dual-mode collaborative or single-mode operation based on ambient lighting conditions or mission requirements.

[0042] S100: Acquire images of the same small target in a distant scene by a visible light camera group and a lidar group respectively. Obtain 2D images of multiple camera groups through the visible light camera group and a second point cloud image through the lidar group.

[0043] In some embodiments, the visible light camera group may be a long-range, small-field-of-view, high-resolution camera group, which may include at least two visible light cameras. After camera calibration, it can form a binocular stereo camera. Each camera synchronously acquires image data of the target scene and synchronizes with the visible light camera group through a satellite positioning and timing module to ensure time consistency. In some embodiments, the lidar group may be a long-range, small-field-of-view lidar group, which may include at least two lidars. Each lidar synchronously acquires three-dimensional point cloud data of the target area and maintains time consistency with the visible light camera group through a satellite positioning and timing module.

[0044] S200, such as Figure 2 As shown, the 2D image is subjected to long-distance small target detection and preliminary localization. The first feature of the target is collected in real time. If there is no preset cooperative identification feature in the first feature of the target, the small target feature detection S210 is performed on the 2D image. Otherwise, the cooperative identification point detection S220 is performed on the 2D image, and the first target detection result is output.

[0045] When using a visible light camera array to quickly and accurately detect small aerial targets in the far field, it can generally be divided into two stages. The first stage is the initial localization stage for small target detection. However, as the target gradually approaches and becomes larger, the pixel ratio of the target in the image increases. At this point, we enter the second stage, which not only needs to detect the target as a whole, but also needs to detect and identify the specific feature components or cooperative identifiers of the target.

[0046] In some embodiments, a deep learning-based target detection algorithm can be used to process 2D images and extract the contour, texture, and motion features of the target. However, due to the strong dependence of deep learning models on training data, the recognition rate of small targets at ultra-long distances of more than 500 meters is not high. Therefore, the small target feature detection S210 of this invention can use an optical flow-based small target detection algorithm, which may include steps S211~S214. Through block-matching-based inverse synthesis search, multi-scale sparse-to-dense interpolation is used to obtain a dense optical flow field. Then, through variational dense optical flow correction, the displacement between each pixel between image frames is calculated to achieve motion perception of scene targets. Figure 3 As shown, the left and right sides are schematic diagrams of the target detection effect of two consecutive frames. This method can also preset multiple constraints, such as motion continuity constraints, constant brightness assumptions, and spatial smoothness constraints, requiring pixels within the local window to maintain motion consistency, requiring brightness between adjacent frames to remain constant, and requiring the motion of the target object between adjacent frames to be small and not exceed a preset threshold, so as to suppress noise interference and improve the detection stability of small targets at a distance.

[0047] S211, based on block matching, the inverse synthesis search divides the current frame image of the 2D image into non-overlapping image blocks of fixed size, with each image block serving as a reference block to be matched; according to a certain range of search windows set for each reference block in the next frame image, the similarity between the reference block and all candidate blocks within the search window is calculated, and the candidate block with the highest similarity is selected as the matching block; the displacement vector of the matching block relative to the reference block is recorded to construct a sparse optical flow field;

[0048] S212, perform multi-scale pyramid reconstruction on the sparse optical flow field, and downsample the current frame image and the next frame image at multiple scales to form image sequences at different scales; interpolate and supplement missing optical flow points at each scale layer, upsample layer by layer and fuse the optical flow information of adjacent scale layers to obtain the first dense optical flow field.

[0049] S213, perform variational optimization on the first dense optical flow field, utilize global smoothness constraints and local brightness constant constraints, and correct discontinuous regions in the optical flow field by minimizing the energy function. Continuously adjust the optical flow vector of each pixel until the energy function converges to the minimum value to obtain the optimized second dense optical flow field and capture the motion trajectory of small targets.

[0050] S214. If there is a significant motion trajectory in the second dense optical flow field output by S223 and the preset small target feature template matching threshold is met, then it is determined that a small target exists and the first target detection result is output; otherwise, it is determined that there is no valid target and returns to S210 to continue detection.

[0051] When performing flight target detection, the primary goal is to quickly identify targets with high motion components between adjacent frames. Therefore, the optical flow method described above does not require additional time for variational dense optical flow correction, thus accelerating the operation of the optical flow method while still ensuring the detection of moving targets. Furthermore, the optical flow method used in this approach is based on block matching, resulting in weak data dependencies and high algorithm parallelism. This allows for further acceleration of the algorithm's execution speed through parallel programming.

[0052] In some embodiments, when the target is closer and its pixel proportion in the image increases, a sub-pixel marker detection and model matching procedure can be enabled, such as... Figure 5 As shown, the cooperative marker detection S220 may include steps S221~S224. Figure 4 As shown, the marking can be a high-contrast pattern or an active luminous mark pre-set on the surface of the aircraft, and can be a marking sticker, marking ball, or luminous marking.

[0053] S221, perform ROI (Region of Interest) extraction. The 2D image is binarized, and regions with brightness above a threshold are extracted as candidate ROIs. These candidate ROIs are then matched against a preset ROI template to select those that meet the geometric shape and size characteristics. Step S221 uses binarization to effectively segment the image, obtaining candidate ROIs with higher brightness. The final ROIs are then obtained through matching with the ROI template.

[0054] To obtain accurate sub-pixel marker location, further filtering and edge extraction operations are required for the marker ROI region.

[0055] S222, anisotropic diffusion filtering is performed on the ROI region of the marker point using a diffusion filter, the gray-level gradient value of the 8 neighboring pixels of each pixel in the ROI region of the marker point is calculated, the diffusion coefficient is dynamically adjusted according to the magnitude of the gray-level gradient value, and the marker point filtered image is output.

[0056] Smoothing filtering is a crucial step in dealing with noisy images. Various filters can be chosen, such as Gaussian filtering, median filtering, mean filtering, and bilateral filtering. However, anisotropic diffusion filtering effectively preserves the edge information of landmark points while suppressing noise, preventing a decrease in positioning accuracy due to excessive smoothing. The diffusion filter treats the 2D image as a matter vector field, where the gray value of each pixel is equivalent to the matter concentration. Image smoothing is analogous to the diffusion process of matter in the vector field. By constructing an appropriate diffusion coefficient, it is possible to increase smoothness within the ROI region and reduce smoothness between regions, thus maintaining clear edges while suppressing noise.

[0057] In step S223, a gradient operator is used to calculate the gradient magnitude and direction of the filtered image of the marker points. Edge points with gradient magnitudes greater than a preset gradient threshold are selected. Adjacent discrete edge points are connected using 8-neighborhood connectivity analysis to form continuous edges, thus extracting the pixel-level contours of the marker points. Sub-pixel edge localization is performed based on pixel-level edges; therefore, in step S223, pixel-level edge points are first extracted from the ROI region image, and then sub-pixel edge points are found near these coarsely located points.

[0058] S224, taking each pixel-level edge point in the pixel-level contour of the marker point as the center, set a preset local window, calculate the Zernike moment of the image within the local window, extract the gray-level distribution features of the edge points, and calculate the precise coordinates of the edge points within the pixel based on the quantitative relationship between the Zernike moment and the edge position to obtain the sub-pixel edge point coordinates. Perform set fitting on all sub-pixel edge points to calculate the sub-pixel contour of the marker point or the sub-pixel center coordinates of the marker point.

[0059] Zernike moments are an integral operator with good noise resistance and significant advantages in detection performance. Step S224, based on the high positioning accuracy of the Zernike moment method, assumes that the edge of the target image is a step model. Utilizing the rotational invariance of moments, equations are established based on the relationship between moments before and after rotation to solve for the edge parameters, thereby achieving sub-pixel positioning.

[0060] Next, the target marker model is reconstructed, mapping the markers on the real target to the target digital model, laying the foundation for subsequent pose detection.

[0061] S225, the sub-pixel-level contour or sub-pixel-level center coordinates of the marker points are transformed with a preset three-dimensional spatial distribution coordinate system of the marker points to obtain the true coordinate system of the target marker points. The true marker point model of the target is reconstructed, and the Euclidean distance between the centroids of each marker point is calculated. Generally, the three-dimensional spatial coordinates of the marker points on the surface of the aerial target are measured using measuring equipment such as a visible light camera group or a total station. Select three points from all the marker points, or select two points and the centroid of all the points (three points in total). Establish an aerial target coordinate system using these three points. Specifically, this can be done by using these three points... Form a plane C, with With origin, vector The x-axis is the axis that passes through. The normal line perpendicular to plane C With the y-axis, Using the z-axis, we obtain, as follows Figure 6 The marker points are distributed in a three-dimensional coordinate system C1. Then, all points are transformed to the C1 coordinate system to obtain the C1 coordinates of the marker points. Thus reconstructing Figure 7 The target marker model is shown.

[0062] Then, the marker points detected in real time, such as the subpixel-level contours or subpixel-level center coordinates of the marker points, are matched and identified with the pre-established target marker point model. Since the spatial position relationship is not deformed, the Euclidean distance between the marker points can be used as the matching basis. The chain code matching method is used to associate the real-time detected marker points with the corresponding points in the preset model to ensure that the identification results are unique and stable.

[0063] S226. Find the two markers with the smallest centroid distance among the Euclidean distances between the centroids of each marker point. Take one of these two markers as the initial starting point and the other as the second point to form the initial chain code. Connect the remaining markers in sequence according to the principle of minimum distance until the last point and the starting point form the final chain code, thereby generating the closed chain code.

[0064] S227: Calculate the first difference of the closed chain code of the real marker model to obtain the differential chain code. Match and compare the differential chain code of the real marker model with the differential chain code of the preset marker model. If the two differential chain codes are the same, it is determined that the measured marker and the marker model are successfully matched, and the first target detection result is output. Otherwise, change the initial starting point and re-execute S226 to reconstruct the chain code until the differential chain code is successfully matched.

[0065] Furthermore, in S300, when the target is at a distance, the small target is treated as a point for single-point visual intersection measurement and localization; when the target enters the mid-range range, it switches to cooperative marker point detection, and performs target marker point model pose measurement and localization by measuring the target pose through multiple points.

[0066] In some embodiments, the single-point target visual intersection measurement and localization includes the following steps: S311 Target centroid extraction, synchronously imaging the target using each camera in the visible light camera group, and after the target is detected in the first target detection result output by each camera in the visible light camera group, calculating the target sub-pixel-level center point in each 2D image using the gray-scale centroid method, such as... Figure 8 The left and right sides shown are the target center points of the left and right cameras in the visible light camera group, respectively. S312 stereo vision single-point intersection measurement, based on the sub-pixel level target center points in each 2D image, combined with the intrinsic and extrinsic parameter matrices of each camera, constructs a set of visual intersection equations, and solves them to obtain the single-point target's three-dimensional coordinates in the world coordinate system.

[0067] In some embodiments, the target marker model pose measurement and positioning includes the following steps: S321 Calculate the measured marker points of the target; based on the successfully matched marker point model in S227, perform one-to-one matching of the measured marker points of each camera in the visible light camera group, calculate the marker point coordinates of the measured target, and obtain the measured marker point point cloud. Generally, the one-to-one matching of marker points in the images of each camera in a multi-view visible light camera group can be achieved through the point cloud matching algorithm ICP (Iterative Closest Point) and the random sample consensus algorithm RANSAC (Random Sample Consensus), achieving high-precision alignment. The single-point intersection measurement method is used to calculate the three-dimensional coordinates of each marker point in the world coordinate system, forming a spatial mapping relationship between the measured point cloud and the preset model point cloud. Generally, the bundle adjustment method can be used to optimize the visual intersection of single-point targets, reduce the impact of observation errors on three-dimensional positioning, and improve the accuracy of single-point intersection measurement.

[0068] S322 performs spatial rigid body transformation registration between the measured marker cloud and the preset marker model, uses singular value decomposition to solve the rotation and translation matrices, completes target pose estimation, outputs the three-dimensional attitude angles and position parameters of the target in the world coordinate system, and completes the pose calculation of the target marker model.

[0069] Because the target may experience occlusion or distortion of marker points during attitude transformation, using all marker points for attitude calculation can lead to decreased registration accuracy or even failure. Therefore, before calculating the attitude transformation, the marker point cloud should be screened to remove occluded or distorted outliers, retaining only stable and visible marker points for registration. Generally, three non-collinear marker points can be randomly selected to form a reference triangle. The distances between each pair of points are calculated. If the change in distance between any two points and their initial positions is less than a threshold, these three points are considered to be unoccluded or significantly distorted, and are used as the reference point cloud. Other marker points are then used as a benchmark for consistency verification. Otherwise, three new points are selected. The distances of each remaining point to the reference point cloud are then calculated. If the distances to the reference point cloud are all consistent with the initial distances, the point is considered undistorted and added to the reference point cloud. If the deviation exceeds a preset tolerance range, it is considered an outlier, added to the deformed point cloud, and removed. This ensures that all marker points participating in registration are stable and reliable, thereby improving the robustness and accuracy of pose estimation. After obtaining the reference point cloud, the pose change of the target marker point model can be calculated based on the matching relationship of the point cloud, namely the rotation matrix R and the translation vector T.

[0070] In the target detection algorithms described above, small target detection and measurement only requires tracking small targets in the image and measuring only one point, thus exhibiting high detection and measurement efficiency. However, as the target distance gradually decreases and its scale in the image gradually increases, extracting feature points in every frame during marker point measurement significantly increases the computational burden, leading to a decrease in computational efficiency. Therefore, a method such as... Figure 9 The lightweight measurement strategy based on feature point tracking, as shown, further includes feature point-based tracking to accelerate the measurement process when measuring the pose of the target marker model. During cooperative marker detection S220 on the 2D image, global feature point detection is first performed on the 2D image acquired by one camera in the visible light camera group. The measured marker points and the target marker model are extracted and matched. After determining the correspondence of successfully matched marker points, subsequent time-series images and images from other cameras in the visible light camera group only undergo local feature searching and marker point relationship matching in the corresponding regions to complete the 3D position calculation of the marker points. This avoids the high computational overhead of frame-by-frame global search and significantly improves the system's real-time performance.

[0071] Throughout the entire 3D position calculation process, global marker search and matching are performed only in the very first frame. In subsequent time-series images, marker extraction is only performed within local areas. Marker matching between subsequent image sequences is also based on prediction and constraints using the matching results of the previous frame, employing optical flow or template matching to search for corresponding points in neighboring regions. This significantly reduces computational load and improves processing speed. Experiments show that the pose calculation time is primarily proportional to the number of cameras and markers, and independent of camera resolution. The average pose calculation time for 50 markers is 12ms (83Hz), far exceeding the expected 15Hz requirement.

[0072] When a target enters the monitoring range of the lidar array, the lidar is used to detect and identify the target. Since we are using lidar for air detection, the radar receives relatively few echoes, resulting in sparse point cloud data and a limited amount of target data. Therefore, to increase the amount of target data, two lidar units can be stacked to form a lidar array, such as... Figure 10 As shown, the two lidars are configured with largely overlapping fields of view, but with misaligned longitudinal scanning beams, thereby increasing the lidar's 3D imaging resolution. For example, the lidar resolution can be changed from 1750×700 to 1750×1400, thus increasing the amount of target data.

[0073] S400, when a target enters the monitoring range of the lidar group, perform fast point cloud target detection based on a grid map. This is done by rasterizing the second point cloud image to create a grid map, such as... Figure 11As shown, the target size and shape are extracted based on the spatial distribution characteristics of the target in the grid map, the target is detected and located, and the second target detection result is output.

[0074] In some embodiments, such as Figure 13 As shown, the fast point cloud target detection based on a grid map may include steps S401-S404. S401: Establish a grid map by dividing the LiDAR detection area into multiple uniform three-dimensional grid units and discretizing the point cloud data of the second point cloud image. This invention uses a three-dimensional LiDAR to scan the monitoring area and obtain corresponding three-dimensional point cloud data. However, since the point cloud data of the three-dimensional LiDAR is non-standardized, when the data volume is large, the LiDAR cannot withstand its huge processing time and spatial complexity, and usually cannot directly process the raw data in a timely manner. Therefore, a grid map can be constructed to simplify the data. In some embodiments, lightweight map structures such as feature maps, topology maps, octree maps, and direct representation methods can be constructed to further reduce computational complexity. S402: Transform the point cloud coordinate system of the discretized point cloud data to the grid map coordinate system, perform point-by-point traversal, determine the grid unit to which each point belongs, and count the number and distribution characteristics of the point cloud within each grid, outputting as shown in the figure. Figure 12 The image shows a raster grayscale map. S403: Based on the distribution characteristics and grayscale information of the point cloud, the raster grayscale map is denoised, and matched against the distribution feature template of the raster image of the preset target model to filter out candidate raster regions where the suspected target is located. S404: The point cloud data within the candidate raster regions is clustered and segmented to extract target point cloud clusters. The centroid coordinates of all points in the target point cloud clusters are calculated to obtain the target location information, completing target detection and localization, and outputting the second target detection result. The raster map is actually a planar map of real size, with each raster corresponding to a fixed size. Therefore, it is possible to determine whether a target of interest is a target based on its actual size in the air and the size and shape of the target in the output raster image.

[0075] S500, such as Figure 14 As shown, a third target detection result is obtained by fusing multi-sensor data based on the second target detection result and the first target detection result. Since the depth measurement accuracy of visible light camera groups is generally relatively low, but their data volume is large, while the depth measurement accuracy of lidar groups is generally relatively high, but their data is sparse, the advantages of both can be combined. By fusing the high-resolution texture information of the visible light camera group with the precise depth information of the lidar group, the lidar can be used to correct the depth value measured by photogrammetry. This results in fused data, after correction using image point clouds, possessing both high-precision depth measurement and high-density data sampling points, thus constructing a dense and accurate environmental perception model.

[0076] In some embodiments, the multi-sensor data fusion detection may include steps S501-S506.

[0077] S501 simultaneously acquires the first target detection result output by the visible light camera group and the second target detection result output by the lidar group.

[0078] S502, the laser point cloud of the second target detection result is projected onto the camera image plane through the calibrated transformation matrix, and the image pixel coordinates corresponding to each laser point cloud are calculated; the sub-pixel level contour and grayscale features of the marker points in the first target detection result are extracted, and the image pixel coordinates are back-projected onto the three-dimensional space of the laser point cloud through the extrinsic matrix of the visible light camera group to obtain the initial visual ray; with grayscale similarity and spatial distance threshold as constraints, the intersection points of the laser point cloud and the initial visual ray are matched to determine the one-to-one correspondence between the laser point cloud and the image pixels, the successfully matched point clouds are fused, and a high-precision target three-dimensional feature set is output.

[0079] S503 inputs a high-precision target 3D feature set into a spatiotemporal consistency filter, performs consistency verification on the matching results of multiple consecutive frames of images, eliminates mismatched point clouds, and outputs a high-density initial visual point cloud covering the complete outline of the target.

[0080] S504, perform point cloud correction on the high-density initial visual point cloud. Based on the distribution law of the difference between laser depth and visual depth, select effective seed points in the high-density initial visual point cloud. Correct the effective seed points using the confidence of laser point cloud and image point cloud to obtain corrected seed points. Calculate the normal vector of the neighborhood of the corrected seed point. If its normal vector deviation is greater than a preset deviation value, readjust the weights of the confidence of laser point cloud and image point cloud, continue iterative correction, and output the high-density corrected point cloud.

[0081] S505, interpolation is performed on the high-density corrected point cloud to complete it. The high-density corrected point cloud is divided into flat regions and edge regions according to the curvature of the target surface. In the flat regions, the depth of non-seed points in the flat regions is completed by bilinear interpolation based on the depth of adjacent correction seed points. In the edge regions, a continuous radial basis function is constructed with the correction seed points as the center, and a continuous surface covering the entire edge region is fitted. The interpolation density is adaptively adjusted in combination with the curvature change to complete the high-precision interpolation of the edge regions. A complete preliminary fused point cloud covering the target surface is generated.

[0082] S506 performs spatiotemporal joint optimization on the preliminary fused point cloud. It uses the motion consistency constraint of the point cloud between consecutive frames to compensate for the dynamic displacement of the target. If the depth difference of the point cloud at the same position in adjacent frames exceeds the preset threshold, the interpolation parameters of S505 are adjusted based on the depth of the correction seed point in the current frame. The surface fitting result is re-optimized to obtain the optimized fused point cloud and output the third target detection result.

[0083] The S600 locks onto the target and continuously tracks it, enabling automatic tracking and measurement of small targets at long distances.

[0084] This invention employs multi-source information for target tracking and measurement, but equipment failures or errors may occur during operation. To eliminate faulty measurement equipment or faulty measurement information, maximize the utilization of available information sources, minimize system performance degradation after a failure or anomaly, and provide protection against transient faults, thereby improving the reliability of the entire measurement system, this invention can also design a redundant measurement fault-tolerant module. In some embodiments, such as... Figure 15 As shown, the target locking and continuous tracking S600 may include redundancy measurement fault tolerance processing S610, which generally includes steps S611-S617.

[0085] S611, using the point cloud coordinates and depth data in the first target detection result as the first redundancy information, and the point cloud coordinates and depth data in the second target detection result as the second redundancy information, to confirm whether the visible light camera group and the lidar group have completed satellite positioning and timing synchronization and three-dimensional coordinate system fusion calibration.

[0086] S612 performs self-testing on the visible light camera group and the lidar group. It performs self-testing on the camera's frame rate, exposure parameters, and interface communication, and on the lidar's echo intensity, point cloud density, and operating voltage. If any hardware parameter exceeds the standard threshold, the margin is marked as a device malfunction.

[0087] S613 performs signal self-testing on the visible light camera group and lidar group, including detection range verification, frame rate timing verification and noise verification. If any signal parameter exceeds the standard threshold, the margin is marked as a signal abnormality.

[0088] S614 performs dual-redundancy cross-consistency verification on the visible light camera group and the lidar group, including spatial deviation calculation and consistency determination. By calculating the Euclidean distance deviation of the same target using the first redundancy information and the second redundancy information at the same time, if the Euclidean distance deviation is less than the preset dynamic deviation threshold adjusted according to the target distance, the spatial consistency is determined to meet the requirements; otherwise, a secondary verification is initiated, and the grayscale features of the marker points of the first redundancy information are extracted and matched with the depth contour gradient features of the second redundancy information. If the feature matching degree is greater than the preset matching threshold, the data drift is determined to be correctable; otherwise, it is determined to be a single redundancy fault.

[0089] S615: When any redundancy is determined to be an equipment malfunction, signal malfunction, or single redundancy failure, its data output is immediately cut off, and only the data channels of normal redundancy and correctable redundancy are retained as effective redundancy.

[0090] S616 performs fault-tolerant reconstruction processing on the retained normal redundancy and correctable redundancy. It calculates the confidence weight of each effective redundancy by using the real-time self-test pass rate, mutual test consistency and environment adaptation coefficient. It also normalizes the redundancy confidence weights of the first redundancy information and the second redundancy information in the same time series to obtain the confidence level of each redundancy.

[0091] S617: Based on the status of the first and second redundancy information in the same time sequence, signal voting is performed. When both are valid redundancies, the final third target detection result is output through weighted fusion. When only one is a valid redundancy, the corresponding target detection data is directly output. If both redundancies fail, the emergency mode is activated, the most recent historical valid fusion data is called and combined with the inertial navigation calculation result to output the predicted third target detection result.

[0092] In some embodiments, the target locking and continuous tracking S600 further includes a target confirmation processing step S620. The target confirmation processing step S620 includes S621-S622, etc. S621: Initiate a target confirmation request to confirm the target in the third target detection result; S622: After receiving the target confirmation instruction, lock the target, activate the multi-sensor collaborative tracking mode, track the target in real time, and complete the automatic tracking and locking measurement of a small target at a long distance. In some embodiments, the target confirmation processing can be manual confirmation or automatic target confirmation by the system according to a preset intelligent confirmation strategy. When using an intelligent confirmation strategy, the system performs a comprehensive score based on indicators such as the continuity of the target's motion trajectory, the saliency of its features, and the degree of background interference.

[0093] It should be noted that the above steps S100-S622 are not intended to limit the order of the processing flow, but are merely used to illustrate several aspects of the processing flow of the present invention and the locking method. Some of these steps do have a logical order, but some processes can be processed sequentially or simultaneously. This is something that those skilled in the art can understand or adjust according to actual needs.

[0094] If the functions described in this application are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium.

[0095] This application proposes a computer-readable storage medium. The storage medium is used to store a computer program, and when a computer reads the computer program from the storage medium, the computer performs the operations corresponding to the locking bit method described above.

[0096] Computer-readable storage media may contain a propagated data signal encoded with a computer program, for example, on baseband or as part of a carrier wave. This propagated signal may take various forms, including electromagnetic, optical, or suitable combinations thereof. The storage medium may be, but is not limited to: floppy disks, optical disks, hard disks, USB flash drives, TF cards (T-Flash Card, also known as MicroSD cards), SD cards (Secure Digital Memory Cards), MMC cards (Multi Media Cards), SM cards (Smart Media Cards), Memory Sticks, XD cards, CF cards (Compact Flash Cards), etc.

[0097] This medium can be connected to an instruction execution system, apparatus, or device to enable communication, propagation, or transmission of a program for use. Program code located on a computer storage medium can be propagated through any suitable medium, including radio, cable, fiber optic cable, RF, or similar media, or any combination of the above media.

[0098] The computer program code required for the operation of each part of this application can be written in any one or more programming languages, including object-oriented programming languages ​​such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB.NET, Python, etc., conventional procedural programming languages ​​such as C, Visual Basic, Fortran 2003, Perl, COBOL 2002, PHP, ABAP, dynamic programming languages ​​such as Python, Ruby, and Groovy, or other programming languages. This program code can run entirely on the user's computer, or as a standalone software package on the user's computer, or partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any network, such as a local area network (LAN) or wide area network (WAN), or connected to an external computer (e.g., via the Internet), or in a cloud computing environment, or used as a service such as Software as a Service (SaaS).

[0099] The basic concepts have been described above. Obviously, for those skilled in the art, the detailed disclosure above is merely illustrative and does not constitute a limitation of this application. Although not explicitly stated herein, those skilled in the art may make various modifications, improvements, and corrections to this application. Such modifications, improvements, and corrections are suggested in this application, and therefore remain within the spirit and scope of the exemplary embodiments of this application.

[0100] Furthermore, this application uses specific terms to describe its embodiments. For example, "one embodiment" or "some embodiments" refers to a particular feature, structure, or characteristic related to at least one embodiment of this application. Therefore, it should be emphasized and noted that "some embodiments" or "one embodiment" mentioned twice or more in different locations in this specification do not necessarily refer to the same embodiment. In addition, certain features, structures, or characteristics in one or more embodiments of this application can be appropriately combined.

[0101] As indicated in this application and claims, unless the context clearly indicates otherwise, the terms "a," "first," "second," and / or "the" are not specifically singular in quantity, but rather descriptive terms used for distinction and classification. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0102] The above descriptions are merely various embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An automatic detection, tracking, and positioning method for aerial targets, characterized in that: The tracking and locking method includes the following steps: S100: Acquire aerial targets in the same scene by a visible light camera group and a lidar group respectively; obtain 2D images of multiple cameras through the visible light camera group and a second point cloud image through the lidar group. S200, perform long-distance small target detection and preliminary localization on the 2D image, collect the first feature of the target in real time, if there is no preset cooperative identification feature in the first feature of the target, then perform small target feature detection on the 2D image; otherwise, perform cooperative identification point detection on the 2D image and output the first target detection result. The small target feature detection mainly includes S211~S214; S211, based on block matching, the inverse synthesis search divides the current frame image of the 2D image into non-overlapping image blocks of fixed size, with each image block serving as a reference block to be matched; according to a certain range of search windows set for each reference block in the next frame image, the similarity between the reference block and all candidate blocks within the search window is calculated, and the candidate block with the highest similarity is selected as the matching block; the displacement vector of the matching block relative to the reference block is recorded to construct a sparse optical flow field; S212, perform multi-scale pyramid reconstruction on the sparse optical flow field, and downsample the current frame image and the next frame image at multiple scales to form image sequences at different scales; interpolate and supplement missing optical flow points at each scale layer, upsample layer by layer and fuse the optical flow information of adjacent scale layers to obtain the first dense optical flow field. S213, perform variational optimization on the first dense optical flow field, utilize global smoothness constraints and local brightness constant constraints, and correct discontinuous regions in the optical flow field by minimizing the energy function. Continuously adjust the optical flow vector of each pixel until the energy function converges to the minimum value to obtain the optimized second dense optical flow field and capture the motion trajectory of small targets. S214, if there is a significant motion trajectory in the second dense optical flow field output by S223 and the preset small target feature template matching threshold is met, then it is determined that a small target exists and the first target detection result is output; otherwise, it is determined that there is no valid target and the process returns to continue small target feature detection. S300 treats small targets as a single point when they are at a distance, performing single-point visual intersection measurement and localization; when the target enters the mid-range range, it switches to cooperative marker point detection, measuring the target pose through multi-point target pose measurement and localization. S400, when a target enters the monitoring range of the lidar group, perform fast point cloud target detection based on grid map, establish grid map by rasterizing the second point cloud image, extract the target size and shape according to the spatial distribution characteristics of the target in the grid map, perform target detection and localization, and output the second target detection result; S500, based on the second target detection result and the first target detection result, perform multi-sensor data fusion detection to obtain the third target detection result; The S600 locks onto the target and continuously tracks it, enabling automatic tracking and measurement of small targets at long distances.

2. The automatic detection and tracking method for air targets according to claim 1, characterized in that: The cooperative marker detection S220 mainly includes S221~S224; S221, Perform ROI extraction, binarize the 2D image, extract areas with brightness higher than the threshold as candidate ROIs, match the candidate ROIs with the preset ROI template, and filter out ROIs that meet the geometric shape and size characteristics. S222, perform anisotropic diffusion filtering on the ROI region of the marker point, calculate the gray-level gradient value of the 8 neighboring pixels of each pixel in the ROI region of the marker point, dynamically adjust the diffusion coefficient according to the magnitude of the gray-level gradient value, and output the marker point filtered image. S223, the gradient operator is used to calculate the gradient magnitude and direction of the marker point filtered image, and edge points with gradient magnitude greater than the preset gradient threshold are selected according to the preset gradient threshold. Adjacent discrete edge points are connected to form continuous edges through 8-neighborhood connectivity analysis, and the pixel-level contour of the marker point is extracted. S224, taking each pixel-level edge point in the pixel-level contour of the marker point as the center, set a preset local window, calculate the Zernike moment of the image within the local window, extract the gray-level distribution features of the edge points, and calculate the precise coordinates of the edge points within the pixel based on the quantitative relationship between the Zernike moment and the edge position to obtain the sub-pixel edge point coordinates. Perform set fitting on all sub-pixel edge points to calculate the sub-pixel contour of the marker point or the sub-pixel center coordinates of the marker point. S225, the sub-pixel-level contour or sub-pixel-level center coordinates of the marker point are transformed with the preset three-dimensional spatial distribution coordinate system of the marker point to obtain the real coordinate system of the target marker point, reconstruct the model of all the real marker points of the target, and calculate the Euclidean distance between the centroids of each marker point. S226, find the two markers with the smallest centroid distance, take one of the two markers as the initial starting point and the other as the second point to form the initial chain code, and connect the remaining markers in sequence according to the minimum distance principle until the last point and the starting point form the final chain code, thereby generating the closed chain code; S227: Calculate the first difference of the closed chain code of the real marker model to obtain the differential chain code. Match and compare the differential chain code of the real marker model with the differential chain code of the preset marker model. If the two differential chain codes are the same, it is determined that the measured marker and the marker model are successfully matched, and the first target detection result is output. Otherwise, change the initial starting point and re-execute S226 to reconstruct the chain code until the differential chain code is successfully matched.

3. The automatic detection and tracking method for air targets according to claim 1, characterized in that: The single-point target visual intersection measurement and localization includes the following steps: S311, synchronously image the target using each camera in the visible light camera group. When the target is detected in the first target detection result output by each camera in the visible light camera group, the target sub-pixel level center point in each 2D image is calculated using the gray-scale centroid method. S312: Based on the sub-pixel-level center point of the target in each 2D image, and combined with the intrinsic and extrinsic parameter matrices of each camera, a set of visual intersection equations is constructed, and the three-dimensional coordinates of the target in the world coordinate system are obtained by solving the equations.

4. The automatic detection and tracking method for air targets according to claim 2, characterized in that: The target marker model pose measurement and positioning includes the following steps: S321, based on the successfully matched marker point model in S227, matches the measured marker points of each camera in the visible light camera group one-to-one, calculates the marker point coordinates of the measured target, and obtains the measured marker point cloud. S322 performs spatial rigid body transformation registration between the measured marker cloud and the preset marker model, uses singular value decomposition to solve the rotation and translation matrices, completes target pose estimation, outputs the three-dimensional attitude angles and position parameters of the target in the world coordinate system, and completes the pose calculation of the target marker model.

5. The automatic detection and tracking method for air targets according to claim 1, characterized in that: The target marker model pose measurement and localization also includes a feature point-based tracking acceleration measurement process. When performing cooperative marker detection on the 2D image, global feature point detection is first performed on the 2D image acquired by one camera in the visible light camera group. The measured marker points and the target marker model are extracted and matched. After determining the correspondence of successfully matched marker points, subsequent time series images and images from other cameras in the visible light camera group are only used for local feature search and marker point relationship matching in the corresponding areas.

6. The automatic detection and tracking method for aerial targets according to claim 1, characterized in that: The fast point cloud target detection based on raster maps includes the following steps: S401, Establish a grid map, divide the lidar detection area into multiple uniform three-dimensional grid units, and discretize the point cloud data of the second point cloud image. S402 transforms the point cloud coordinate system of the discretized point cloud data to the raster map coordinate system, performs point-by-point traversal, determines the raster cell to which each point belongs, and counts the number and distribution characteristics of point clouds in each raster, outputting a raster grayscale map. S403, based on the distribution characteristics and grayscale information of the point cloud, the raster grayscale map is denoised, and the distribution characteristic template of the raster image of the preset target model is matched to filter out the candidate raster area where the suspected target is located. S404 performs clustering and segmentation on the point cloud data within the candidate grid area, extracts the target point cloud cluster, calculates the centroid coordinates of all points in the target point cloud cluster, obtains the target location information, completes target detection and localization, and outputs the second target detection result.

7. The automatic detection and tracking method for air targets according to claim 1, characterized in that: The multi-sensor data fusion detection includes the following steps: S501, synchronously acquires the first target detection result output by the visible light camera group and the second target detection result output by the lidar group; S502, the laser point cloud of the second target detection result is projected onto the camera image plane through the calibrated transformation matrix, and the image pixel coordinates corresponding to each laser point cloud are calculated; the sub-pixel level contour and grayscale features of the marker points in the first target detection result are extracted, and the image pixel coordinates are back-projected onto the three-dimensional space of the laser point cloud through the extrinsic matrix of the visible light camera group to obtain the initial visual ray; with grayscale similarity and spatial distance threshold as constraints, the intersection points of the laser point cloud and the initial visual ray are matched to determine the one-to-one correspondence between the laser point cloud and the image pixels, the successfully matched point clouds are fused, and a high-precision target three-dimensional feature set is output; S503 inputs the high-precision target 3D feature set into the spatiotemporal consistency filter, performs consistency verification on the matching results of multiple consecutive frames of images, removes mismatched point clouds, and outputs a high-density initial visual point cloud covering the complete outline of the target. S504, perform point cloud correction on the high-density initial visual point cloud. Based on the distribution law of the difference between laser depth and visual depth, select effective seed points in the high-density initial visual point cloud. Correct the effective seed points using the confidence of laser point cloud and image point cloud to obtain corrected seed points. Calculate the normal vector of the neighborhood of the corrected seed point. If its normal vector deviation is greater than a preset deviation value, readjust the weights of the confidence of laser point cloud and image point cloud, continue iterative correction, and output the high-density corrected point cloud. S505, interpolation is performed on the high-density corrected point cloud to complete it. The high-density corrected point cloud is divided into flat regions and edge regions according to the curvature of the target surface. In the flat regions, the depth of non-seed points in the flat regions is completed by bilinear interpolation based on the depth of adjacent correction seed points. In the edge regions, a continuous radial basis function is constructed with the correction seed points as the center, and a continuous surface covering the entire edge region is fitted. The interpolation density is adaptively adjusted in combination with the curvature change to complete the high-precision interpolation of the edge regions. A complete preliminary fused point cloud covering the target surface is generated. S506 performs spatiotemporal joint optimization on the preliminary fused point cloud. It uses the motion consistency constraint of the point cloud between consecutive frames to compensate for the dynamic displacement of the target. If the depth difference of the point cloud at the same position in adjacent frames exceeds the preset threshold, the interpolation parameters of S505 are adjusted based on the depth of the correction seed point in the current frame. The surface fitting result is re-optimized to obtain the optimized fused point cloud and output the third target detection result.

8. The automatic detection and tracking method for air targets according to claim 1, characterized in that: The continuous target tracking S600 includes a redundancy measurement fault-tolerant processing S610: S611, using the point cloud coordinates and depth data in the first target detection result as the first redundancy information, and the point cloud coordinates and depth data in the second target detection result as the second redundancy information, to confirm whether the visible light camera group and the lidar group have completed satellite positioning and timing synchronization and three-dimensional coordinate system fusion calibration; S612 performs device self-testing on the visible light camera group and the lidar group. It performs self-testing on the camera's frame rate, exposure parameters, and interface communication, and on the lidar's echo intensity, point cloud density, and operating voltage. If any hardware parameter exceeds the standard threshold, the margin is marked as a device malfunction. S613 performs signal self-testing on the visible light camera group and lidar group, including detection range verification, frame rate timing verification and noise verification. If any signal parameter exceeds the standard threshold, the margin is marked as a signal abnormality. S614 performs dual-redundancy cross-consistency verification on the visible light camera group and the lidar group, including spatial deviation calculation and consistency determination; by calculating the Euclidean distance deviation of the first redundancy information and the second redundancy information at the same time for the same target, if the Euclidean distance deviation is less than the preset dynamic deviation threshold adjusted according to the target distance, the spatial consistency is determined to meet the requirements; otherwise, a secondary verification is initiated, extracting the grayscale features of the marker points of the first redundancy information and matching them with the depth contour gradient features of the second redundancy information. If the feature matching degree is greater than the preset matching threshold, the data drift is determined to be correctable; otherwise, it is determined to be a single redundancy fault. S615: When any redundancy is determined to be an equipment malfunction, signal malfunction, or single redundancy failure, its data output is immediately cut off, and only the data channels of normal redundancy and correctable redundancy are retained as effective redundancy. S616 performs fault-tolerant reconstruction processing on the retained normal redundancy and correctable redundancy. It calculates the confidence weight of each effective redundancy by using the real-time self-test pass rate, mutual test consistency and environment adaptation coefficient. It also normalizes the redundancy confidence weights of the first redundancy information and the second redundancy information in the same time series to obtain the confidence level of each redundancy. S617: Based on the status of the first and second redundancy information in the same time sequence, signal voting is performed. When both are valid redundancies, the final third target detection result is output through weighted fusion. When only one is a valid redundancy, the corresponding target detection data is directly output. If both redundancies fail, the emergency mode is activated, the most recent historical valid fusion data is called and combined with the inertial navigation calculation result to output the predicted third target detection result.

9. The automatic detection and tracking method for air targets according to claim 1, characterized in that: The continuous target tracking S600 includes target confirmation processing S622: S621, Initiate a target confirmation request to confirm the target in the third target detection result; After receiving the target confirmation command, the S622 locks onto the target, activates the multi-sensor collaborative tracking mode, tracks the target in real time, and completes automatic tracking and measurement of small targets at long distances.

Images