An unmanned aerial vehicle multi-modal fusion detection method and system based on space-time projection and asynchronous trajectory matching

Abstract

The application provides a kind of unmanned aerial vehicle multimodal fusion detection method and system based on space-time projection and asynchronous trajectory matching, through the dimension reduction projection based on three-dimensional position, dynamic attitude of cloud platform and camera internal parameter, radar / radio target is accurately mapped to image plane, a unified trajectory measurement space is established, and the problem of missing heterogeneous space correlation benchmark is solved.The application realizes the unity of radar / radio and photoelectric heterogeneous sensor in spatial dimension, overcomes the data asynchronous problem caused by sampling rate difference, and can realize accurate identity correlation of target in high dynamic scene.Through full-dimensional information fusion display, the defect that monocular vision cannot measure distance is made up, and the problem that radar target is difficult to intuitively confirm is solved, which significantly enhances the perception ability and detection reliability of "low, slow and small" target.

Description

Technical Field

[0001] This invention relates to the field of low-altitude security defense and multi-sensor data fusion technology, specifically to a UAV multimodal fusion detection method and system based on spatiotemporal projection and asynchronous trajectory matching. Background Technology

[0002] With the increasing prevalence of consumer and industrial drones, low-altitude security threats are becoming increasingly severe. In counter-drone systems, single sensors often have limitations: radar or radio monitoring equipment can provide long-range latitude, longitude, altitude, and speed information of targets, but lacks visual features, making it difficult to obtain evidence and susceptible to electromagnetic interference that could generate false alarms; optoelectronic systems (such as high-magnification cameras) can provide intuitive images and high-precision angle information, but lack a sense of distance and have a narrow field of view (FOV), making it difficult to independently detect targets over a wide airspace.

[0003] Therefore, multi-sensor fusion is an inevitable trend in industry development. However, existing fusion technologies face the following challenges in practical applications: 1. Difficulty in mapping heterogeneous spaces: Radar / radio outputs sparse point traces in the geodetic coordinate system (WGS-84), while photoelectric tracking outputs two-dimensional bounding boxes in the image pixel coordinate system. The two have different dimensions and lack a unified reference for association.

[0004] 2. Data Asynchronization and Sampling Rate Differences: Radar / radio typically have slow scanning cycles (e.g., 1Hz-5Hz), while photoelectric tracking algorithms (such as the YOLO series) usually operate at high frame rates (25Hz-60Hz). Traditional "nearest neighbor matching" or "Kalman filter synchronization" methods are prone to matching errors or fusion delays when facing highly maneuverable targets such as UAVs due to incomplete timestamp alignment.

[0005] 3. Limitations of existing algorithms: Most existing trajectory matching algorithms are based on two-dimensional moving targets, while UAVs have three-dimensional high-dynamic flight characteristics. Directly applying two-dimensional matching algorithms will lead to association failure when the pitch angle changes drastically.

[0006] Therefore, there is an urgent need for a UAV multimodal fusion detection method that can adapt to three-dimensional high dynamic scenes and effectively solve the problem of asynchronous data timing alignment. Summary of the Invention

[0007] This invention addresses the problems in existing low-altitude anti-UAV systems, such as insufficient detection capability of single sensors, inconsistent spatial dimensions of heterogeneous sensors (radar / radio and electro-optical), and misalignment of high-dynamic target trajectories due to asynchronous sampling rates. It provides a multi-modal fusion detection method for UAVs based on spatiotemporal projection and asynchronous trajectory matching. This invention aims to achieve precise fusion of radar / radio detection data and electro-optical tracking video streams by constructing a unified projection plane and a flexible temporal matching algorithm, thus solving the problems of identity locking and full-dimensional information enhancement for "low, slow, and small" targets in three-dimensional high-dynamic flight scenarios and asynchronous sensor data transmission.

[0008] To achieve the above objectives, this invention provides a UAV multimodal fusion detection method based on spatiotemporal projection and asynchronous trajectory matching, comprising the following steps: S1. Acquire target data reported by radar or radio, and map it onto the image pixel plane through spatial projection operation to obtain radar / radio projection points; S2. Based on the radar / radio projection points and the center points of the visual detection boxes generated by visual detection in the photoelectric tracking video stream, construct radar / radio projection trajectory sequences and visual detection trajectory sequences based on sliding time windows, respectively. S3. Based on the radar / radio projection trajectory sequence and the visual detection trajectory sequence, the similarity distance between the two is calculated using an asynchronous trajectory multimodal matching algorithm; S4. Based on the similarity distance, construct the association cost matrix and perform global optimal allocation to determine the matching relationship between radar / radio targets and visual targets.

[0009] Preferably, S1 includes: Obtain the sphere-centered, geofixed coordinates of radar or radio targets in the WGS-84 geodetic coordinate system; Based on the coordinate transformation matrix, the sphere-centered geofixed coordinates are converted to coordinates in the local northeastern sky coordinate system with the photoelectric observation station as the origin. Based on the azimuth and pitch angles of the gimbal of the optoelectronic device, and combined with the rotation matrix and translation vector, the coordinates in the northeast-sky coordinate system are converted to the coordinates in the camera coordinate system. Based on the camera's intrinsic parameter matrix, the three-dimensional coordinates in the camera coordinate system are mapped to the two-dimensional image pixel plane through perspective projection to obtain the radar / radio projection points.

[0010] Preferably, S2 includes: Set a time window and maintain two queues in memory; Extract the historical projection point sequence of the same radar / radio ID within the time window as the radar / radio projection trajectory sequence; Extract the sequence of center points of historical visual detection boxes with the same visual tracking ID within the time window, and use it as the visual detection trajectory sequence.

[0011] Preferably, S3 includes: Based on the projection points in the radar / radio projection trajectory sequence and the center points of the visual detection boxes in the visual detection trajectory sequence, a distance matrix is ​​constructed, and the local distance cost between the two is calculated. Based on the distance matrix, the optimal regular path is found using a coarse-to-fine search strategy, and the cumulative minimum matching cost is calculated. The final similarity distance is obtained by normalizing the cumulative minimum matching cost and the lengths of the two sequences.

[0012] Preferably, the method for calculating the local distance cost between the two includes: using Euclidean distance to measure the spatial deviation between the projection point in the radar / radio projection trajectory sequence and the center point of the visual detection box in the visual detection trajectory sequence as the local distance cost.

[0013] Preferably, the cumulative minimum matching cost is calculated, which is achieved through the following recursive relationship: Based on the local distance cost of the current point, and combined with the cumulative path cost in three cases—radar / radio point corresponding to multiple visual frames, visual frame corresponding to multiple radar / radio points, and synchronous matching of the two—the minimum cumulative path cost from the starting point of the sequence to the current position is recursively calculated.

[0014] Preferably, S4 includes: Based on the similarity distance between all radar / radio trajectories and all visual trajectories, construct an M-row N-dimensional association cost matrix, where M is the number of radar / radio trajectories and N is the number of visual trajectories; Apply threshold filtering to the association cost matrix, setting elements greater than a preset threshold to infinity; The Hungarian algorithm is applied to solve the correlation cost matrix after threshold filtering to obtain the optimal matching Boolean matrix, thereby determining the matching relationship between radar / radio targets and visual targets.

[0015] This invention also provides a UAV multimodal fusion detection system based on spatiotemporal projection and asynchronous trajectory matching, the system being used to implement the above method, comprising: The acquisition module is used to acquire target data reported by radar or radio, and map it onto the image pixel plane through spatial projection operations to obtain radar / radio projection points. The module is used to construct radar / radio projection trajectory sequences and visual detection trajectory sequences based on sliding time windows, respectively, based on radar / radio projection points and the center points of visual detection boxes generated by visual detection in photoelectric tracking video streams. The calculation module is used to calculate the similarity distance between radar / radio projection trajectory sequences and visual detection trajectory sequences using an asynchronous trajectory multimodal matching algorithm; The matching module is used to construct an association cost matrix based on similarity distance and perform a globally optimal allocation to determine the matching relationship between radar / radio targets and visual targets.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. Spatiotemporal alignment and heterogeneous projection: Instead of simple homography matrix projection, it is a method that uses the three-dimensional position (longitude, latitude, altitude) detected by radar / radio, combined with the real-time dynamic attitude (azimuth angle, pitch angle) of the electro-optical gimbal and camera intrinsic parameters to "reduced the dimension" of the three-dimensional target to the two-dimensional image pixel plane, and establishes a unified trajectory measurement space.

[0017] 2. Asynchronous trajectory sequence construction: To address the issue of significant differences in sampling rates among heterogeneous sensors, instead of relying on strict alignment of single-frame timestamps, a method is constructed based on sliding time windows to create "sparse radar / radio projection sequences" and "dense visual detection sequences".

[0018] 3. ATMM Asynchronous Matching: This method uses the Asynchronous Trajectory Multimodal Matching (ATMM) algorithm to find the optimal temporal regularization path between two sequences with inconsistent sampling rates, calculates the nonlinear similarity distance, and completes the ID association of the target based on this distance.

[0019] 4. Full-dimensional information fusion display: In the video stream, by utilizing the successfully matched correlations, the radar / radio-specific data such as latitude, longitude, altitude, speed, and heading are overlaid and displayed in real time below the vision of the optical camera, forming a fused target object with full-dimensional perception data. Attached Figure Description

[0020] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention; Figure 2 This is a schematic diagram of heterogeneous coordinate system transformation and projection geometry according to an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the principle of ATMM according to an embodiment of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0024] like Figure 1 The diagram shown is a schematic representation of the method flow in this embodiment, and the steps include: S1. Acquire target data reported by radar or radio, and map it onto the image pixel plane through spatial projection operation to obtain radar / radio projection points.

[0025] To measure the similarity of heterogeneous sensors in a unified spatial dimension, a spatial projection operation is first performed. For example... Figure 2 The diagram shown is a schematic representation of the heterogeneous coordinate system transformation and projection geometry in this embodiment. This diagram visually illustrates the transformation of radar electronic target points in the three-dimensional world. The process involves converting the target's coordinates to a local coordinate system with the camera as the origin, and finally mapping it onto a two-dimensional image plane via perspective projection. Real-time reception of target data reported by radar or radio is required, typically based on the WGS-84 geodetic coordinate system. First, the sphere-centered geodetic coordinates of the radar / radio target need to be converted to local tangent plane coordinates (such as the Northeast-Eastern Sky System, ENU) with the optoelectronic observation station as the origin. For the radar / radio detected... i There are 1 target, and its location is denoted as _____. Using coordinate transformation matrix Convert it to local coordinates .

[0026] Considering that the optoelectronic equipment is mounted on a rotatable pan-tilt-zoom (PTZ) unit, the system needs to further convert the ENU coordinates into camera coordinates by combining real-time pan-tilt-zoom attitude information. Assume the azimuth angle of the pan-tilt-zoom at this point is... α The pitch angle is β Then the rotation matrix Translation vector This transformation relationship has been determined. Next, perspective projection is performed using the camera's intrinsic parameter matrix to map the 3D points onto the 2D image pixel plane. The mathematical expression of this process is as follows: in, and This represents the x and y coordinates of the pixels after the radar / radio target is projected onto the image plane; This represents the depth value of the target in the camera coordinate system (normalization factor). This represents the camera's intrinsic parameter matrix, including focal length. and optical center coordinates ; Indicates the azimuth angle of the gimbal α and pitch angle β The determined rotation matrix; This represents the target's three-dimensional coordinates in the northeast-central coordinate system; This represents the translation vector of the camera relative to the reference origin. Through this process, the UAV target in physical space is virtually mapped onto the image, forming a reference point of the same dimension as the visual detection result.

[0027] S2. Based on the radar / radio projection points and the center points of the visual detection boxes generated by visual detection in the photoelectric tracking video stream, construct radar / radio projection trajectory sequences and visual detection trajectory sequences based on sliding time windows, respectively.

[0028] Because of the significant differences in sampling rates between radar / radio and cameras, and the inconsistent arrival times of their data at the processing unit, direct single-frame point-to-point matching is highly prone to errors. Therefore, this embodiment employs a trajectory sequence construction strategy based on a sliding time window. The system sets a time window (e.g., 10 seconds) and maintains two queues in memory.

[0029] For radar / radio data, extract the historical projection point sequence of the same radar / radio ID within the window, and denot it as sequence. .in, Represents a sequence of radar / radio projection trajectories; Indicates the first position in the sequence i The coordinates of the projection point at each moment ; N This indicates the number of radar / radio data points within the window (usually a small number).

[0030] For visual data, extract the historical center point sequence of the same visual tracking ID (generated by the visual detection and tracking algorithm) within the window, and denote it as the sequence. .in, Represents a sequence of visually detected trajectories; Indicates the first position in the sequence The center coordinates of the visual inspection box at each moment ; The number of frames representing visual data within the window (usually quite large, and...) ).

[0031] S3. Based on the radar / radio projection trajectory sequence and the visual detection trajectory sequence, the similarity distance between the two is calculated using an asynchronous trajectory multimodal matching algorithm.

[0032] After acquiring radar / radio projection sequences and visual trajectory sequence The core task then becomes calculating the similarity between the two sequences. To address the issue of unequal sequence lengths and nonlinear time distortion, this embodiment employs the Asynchronous Trajectory Multimodal Matching (ATMM) algorithm. For example... Figure 3 The diagram shown is a schematic of the principle of ATMM in this embodiment. This algorithm is an improvement on DTW and can measure the similarity of timing patterns while ensuring computational efficiency.

[0033] First, construct the distance matrix. Define the sequence. The first in Points and sequences The first in Local distance cost between points In this embodiment, Euclidean distance is used to measure this spatial deviation: in, Indicates the radar / radio sequence number Points and visual sequences Euclidean distance between points; Represents the pixel coordinates of the radar / radio projection point; This represents the pixel coordinates of the center of the visual inspection box.

[0034] Next, to handle the asynchronous nature of the time sequence, the algorithm searches for an optimal regularized path traversing the distance matrix. To avoid the high time consumption caused by full matrix computation, a coarse-to-fine search strategy is adopted, recursively calculating the cumulative minimum matching cost within the constraint window. : in, Indicates arrival from the beginning of the sequence. Minimum cumulative path cost of a location; This is the current local distance; This indicates the case where a radar / radio point corresponds to multiple visual frames (i.e., fast visual sampling). This indicates the case where a visual frame corresponds to multiple radar / radio points (i.e., visual frame loss). This indicates a situation where both are matched synchronously.

[0035] Finally, the final similarity distance between the two trajectory sequences is obtained through normalization. : in, Indicates radar / radio trajectory With visual trajectory The final matching distance (the smaller the value, the more similar); This represents the total cost of the regularized path; and These are the lengths of the two sequences, respectively.

[0036] S4. Based on the similarity distance, construct the association cost matrix and perform global optimal allocation to determine the matching relationship between radar / radio targets and visual targets.

[0037] After calculating the similarity distance between any radar / radio trajectory and any visual trajectory, the system needs to solve the global optimal allocation problem. Assume that in the current scene there are... Radar / radio tracks and A visual trajectory, constructing a dimensional correlation cost matrix Each element in the matrix That is, the result of the above calculation. .

[0038] To prevent false matches, the system first processes the matrix. Threshold filtering will be applied to values ​​exceeding a preset threshold. The elements are set to infinity. Then, the Hungarian algorithm is applied to solve the matrix: in, Represents the optimal matching Boolean matrix; For a binary variable, when Time indicates the first k The radar / radio target and the first l If a visual target is successfully associated, the value is 0; otherwise, the value is 0. Indicates the first k The radar / radio track and the first l The ATMM distance cost of a visual trajectory.

[0039] For successfully matched pairs The system executes the fusion output logic. Specifically, the system combines high-precision distance and altitude data detected by radar / radio. latitude and longitude In addition, speed information is overlaid on the video stream as an enhanced label. Below the corresponding visual detection box. This not only compensates for the inability of monocular vision to measure distance, but also solves the problem that radar / radio cannot intuitively confirm targets, thus achieving full-dimensional, highly reliable fusion detection of UAV targets.

[0040] Example 2 This embodiment also provides a UAV multimodal fusion detection system based on spatiotemporal projection and asynchronous trajectory matching, including: an acquisition module for acquiring target data reported by radar or radio, and mapping it to an image pixel plane through spatial projection operations to obtain radar / radio projection points; a construction module for constructing radar / radio projection trajectory sequences and visual detection trajectory sequences based on sliding time windows, respectively, according to the radar / radio projection points and the center points of visual detection boxes generated by visual detection in the photoelectric tracking video stream; a calculation module for calculating the similarity distance between the radar / radio projection trajectory sequences and the visual detection trajectory sequences using an asynchronous trajectory multimodal matching algorithm; and a matching module for constructing an association cost matrix and performing global optimal allocation based on the similarity distance to determine the matching relationship between radar / radio targets and visual targets.

[0041] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A multimodal fusion detection method for unmanned aerial vehicles (UAVs) based on spatiotemporal projection and asynchronous trajectory matching, characterized in that, Includes the following steps: S1. Acquire target data reported by radar or radio, and map it onto the image pixel plane through spatial projection operation to obtain radar / radio projection points; S2. Based on the radar / radio projection points and the center points of the visual detection boxes generated by visual detection in the photoelectric tracking video stream, construct radar / radio projection trajectory sequences and visual detection trajectory sequences based on sliding time windows, respectively. S3. Based on the radar / radio projection trajectory sequence and the visual detection trajectory sequence, the similarity distance between the two is calculated using an asynchronous trajectory multimodal matching algorithm; S4. Based on the similarity distance, construct the association cost matrix and perform global optimal allocation to determine the matching relationship between radar / radio targets and visual targets.

2. The UAV multimodal fusion detection method based on spatiotemporal projection and asynchronous trajectory matching according to claim 1, characterized in that, S1 includes: Obtain the sphere-centered, geofixed coordinates of radar or radio targets in the WGS-84 geodetic coordinate system; Based on the coordinate transformation matrix, the sphere-centered geofixed coordinates are converted to coordinates in the local northeastern sky coordinate system with the photoelectric observation station as the origin. Based on the azimuth and pitch angles of the gimbal of the optoelectronic device, and combined with the rotation matrix and translation vector, the coordinates in the northeast-sky coordinate system are converted to the coordinates in the camera coordinate system. Based on the camera's intrinsic parameter matrix, the three-dimensional coordinates in the camera coordinate system are mapped to the two-dimensional image pixel plane through perspective projection to obtain the radar / radio projection points.

3. The UAV multimodal fusion detection method based on spatiotemporal projection and asynchronous trajectory matching according to claim 1, characterized in that, S2 includes: Set a time window and maintain two queues in memory; Extract the historical projection point sequence of the same radar / radio ID within the time window as the radar / radio projection trajectory sequence; Extract the sequence of center points of historical visual detection boxes with the same visual tracking ID within the time window, and use it as the visual detection trajectory sequence.

4. The UAV multimodal fusion detection method based on spatiotemporal projection and asynchronous trajectory matching according to claim 1, characterized in that, S3 includes: Based on the projection points in the radar / radio projection trajectory sequence and the center points of the visual detection boxes in the visual detection trajectory sequence, a distance matrix is ​​constructed, and the local distance cost between the two is calculated. Based on the distance matrix, the optimal regular path is found using a coarse-to-fine search strategy, and the cumulative minimum matching cost is calculated. The final similarity distance is obtained by normalizing the cumulative minimum matching cost and the lengths of the two sequences.

5. The UAV multimodal fusion detection method based on spatiotemporal projection and asynchronous trajectory matching according to claim 4, characterized in that, Methods for calculating the local distance cost between the two include using Euclidean distance to measure the spatial deviation between the projected points in the radar / radio projection trajectory sequence and the center point of the visual detection box in the visual detection trajectory sequence as the local distance cost.

6. The UAV multimodal fusion detection method based on spatiotemporal projection and asynchronous trajectory matching according to claim 4, characterized in that, The cumulative minimum matching cost is calculated using the following recursive relationship: Based on the local distance cost of the current point, and combined with the cumulative path cost in three cases—radar / radio point corresponding to multiple visual frames, visual frame corresponding to multiple radar / radio points, and synchronous matching of the two—the minimum cumulative path cost from the starting point of the sequence to the current position is recursively calculated.

7. The UAV multimodal fusion detection method based on spatiotemporal projection and asynchronous trajectory matching according to claim 1, characterized in that, S4 includes: Based on the similarity distance between all radar / radio trajectories and all visual trajectories, construct an M-row N-dimensional association cost matrix, where M is the number of radar / radio trajectories and N is the number of visual trajectories; Apply threshold filtering to the association cost matrix, setting elements greater than a preset threshold to infinity; The Hungarian algorithm is applied to solve the correlation cost matrix after threshold filtering to obtain the optimal matching Boolean matrix, thereby determining the matching relationship between radar / radio targets and visual targets.

8. A UAV multimodal fusion detection system based on spatiotemporal projection and asynchronous trajectory matching, the system being used to implement the method described in any one of claims 1-7, characterized in that, include: The acquisition module is used to acquire target data reported by radar or radio, and map it onto the image pixel plane through spatial projection operations to obtain radar / radio projection points. The module is used to construct radar / radio projection trajectory sequences and visual detection trajectory sequences based on sliding time windows, respectively, based on radar / radio projection points and the center points of visual detection boxes generated by visual detection in photoelectric tracking video streams. The calculation module is used to calculate the similarity distance between radar / radio projection trajectory sequences and visual detection trajectory sequences using an asynchronous trajectory multimodal matching algorithm; The matching module is used to construct an association cost matrix based on similarity distance and perform a globally optimal allocation to determine the matching relationship between radar / radio targets and visual targets.

Images