Unmanned aerial vehicle target detection and tracking method and device based on thunder-vision fusion, and storage medium

Abstract

The invention discloses an unmanned aerial vehicle target detection and tracking method based on thunder-vision fusion, and the method comprises the following steps: obtaining camera measurement data obtained through a camera and radar measurement data obtained through a radar, and carrying out the alignment of the camera measurement data and the radar measurement data; a camera is used as an auxiliary sensor to screen radar detection; and fusing the radar measurement data and the camera measurement data, outputting the fused data to the tracker, and generating and updating the track of the target identified in the field of view of the sensor. According to the invention, through fusion of the double-level radar sensor and the visual sensor, the radar and visual data are preliminarily integrated and processed before the target is tracked to reduce noise and clutter, so that the calculation efficiency can be improved, the situation that the radar is submerged by environmental noise due to weak sensor signals during low-altitude flight can be prevented, and the reliability of flight is improved. Therefore, the detection accuracy is improved.

Description

Technical Field

[0001] The present invention relates to a method, device, and storage medium for detecting and tracking unmanned aerial vehicle (UAV) targets based on the fusion of radar and vision. Background Art

[0002] With the rapid development of UAV technology, the low-altitude space has gradually become a new productive resource. The application fields of UAVs have been continuously expanding, covering multiple fields such as precision agriculture, package delivery, aerial patrol, and emergency rescue. However, with the increase in the number of UAVs, the accuracy and autonomy of UAV missions have gradually attracted attention.

[0003] The perception of the surrounding environment by UAVs can be supported by information provided by systems such as the Traffic Alert and Collision Avoidance System (TCAS) and the Automatic Dependent Surveillance-Broadcast system (ADS-B). However, the use of these systems requires communication exchanges between aircraft or between the aircraft and the ground station. Therefore, it is only limited to the case of cooperative target detection and tracking. For the case of non-cooperative target detection and tracking, solutions based on visual or radar sensors for data processing must be adopted. However, their performance in detection is hindered in many aspects, especially in the case of low-altitude flight. Visual sensors will face the problem of low signal-to-background ratio. When the signal generated by the target is weak, it is difficult to distinguish it from the background. Radar sensors are difficult to detect targets with a very small radar cross-section. The reflected signals they generate are weak and are easily overwhelmed by environmental noise. At the same time, ground clutter will cause abnormal detection, which may generate false conflict alarms and affect the accuracy of detection. Therefore, multi-sensor fusion strategies have attracted much attention.

[0004] Currently, different fusion strategies have proven their effectiveness in surpassing the performance of individual visual and radar configurations, but there are still some limitations. Existing methods rely on the fusion of radar and visual data at the tracking level, requiring detections collected during the entire radar field-of-view scan, resulting in inevitable data delays. Secondly, the processing of noise and ground clutter needs to be carried out in the radar data preprocessing step, which may lead to the complexity of the processing process. At the same time, undetected clutter detections may increase the computational burden on the tracker and increase the probability of false track generation. Summary of the Invention

[0005] The main object of the present invention is to provide a method, device, and storage medium for detecting and tracking UAV targets based on the fusion of radar and vision, aiming to solve the above technical problems.

[0006] To achieve the above object, the present invention provides a method for detecting and tracking UAV targets based on the fusion of radar and vision.

[0007] The method for detecting and tracking UAV targets based on the fusion of radar and vision includes the following steps: Obtain the camera measurement data acquired by the camera and the radar measurement data acquired by the radar, and align the camera measurement data and the radar measurement data; Use the camera as an auxiliary sensor to screen the radar detections to ensure that both sensors have detected the same target; Fuse the radar measurement data and the camera measurement data, and output the fused data to the tracker to generate and update the trajectories of the targets recognized in the sensor's field of view.

[0008] In one embodiment, the step of obtaining the camera measurement data acquired by the camera and the radar measurement data acquired by the radar, and aligning the camera measurement data and the radar measurement data includes: Convert these data into unit vector observations in the NED frame, and then use the QUEST algorithm for alignment to determine the poses of the radar and the camera relative to the NED frame.

[0009] In one embodiment, the step of using the camera as an auxiliary sensor to screen the radar detections includes: For the time when the camera acquires an image , obtain all the radar data whose time tags fall within the time interval as a subset of radar measurements, and the radar data is the candidate measurement for the confirmation step, labeled as ; is the time interval between two consecutive frame acquisitions; Delete the radar echoes with all Doppler measurements lower than the preset threshold , and project the remaining radar beam directions onto the image plane.

[0010] In one embodiment, convert the radar data into pixel coordinates according to the following formula: ; ; ; where , and are the coordinates of the radar beam direction in the CRF respectively, , are the components of the camera focal length in the pixel, to are the radial and tangential distortion coefficients, , are the principal point coordinates.

[0011] In one embodiment, the step of generating and updating the trajectory of a target recognized within the sensor's field of view includes: Generating trajectories with different reliability levels, including one-time trajectories, temporary trajectories, and deterministic trajectories; After generating the deterministic trajectory, associating the visual measurement data with the deterministic trajectory.

[0012] In one embodiment, after the step of fusing the radar measurement data and the camera measurement data, outputting the fused data to the tracker, and generating and updating the trajectory of a target recognized within the sensor's field of view, the method further includes: After generating the deterministic trajectory, calculating the time of closest approach and the distance of closest approach ; If the time of closest approach and / or the distance of closest approach is / are such that, it is considered that there is a conflict; Then, generate and update the trajectory by adjusting the heading or speed.

[0013] In addition, to achieve the above object, the present invention further provides a method for detecting and tracking an unmanned aerial vehicle (UAV) target based on radar-vision fusion. The method for detecting and tracking an unmanned aerial vehicle (UAV) target based on radar-vision fusion includes: a memory, a processor, and a UAV target detection and tracking program stored on the memory and executable on the processor. When the UAV target detection and tracking program is executed by the processor, the steps of the method for detecting and tracking an unmanned aerial vehicle (UAV) target based on radar-vision fusion as described above are implemented.

[0014] In addition, to achieve the above object, the present invention further provides a computer-readable storage medium having a UAV target detection and tracking program stored thereon. When the UAV target detection and tracking program is executed by a processor, the steps of the method for detecting and tracking an unmanned aerial vehicle (UAV) target based on radar-vision fusion as described above are implemented.

[0015] A method for detecting and tracking an unmanned aerial vehicle (UAV) target based on radar-vision fusion proposed by an embodiment of the present invention includes obtaining camera measurement data acquired by a camera and radar measurement data acquired by a radar, and aligning the camera measurement data and the radar measurement data; Using the camera as an auxiliary sensor to screen the radar detection to ensure that both sensors have detected the same target; Fusing the radar measurement data and the camera measurement data, and outputting the fused data to the tracker to generate and update the trajectory of a target recognized within the sensor's field of view.

[0016] In this application, by adopting the fusion of a two - level radar sensor and a vision sensor, the radar and vision data are preliminarily integrated and processed before tracking the target to reduce noise and clutter, which can improve the calculation efficiency and prevent the situation where the sensor signals are weak during low - altitude flight and are overwhelmed by environmental noise, thereby improving the detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a schematic structural diagram of the device in the hardware operating environment related to the embodiment solution of the present invention; Figure 2 It is a schematic diagram of the radar / vision sensor fusion strategy in the present invention; Figure 3 It is a schematic diagram of the tracking process of the tracker based on the Extended Kalman Filter (EKF) in the present invention.

[0018] The realization, functional features and advantages of the object of the present invention will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0019] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0020] As Figure 1 shown, Figure 1 It is a schematic structural diagram of the terminal in the hardware operating environment related to the embodiment solution of the present invention.

[0021] The terminal in the embodiment of the present invention can be a drone, or a PC, a smart phone, a tablet computer, an e - book reader, an MP3 (Moving Picture Experts Group Audio Layer III) player, an MP4 (Moving Picture Experts Group Audio Layer IV) player, a portable computer and other movable terminal devices with a display function.

[0022] As Figure 1As shown in the figure, the terminal may include: a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display) and an input unit such as a keyboard (Keyboard). Optionally, the user interface 1003 may further include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory or a stable memory (non-volatile memory), such as a disk memory. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0023] Optionally, the terminal may further include a camera, an RF (Radio Frequency) circuit, sensors, an audio circuit, a WiFi module, etc. Among them, the sensors include, for example, a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor. Among them, the ambient light sensor can adjust the brightness of the display screen according to the brightness of the ambient light, and the proximity sensor can turn off the display screen and / or the backlight when the mobile terminal is moved to the ear. As a kind of motion sensor, the gravity acceleration sensor can detect the magnitude of acceleration in each direction (generally three axes). When stationary, it can detect the magnitude and direction of gravity and can be used for applications that identify the posture of the mobile terminal (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as a pedometer, tapping), etc.; of course, the mobile terminal can also be configured with other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, etc., which will not be elaborated here.

[0024] Those skilled in the art can understand that Figure 1 the terminal structure shown in

[0025] does not constitute a limitation on the terminal and may include more or fewer components than shown in the figure, or combine some components, or have different component arrangements. Figure 1 As shown in

[0026] In Figure 1In the terminal shown, the network interface 1004 is mainly used to connect to the background server and communicate data with the background server; the user interface 1003 is mainly used to connect to the client (user side) and communicate data with the client; and the processor 1001 can be used to call the drone target detection and tracking program stored in the memory 1005 and perform the following operations: Obtain the camera measurement data obtained through the camera and the radar measurement data obtained through the radar, and align the camera measurement data and the radar measurement data; Use the camera as an auxiliary sensor to screen the radar detection to ensure that both sensors have detected the same target; Fuse the radar measurement data and the camera measurement data, and output the fused data to the tracker to generate and update the trajectory of the target recognized in the sensor field of view.

[0027] Furthermore, the processor 1001 can call the drone target detection and tracking program stored in the memory 1005 and also perform the following operations: Convert these data into unit vector observations in the NED frame, and then use the QUEST algorithm for alignment to determine the attitudes of the radar and the camera relative to the NED frame.

[0028] Furthermore, the processor 1001 can call the drone target detection and tracking program stored in the memory 1005 and also perform the following operations: For the time when the camera acquires an image , acquire all radar data whose time tags fall within the time interval as a subset of radar measurements, and the radar data is a candidate measurement for the confirmation step, identified as ; is the time interval between two consecutive frame acquisitions; Delete the radar echoes with all Doppler measurements below the preset threshold , and project the remaining radar beam directions onto the image plane.

[0029] Furthermore, the processor 1001 can call the drone target detection and tracking program stored in the memory 1005 and also perform the following operations: Convert the radar data into pixel coordinates according to the following formula: ; ; ; where , and They are the coordinates of the radar beam direction in the CRF, respectively. , are the components of the camera focal length in the pixel. to are the radial and tangential distortion coefficients. , are the principal point coordinates.

[0030] Furthermore, the processor 1001 can call the UAV target detection and tracking program stored in the memory 1005 and further perform the following operations: Generate trajectories with different reliability levels, including one-time trajectories, temporary trajectories, and deterministic trajectories. After generating the deterministic trajectory, associate the visual measurement data with the deterministic trajectory.

[0031] Furthermore, the processor 1001 can call the UAV target detection and tracking program stored in the memory 1005 and further perform the following operations: After generating the deterministic trajectory, calculate the time of the closest point and the distance of the closest point ; If the time of the closest point and / or the distance of the closest point , then a conflict is considered to exist. Then, generate and update the trajectory by adjusting the heading or speed.

[0032] The specific embodiments of the present invention applying the data storage device are basically the same as those of the following embodiments of the UAV target detection and tracking method based on radar-vision fusion, and will not be elaborated herein.

[0033] Referring to Figures 2-3 , the first embodiment of the present invention provides a UAV target detection and tracking method based on radar-vision fusion. The UAV target detection and tracking method based on radar-vision fusion includes: Obtain the camera measurement data acquired by the camera and the radar measurement data acquired by the radar, and align the camera measurement data and the radar measurement data. Use the camera as an auxiliary sensor to screen the radar detection to ensure that both sensors have detected the same target. Fuse the radar measurement data and the camera measurement data, and output the fused data to the tracker to generate and update the trajectories of the targets recognized within the sensor field of view.

[0034] The vision sensor uses a visible light camera, and the radar is an electronic scanning array radar in millimeter wave radars. For the vision camera, a CNN architecture of YOLOv2 is adopted for target detection. For the radar, the electronic scanning array radar generates grid-like scans over the entire fixed field of view to obtain complete information.

[0035] In the two-level radar / vision sensor fusion strategy, the first level: detection-level fusion, also known as FBT (Fuse-Before-Track) fusion before tracking.

[0036] In the FBT step, the camera is used as an auxiliary sensor to screen the radar detections to ensure that both sensors have detected the same target. Its main purpose is to perform preliminary integration and processing on the radar and vision data before tracking the target, so as to reduce noise and clutter and improve computational efficiency.

[0037] The second level: tracking-level fusion. The radar measurements and vision measurements confirmed through cross-check are fused at the tracking level, and the fused detection results are input into a tracker based on the Extended Kalman Filter (EKF). It aims to estimate the motion of the UAV in the local North-East-Down (NED) reference frame to generate and update the trajectories of the targets identified within the sensor's field of view. Once a deterministic trajectory is generated, a collision detection method is further used to determine whether there is a collision threat.

[0038] A method for detecting and tracking UAVs based on radar-vision fusion specifically includes the following steps: Step 1: Sensor spatial alignment.

[0039] Spatial alignment is determined through radar / camera calibration to obtain the rotation matrices ( and for the radar and camera respectively) between the sensor reference frame and the local NED reference frame. The QUEST algorithm for quaternion estimation is used to achieve the spatial alignment of the sensors.

[0040] The specific situation is as follows: For the camera alignment procedure, a series of measurement data obtained by the camera, such as the target position in the image, etc., are represented in the form of the camera reference frame (CRF).

[0041] For the radar alignment procedure, the measurement data obtained by the radar, such as the distance and speed of the target, etc., are represented in the form of the radar reference frame (RRF).

[0042] By converting this data into unit vector observations in the NED frame and then using the QUEST algorithm for alignment, the postures of the radar and camera relative to the NED frame are determined.

[0043] The CDGNSS reference refers to the Carrier Differential Global Navigation Satellite System (CDGNSS). Through means such as high-precision satellite positioning, real-time correction, and multi-source information fusion, its determination of the target position in the NED (North-East-Down) frame is very accurate, so that the data of both the camera and the radar can be aligned with this as a reference. The independent observations generated by CDGNSS are input into the QUEST algorithm to estimate the alignment of the camera and the radar in the NED frame. Since the position in the NED frame can be accurately determined by the CDGNSS reference, and the measurements of the radar and the camera can be accurately extracted using supervised methods, the QUEST algorithm can reliably deduce the respective spatial postures (i.e., rotation matrix or Euler angles) of the camera and the radar.

[0044] By selecting a specific target object (Eagle) for sensor space alignment, since the specific position of the Eagle is accurately known through CDGNSS technology before starting the calibration of the camera and the radar, the exact coordinates of the Eagle on the earth are also known. Then, this known position is used to help the camera and the radar align accurately. By confirming the position of the Eagle, it can be ensured that both the camera and the radar can accurately see the Eagle, thus ensuring that their measurement data is correct.

[0045] In this process, only the measurement values that meet specific conditions are used. For example, the camera measurement values must meet specific conditions, such as: The measurement conditions of the camera include: (1) Visibility, the Eagle must be in the field of view of the camera, the image is clear and there is no blur; (2) Position, the distance and angle between the Eagle and the camera should be appropriate, not too far or too close; (3) Background interference, there should be no other targets in the image captured by the camera interfering with the recognition of the Eagle. The measurement conditions of the radar include: (1) Effective echo, the radar signal must be clearly reflected back to accurately measure the distance and speed of the Eagle; (2) Stray signal filtering, it is necessary to exclude irrelevant signals from the ground or other directions to reduce false alarms.

[0046] Further define that other targets are objects different from Eagle that exist within the field of view of the sensor. These objects may make us mistakenly think they are Eagle, so special attention needs to be paid to distinguish them. Eagle and other targets can be distinguished by the following metrics: The appearance characteristics (such as color, shape) of Eagle are different from other objects. If the measurement results of the radar and camera show that the distance and speed of a certain target match those of Eagle, it can be considered as Eagle; if the radar detects an object and the camera also recognizes the same object, it can be more certain that this target is Eagle.

[0047] Step 2: Detection-level fusion, also known as pre-tracking fusion of FBT.

[0048] The specific situation is as follows: In the steps of FBT, the radar measurements are projected onto the image plane and compared with the corresponding visual detections to verify their consistency with the visual data, while the visual data is not affected by the same clutter phenomenon. Since the acquisition frequencies of the original radar detections and RGB images are different, the asynchrony between the two needs to be considered.

[0049] For the time when the camera acquires an image , consider all the radar data whose time tags fall within the time interval . These data are regarded as candidate measurements for the confirmation step and are labeled as . is the time interval between two consecutive frame acquisitions, which is used to determine the time match between the camera image and the corresponding radar data. This method does not depend on the acquisition frequency of a specific sensor and has generality.

[0050] Furthermore, once the subset of radar measurements with the closest time is determined, a filter based on Doppler measurement is applied to discard the detections that may be caused by static ground echoes, that is, all radar echoes with Doppler measurements lower than the threshold are deleted. Then, the directions of the remaining radar beams are projected onto the image plane. Considering the inverse mapping problem of the pinhole camera internal parameter model, they are converted into pixel coordinates through the following formula.

[0051]

[0052]

[0053]

[0054] where , and are the coordinates of the radar beam direction in the CRF respectively. In fact, , is the component of the camera focal length in the pixel, to are the radial and tangential distortion coefficients, , are the principal point coordinates.

[0055] By fusing and then tracking using the overlapping area of the radar and camera fields of view, the complementarity of radar and camera data can be utilized to improve the accuracy and robustness of target tracking.

[0056] Use the projected radar beam to construct a rectangular region of interest ROI to verify the consistency of radar measurements and decide whether to use them for target tracking. The following two types of measurement data will be used during the tracking process: radar measurements, such as (where ), from the FBT process; visual detections, such as (where ), output by the CNN.

[0057] Step 3: Tracking-level fusion. The radar detection results generated by the FBT process and the CNN visual detections are input into a tracker based on the extended Kalman filter EKF to generate and update the target trajectory within the sensor field of view.

[0058] The tracker adopts a hierarchical structure, where the radar serves as the main sensor, and the visual camera is used to provide additional and more accurate angular observation data, thereby improving the tracking performance.

[0059] The tracker's process is divided into three stages, generating trajectories with different reliability levels respectively.

[0060] First is the one-time trajectory OPT, then the temporary trajectory TT, and finally the firm trajectory FT. The reliability of these trajectories gradually increases, and visual data is only used when a reliable radar-based firm trajectory FT has been generated to further improve the tracking performance. Thus, the tracker function is realized, and the current state estimate of the target is output The specific situation is as follows: In the prediction stage, the target state is predicted using the nearly constant velocity NCV model to clarify the uncertainty and external interference of this trajectory prediction. Use the state transition matrix and the process noise matrix to estimate the state vector and covariance matrix .

[0061] The scale factor of the process noise matrix needs to be based on The expected maximum speed of the tracked object within a time interval (in seconds) is adjusted.

[0062] During the correction phase, the detection data from different sensors are transformed into a unified NED reference frame to facilitate comparison with the predicted state vector. When the Kalman filter attempts to associate new radar measurement data with the existing trajectory, it uses the Mahalanobis distance as the evaluation criterion for the association condition.

[0063] The Mahalanobis distance not only considers the current actual measurement value and the measurement value estimated by the filter but also incorporates measurement error and state uncertainty. The calculation formula is as follows:

[0064] When this distance is less than the pre-set threshold GRAD, it can be considered a successful association.

[0065] Where the calculation of involves the measurement covariance matrix and the Jacobian matrix . is a diagonal matrix that contains the uncertainties of radar measurements, such as the variances of distance, azimuth angle, elevation angle, and radial velocity. represents the relationship between the radar measurement value and the state vector and can be calculated through the partial derivatives of the radar measurement model, that is . The formula is as follows, which defines the four components of the radar measurement model, namely the distance of the target, the azimuth angle , the elevation angle , and the calculation expressions of the radial velocity .

[0066]

[0067] When the radar Doppler measurement exceeds the maximum unambiguous limit of the instrument, the relative velocity information provided by the radar becomes unreliable and cannot accurately reflect the true velocity of the target. In response to this situation, a tracking-level radar-vision fusion method is proposed. Even when the radar Doppler measurement fails, the system can still maintain and update the target trajectory by relying on the information provided by the vision sensor, such as measurement data of azimuth angle and elevation angle.

[0068] When there is at least one deterministic trajectory FT, the vision measurement is associated with the trajectory.

[0069] The vision module of the fixed tracker runs after the radar module. The state and state covariance to be updated can be either the state of the radar filter or the predicted state, depending on the result of the association between the previous radar measurement and the trajectory. Even when there is insufficient or unavailable radar measurement data, the system can still update the target tracking through angle measurements based on a CNN (Convolutional Neural Network).

[0070] Visual detection vector Contains only two measurement quantities and , a new measurement Jacobian matrix needs to be defined . The measurement covariance matrix used in the case of vision update Considers fewer measurement quantities, and the azimuth angle and elevation angle on its diagonal are the squared uncertainties. The association between the vision measurement and the trajectory again adopts the criterion based on the Mahalanobis distance . The updated calculation formula is as follows

[0071]

[0072] If is less than the threshold GCAM, the success of the association can be verified, and at the same time, the verification condition GCAM < GRAD should be satisfied

[0073] Step 4: Conflict detection. The specific situation is as follows Once the deterministic trajectory FT is generated, further processing is carried out to determine whether it represents a collision threat. This is completed through conflict detection analysis, which relies on calculating the time to closest point and the distance to closest point . Once the estimated distance to closest point is below a predefined threshold, a conflict is considered to exist

[0074] Generally, this threshold takes into account uncertainty and adds it to increase the conservativeness of conflict determination. This uncertainty comes from measurement errors in the detection and tracking process and is considered by calculating the state covariance matrix and converting it to the form in spherical coordinate system

[0075] If the time to closest point is short, the distance to closest point is small, and below the predefined threshold, it can be judged that there is a potential collision threat. If there is a threat, the system can take preventive measures in a timely manner, such as adjusting the heading or speed, generating and updating the trajectory, to avoid collisions, thereby ensuring the safety and reliability of the system

[0076] In addition, an embodiment of the present invention further provides a computer-readable storage medium, on which a drone target detection and tracking program is stored. When the drone target detection and tracking program is executed by a processor, the following operations are implemented: Obtain camera measurement data acquired by a camera and radar measurement data acquired by a radar, and align the camera measurement data and the radar measurement data; Use the camera as an auxiliary sensor to screen the radar detection to ensure that both sensors have detected the same target; Fuse the radar measurement data and the camera measurement data, and output the fused data to a tracker to generate and update the trajectory of the target recognized within the sensor's field of view.

[0077] Further, when the drone target detection and tracking program is executed by a processor, the following operations are also implemented: Convert this data into unit vector observations in the NED frame, and then use the QUEST algorithm for alignment to determine the attitudes of the radar and the camera relative to the NED frame.

[0078] Further, when the drone target detection and tracking program is executed by a processor, the following operations are also implemented: For the time when the camera acquires an image , obtain all radar data whose time tags fall within the time interval as a subset of radar measurement values. The radar data is a candidate measurement for the confirmation step and is labeled as ; is the time interval between two consecutive frame acquisitions; Delete radar echoes with all Doppler measurement values lower than a preset threshold , and project the remaining radar beam directions onto the image plane.

[0079] Further, when the drone target detection and tracking program is executed by a processor, the following operations are also implemented: Convert the radar data into pixel coordinates according to the following formula: ; ; ; where , and are the coordinates of the radar beam direction in the CRF respectively, , are the components of the camera focal length in the pixel, to are the radial and tangential distortion coefficients, , are the principal point coordinates.

[0080] Furthermore, when the drone target detection and tracking program is executed by a processor, the following operations are also implemented: Generate trajectories with different reliability levels, including one-time trajectories, temporary trajectories, and deterministic trajectories; After generating a deterministic trajectory, associate the visual measurement data with the deterministic trajectory.

[0081] Furthermore, when the drone target detection and tracking program is executed by a processor, the following operations are also implemented: After generating a deterministic trajectory, calculate the time of the closest point and the distance of the closest point ; If the time of the closest point and / or the distance of the closest point , it is considered that there is a conflict; Then, generate and update the trajectory by adjusting the heading or speed The specific embodiments of the computer-readable storage medium of the present invention are basically the same as the above-described embodiments of the method for detecting and tracking a drone target based on radar-vision fusion, and will not be elaborated herein.

[0082] The above-described method for detecting and tracking a drone target based on radar-vision fusion can be installed on a drone, and data is transmitted through the drone to the processor of the drone for detecting and tracking the target of the drone.

[0083] It should be noted that, in this article, the terms "including", "comprising" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such process, method, article or system. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of another identical element in the process, method, article or system including that element.

[0084] The serial numbers of the above-described embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0085] Through the description of the above embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above and includes several instructions for causing a terminal device (which can be a drone, mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in various embodiments of the present invention.

[0086] The above are only the preferred embodiments of the present invention, and do not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of the present invention, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of the present invention.

Claims

1. A method for detecting and tracking unmanned aerial vehicle (UAV) targets based on radar-vision fusion, characterized in that, The method for drone target detection and tracking based on radar-vision fusion includes the following steps: Obtain the camera measurement data acquired by the camera and the radar measurement data acquired by the radar, and align the camera measurement data and the radar measurement data; Use the camera as an auxiliary sensor to screen the radar detection to ensure that both sensors have detected the same target; Fuse the radar measurement data and the camera measurement data, and output the fused data to the tracker to generate and update the trajectories of the targets recognized in the sensor field of view.

2. The method for detecting and tracking drone targets based on radar-vision fusion according to claim 1, wherein The step of obtaining the camera measurement data acquired by the camera and the radar measurement data acquired by the radar, and aligning the camera measurement data and the radar measurement data includes: Convert these data into unit vector observations in the NED frame, and then use the QUEST algorithm for alignment to determine the postures of the radar and the camera relative to the NED frame.

3. The method for drone target detection and tracking based on radar-vision fusion according to claim 2, wherein, The step of using the camera as an auxiliary sensor to screen the radar detection includes: For the time when the camera acquires an image , obtain all radar data whose acquisition time tags fall within the time interval as a subset of radar measurement values, where the radar data are candidate measurements for the confirmation step and are identified as ; is the time interval between two consecutive frame acquisitions; Delete all radar echoes with Doppler measurements below a preset threshold and project the remaining radar beam directions onto the image plane.

4. The method for detecting and tracking an unmanned aerial vehicle target based on the fusion of radar and vision according to claim 3, wherein Convert the radar data into pixel coordinates according to the following formula: ； ； ； Among them, , and are the coordinates of the radar beam direction in the CRF respectively, , are the components of the camera focal length in the pixel, to are the radial and tangential distortion coefficients, , are the principal point coordinates.

5. The method for drone target detection and tracking based on radar-vision fusion according to claim 4, wherein The step of generating and updating the trajectories of the targets recognized in the sensor field of view includes: Generate trajectories with different reliability levels, including one-time trajectories, temporary trajectories, and deterministic trajectories; When a deterministic trajectory is generated, associate the visual measurement data with the deterministic trajectory.

6. The method for drone target detection and tracking based on radar-vision fusion according to claim 5, characterized in that, After the step of fusing the radar measurement data and the camera measurement data, and outputting the fused data to the tracker to generate and update the trajectories of the targets recognized in the sensor field of view, the method further includes: After generating the deterministic trajectory, calculate the time of the closest point and the distance to the closest point ; If the closest point in time and / or the closest point distance then a conflict is considered to exist; Generate and update the trajectory by adjusting the heading or speed.

7. An unmanned aerial vehicle target detection and tracking device based on radar-vision fusion, characterized in that, The drone target detection and tracking device based on radar-vision fusion includes: a memory, a processor, and a drone target detection and tracking program stored on the memory and executable on the processor. When the drone target detection and tracking program is executed by the processor, it implements the steps of the method for drone target detection and tracking based on radar-vision fusion according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, A drone target detection and tracking program is stored on the computer-readable storage medium. When the drone target detection and tracking program is executed by the processor, it implements the steps of the method for drone target detection and tracking based on radar-vision fusion according to any one of claims 1 to 6.

Images