Foundation unmanned aerial vehicle positioning scheme based on event camera and millimeter wave radar high-frequency fusion

Abstract

The invention provides a ground-based unmanned aerial vehicle positioning scheme based on event camera and millimeter-wave radar high-frequency fusion, and belongs to the technical field of the Internet of Things. Event stream data of an unmanned aerial vehicle are acquired based on an event camera; acquiring radar measurement data of the unmanned aerial vehicle based on a millimeter wave radar; and based on the event stream data and the radar measurement data, carrying out time synchronization cooperative tracking processing on the unmanned aerial vehicle to obtain ground positioning measurement data of the unmanned aerial vehicle. According to the technical scheme of the invention, the method can achieve the stable high-precision and low-delay ground positioning of the unmanned plane through the coordination of the ultrahigh sampling frequency characteristics of the event camera and the millimeter-wave radar.

Description

Technical Field

[0001] The present application relates to the field of Internet of Things technology, and in particular to a ground-based UAV positioning solution based on high-frequency fusion of event cameras and millimeter-wave radars. Background Art

[0002] Most UAV ground positioning solutions are based on satellite systems. The Global Positioning System (GPS) can provide meter-level accuracy in outdoor environments, and Real-time Kinematic Positioning (RTK) can achieve centimeter-level accuracy. However, the cost of UAV ground positioning based on satellite systems is high, and the performance of UAV ground positioning based on satellite systems is poor in complex environments such as cities and canyons.

[0003] In addition, there are also optical-based ground positioning solutions for drones in related technologies (such as motion capture technology), which can achieve centimeter-level accuracy indoors. However, since optical-based positioning is highly dependent on precise calibration, it becomes impractical for use in outdoor environments.

[0004] In summary, how to stably perform high-precision ground positioning of UAVs is a technical problem that needs to be solved urgently in the relevant technical field. Summary of the invention

[0005] The main purpose of the embodiments of the present application is to propose a ground-based UAV positioning solution based on the high-frequency fusion of event camera and millimeter-wave radar, aiming to coordinate the ultra-high sampling frequency characteristics of the two sensors, event camera and millimeter-wave radar, to achieve stable high-precision and low-latency UAV ground positioning.

[0006] To achieve the above objectives, the first aspect of the embodiment of the present application proposes a ground-based UAV positioning solution based on high-frequency fusion of event camera and millimeter-wave radar, including:

[0007] Acquiring event stream data of the drone based on an event camera; and acquiring radar measurement data of the drone based on a millimeter wave radar;

[0008] The UAV is subjected to time-synchronized collaborative tracking processing based on the event stream data and the radar measurement data to obtain ground positioning measurement data of the UAV.

[0009] In some embodiments, the performing time-synchronized collaborative tracking processing on the UAV based on the event stream data and the radar measurement data to obtain ground positioning measurement data of the UAV includes:

[0010] Acquire target event stream data in the event stream data, and acquire target radar measurement data in the radar measurement data; wherein the target event stream data is time synchronized with the target radar measurement data;

[0011] Input the target event stream data into an event tracking model to obtain a first detection result; and, input the target radar measurement data into a radar tracking model to obtain a second detection result;

[0012] Determine the ground positioning measurement data of the UAV based on the first detection result and the second detection result.

[0013] In some embodiments, the determining the ground positioning measurement data of the UAV based on the first detection result and the second detection result includes:

[0014] Obtain a cross-modal alignment result of the first detection result and the second detection result;

[0015] Extract the ground positioning measurement data of the UAV from the cross-modal alignment result based on the micro-motion period of the UAV.

[0016] In some embodiments, the obtaining the cross-modal alignment result of the first detection result and the second detection result includes:

[0017] Input the first detection result and the second detection result into a measurement filter;

[0018] Perform cross-modal alignment on the first detection result and the second detection result through the measurement filter based on the time consistency and / or spatial relationship between the event camera and the millimeter-wave radar to obtain the cross-modal alignment result of the first detection result and the second detection result.

[0019] In some embodiments, the determining the ground positioning measurement data of the UAV based on the first detection result and the second detection result includes:

[0020] Construct a factor graph based on the first detection result and the second detection result;

[0021] Based on dynamic joint optimization of the connected factor nodes in the factor graph, obtain the position state estimation of the UAV;

[0022] Determine the ground positioning measurement data of the UAV based on the position state estimation.

[0023] In some embodiments, the dynamic joint optimization of the factor nodes in the factor graph includes:

[0024] Optimize the prior probability represented by the connected factor nodes in the factor graph based on maximum a posteriori estimation, and optimize the likelihood probability of the event tracking model and the radar tracking model represented by the connected factor nodes;

[0025] Among them, the prior probability is obtained based on an analysis of the flight characteristics of the UAV. The likelihood probability of the event tracking model includes the likelihood probability of the first detection result, and the likelihood probability of the radar tracking model includes the likelihood probability of the second detection result.

[0026] To achieve the above object, a second aspect of the embodiments of the present application proposes a ground-based UAV positioning device based on high-frequency fusion of an event camera and a millimeter-wave radar. The device includes:

[0027] An acquisition module, configured to acquire event stream data of the UAV based on the event camera; and acquire radar measurement data of the UAV based on the millimeter-wave radar;

[0028] A positioning module, configured to perform time-synchronized collaborative tracking processing on the UAV based on the event stream data and the radar measurement data to obtain ground positioning measurement data of the UAV.

[0029] To achieve the above object, a third aspect of the embodiments of the present application proposes a computer device, which includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the ground-based UAV positioning solution described in the first aspect above.

[0030] To achieve the above object, a fourth aspect of the embodiments of the present application proposes a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, it implements the ground-based UAV positioning solution described in the first aspect above.

[0031] To achieve the above object, a fifth aspect of the embodiments of the present application proposes a computer program product, which stores a computer program, and when the computer program is executed by a processor, it implements the ground-based UAV positioning solution described in the first aspect above.

[0032] The ground-based UAV positioning solution, device, equipment, storage medium, and product proposed by the embodiments of the present application first acquire event stream data of the UAV based on the event camera; and acquire radar measurement data of the UAV based on the millimeter-wave radar; then, based on the acquired event stream data and radar measurement data, perform time-synchronized collaborative tracking processing on the UAV, so as to obtain accurate ground positioning measurement data of the UAV.

[0033] Thus, compared with satellite - based or optical - based UAV ground positioning solutions, in the embodiments of the present application, by utilizing the millisecond - level sampling delay provided by an event camera, which is a bionic sensor (the event camera can report pixel - level intensity changes with millisecond - level resolution, capture high - speed motion without blurring, and is thus particularly suitable for fast - tracking tasks), the basic fact that it highly coincides with the high sampling frequency of millimeter - wave radar, and the two - dimensional imaging ability of the event camera can effectively complement the deficiency of radar in spatial resolution, in a working mode similar to that of a traditional frame camera and the working mode of millimeter - wave radar, the combination in terms of time consistency and spatial complementarity effectively stimulates the possibility of using an event camera to fuse with millimeter - wave radar to achieve more efficient ground positioning and tracking of UAVs. That is to say, in the embodiments of the present application, the event - stream data of the UAV is obtained through the event camera, and the radar measurement data of the UAV is obtained through the millimeter - wave radar. Then, based on the event - stream data and the radar measurement data, accurate ground positioning of the UAV is performed. By fusing the ultra - high sampling frequency characteristics of these two sensors, namely the event camera and the millimeter - wave radar, the accuracy and latency bottlenecks of UAV ground positioning are overcome, thereby improving the performance of UAV ground positioning, enabling stable high - precision and low - latency UAV ground positioning, and in scenarios facing UAV landing, enabling high - frequency precise positioning of UAVs. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a schematic flowchart of the steps in some embodiments of the ground - based UAV positioning solution based on high - frequency fusion of an event camera and a millimeter - wave radar provided by the embodiments of the present application;

[0035] Figure 2 For Figure 1 a schematic flowchart of the refined steps of step S102 in

[0036] Figure 3 It is a schematic diagram of a model for calculating distance through frequency difference involved in the ground - based UAV positioning solution based on high - frequency fusion of an event camera and a millimeter - wave radar provided by the embodiments of the present application;

[0037] Figure 4 It is a schematic diagram of a model for calculating direction based on phase difference involved in the ground - based UAV positioning solution based on high - frequency fusion of an event camera and a millimeter - wave radar provided by the embodiments of the present application;

[0038] Figure 5 It is a schematic diagram of an event - tracking model involved in the ground - based UAV positioning solution based on high - frequency fusion of an event camera and a millimeter - wave radar provided by the embodiments of the present application;

[0039] Figure 6 For Figure 2 a schematic flowchart of a refined step of step S203 in

[0040] Figure 7 Schematic diagram of the collaborative tracking method for the consistency guidance involved in some embodiments of the ground-based UAV positioning solution based on the high-frequency fusion of an event camera and a millimeter-wave radar provided by the embodiments of the present application;

[0041] Figure 8 is Figure 2 Another refined step flow schematic diagram of step S203 in;

[0042] Figure 9 Schematic diagram of the graph-driven dynamic joint optimization method involved in some embodiments of the ground-based UAV positioning solution based on the high-frequency fusion of an event camera and a millimeter-wave radar provided by the embodiments of the present application;

[0043] Figure 10 Schematic diagram of the overall system structure involved in some embodiments of the ground-based UAV positioning solution based on the high-frequency fusion of an event camera and a millimeter-wave radar provided by the embodiments of the present application;

[0044] Figure 11 Schematic diagram of the structure of the ground-based UAV positioning device based on the high-frequency fusion of an event camera and a millimeter-wave radar provided by the embodiments of the present application;

[0045] Figure 12 Schematic diagram of the hardware structure of the computer device provided by the embodiments of the present application. Detailed implementation manners

[0046] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application.

[0047] It should be noted that although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order from the module division in the device or the order in the flowchart. Terms such as "first" and "second" in the specification, claims and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0049] First, the overall concept of the embodiments of the present application will be described.

[0050] In related technologies, the UAV ground positioning solutions based on satellite systems include the UAV ground positioning solutions based on the Global Positioning System (GPS) and Real-Time Kinematic (RTK). Among them, the Global Positioning System (GPS) can provide meter-level accuracy in outdoor environments, while Real-Time Kinematic (RTK) achieves centimeter-level accuracy. However, its high cost limits its widespread application. At the same time, these satellite system-based positioning solutions do not perform well in complex environments such as cities and canyons. In addition, the UAV positioning solutions based on optics (such as motion capture technology) can achieve centimeter-level accuracy indoors, but their dependence on precise calibration makes their application in outdoor environments impractical.

[0051] To overcome the various challenges existing in the above UAV positioning solutions, the academic community has proposed a variety of sensor-based technologies, including cameras, radars, and lidars, which are usually combined with Simultaneous Localization and Mapping (SLAM) or deep learning algorithms, aiming to improve the positioning accuracy of UAVs. However, the spatio-temporal resolution of these sensors is limited, often affecting the high-precision and low-latency UAV landing positioning. Specifically, millimeter-wave radars perform well in estimating the depth of the radial axis but have difficulties in capturing tangential motion; cameras and lidars with low frame rates (<50Hz) may also reduce the UAV positioning accuracy due to missing frames with fast motion.

[0052] In addition, due to its sub-millimeter wavelength, millimeter-wave radar has higher sensitivity and precision, and millimeter-wave radar can provide high-sensitivity and high-precision measurements with its shorter wavelength. Therefore, the UAV ground positioning solution based on millimeter-wave radar usually combines the signal strength method when performing UAV positioning and tracking, but it still faces difficulties in accurately tracking the center of the UAV. This difficulty stems from the relatively large volume of the UAV (for example, the cross diameter of the UAV can reach 80 cm), making the UAV appear as an uneven patch in the radar echo. In addition, these solutions often produce unstable results and are accompanied by frequent outliers. This is because multipath scattering may obscure the main signal, and the low spatial resolution of the radar exacerbates this problem. Some other solutions adopt deep learning technologies, but these solutions require extensive pre-modeling and neural network training for each UAV model, resulting in these methods usually performing poorly when tracking different types of UAVs and significantly degrading in environments not covered by the training dataset. There are also some solutions that combine visual sensors (such as cameras) with radars to improve the positioning accuracy. However, these methods will introduce additional time delays due to the camera exposure time and image processing delays. In addition, motion blur in camera frames may also interfere with the algorithm, resulting in increased positioning errors.

[0053] Based on this, the embodiments of the present application propose a ground-based UAV positioning solution, device, computer device, computer-readable storage medium, and computer program product based on the high-frequency fusion of an event camera and a millimeter-wave radar. It belongs to a high-frequency precise positioning solution for the UAV landing scenario. By fusing the high sampling frequency characteristics of the event camera and the millimeter-wave radar, it overcomes the accuracy and latency bottlenecks and improves the system performance for ground positioning of UAVs.

[0054] In the embodiments of the present application, first, event stream data of the UAV is obtained based on the event camera; and radar measurement data of the UAV is obtained based on the millimeter-wave radar; then, based on the obtained event stream data and radar measurement data, collaborative tracking processing with time synchronization is performed on the UAV, so as to obtain accurate ground positioning measurement data of the UAV.

[0055] In this way, the embodiments of the present application utilize the millisecond-level sampling delay provided by this bionic sensor of the event camera, which highly coincides with the high sampling frequency of the millimeter-wave radar, and the fact that the two-dimensional imaging ability of the event camera can effectively complement the deficiency of the radar in spatial resolution. In a way similar to the working mode of a traditional frame camera and the working mode of the millimeter-wave radar, the combination of time consistency and spatial complementarity effectively stimulates the possibility of using the event camera to fuse the millimeter-wave radar to achieve more efficient ground positioning and tracking of UAVs. That is to say, the embodiments of the present application obtain the event stream data of the UAV through the event camera, and obtain the radar measurement data of the UAV through the millimeter-wave radar, and then perform accurate ground positioning on the UAV based on the event stream data and the radar measurement data. By fusing the ultra-high sampling frequency characteristics of these two sensors, namely the event camera and the millimeter-wave radar, the accuracy and latency bottlenecks of UAV ground positioning are overcome, thereby improving the performance of UAV ground positioning, enabling stable high-precision and low-latency UAV ground positioning, and in the scenario facing UAV landing, enabling high-frequency precise positioning of UAVs.

[0056] Based on the overall concept of the above embodiments of the present application, specific embodiments of the ground-based UAV positioning solution, device, computer device, computer-readable storage medium, and computer program product based on the high-frequency fusion of an event camera and a millimeter-wave radar provided by the embodiments of the present application are proposed. First, various specific embodiments of the ground-based UAV positioning solution based on the high-frequency fusion of an event camera and a millimeter-wave radar in the embodiments of the present application are described in detail.

[0057] Embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, Artificial Intelligence (AI) is a theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results.

[0058] Artificial intelligence basic technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. Artificial intelligence software technologies mainly include several major directions such as computer vision technology, robotics, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0059] It should be noted that in each specific embodiment of the present application, when it comes to performing relevant processing based on data related to the user's identity or characteristics, such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first. Moreover, the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiments of the present application need to obtain the user's sensitive personal information, the user's separate permission or separate consent will be obtained through methods such as pop-up windows or redirecting to a confirmation page. After clearly obtaining the user's separate permission or separate consent, the necessary user-related data for the normal operation of the embodiments of the present application will be obtained.

[0060] In addition, the ground-based drone positioning solution based on the high-frequency fusion of event cameras and millimeter-wave radars provided by the embodiments of the present application can be applied to terminals, can also be applied to the server side, or can be software running on the terminal or the server side. In some embodiments, the terminal can be a ground base station of a drone, a smartphone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, can also be configured as a server cluster or distributed system composed of multiple physical servers, or can be configured as a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the ground-based drone positioning solution based on the high-frequency fusion of event cameras and millimeter-wave radars, etc., but is not limited to the above forms.

[0061] Alternatively, the ground-based UAV positioning solution based on the high-frequency fusion of an event camera and a millimeter-wave radar provided by the embodiments of the present application can also be used in numerous general or special computer system environments or configurations. For example: UAV positioning systems, personal computers, server computers, handheld or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer computer devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and so on. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0062] For ease of understanding and elaboration, in the following text, the application of the ground-based UAV positioning solution based on the high-frequency fusion of an event camera and a millimeter-wave radar provided by the embodiments of the present application to a terminal device will be taken as an example for detailed description. The implementation of the ground-based UAV positioning solution based on the high-frequency fusion of an event camera and a millimeter-wave radar provided by the embodiments of the present application for any of the above forms of the subject can refer to the process of the application of the ground-based UAV positioning solution based on the high-frequency fusion of an event camera and a millimeter-wave radar to a terminal device described later.

[0063] Please refer to Figure 1 , Figure 1 which is a schematic flowchart of the steps of the ground-based UAV positioning solution based on the high-frequency fusion of an event camera and a millimeter-wave radar provided by the embodiments of the present application in some embodiments. It should be understood that although Figure 1 shows the execution order of some method steps, due to different design requirements in actual applications, the ground-based UAV positioning solution based on the high-frequency fusion of an event camera and a millimeter-wave radar provided by the embodiments of the present application can of course adopt an execution order different from that shown in the figure. That is, Figure 1 the order of the method steps shown does not constitute a limitation on the execution logic order of the ground-based UAV positioning solution based on the high-frequency fusion of an event camera and a millimeter-wave radar provided by the embodiments of the present application, and any reasonable changes to the order of the method steps shown based on Figure 1 should be included within the protection scope of the ground-based UAV positioning solution based on the high-frequency fusion of an event camera and a millimeter-wave radar provided by the embodiments of the present application.

[0064] Such as Figure 1As shown, in some embodiments, the ground-based UAV positioning solution provided by the embodiments of the present application based on the high-frequency fusion of an event camera and a millimeter-wave radar may include, but is not limited to, steps S101 to S102.

[0065] Step S101: Obtain the event stream data of the UAV based on the event camera; and obtain the radar measurement data of the UAV based on the millimeter-wave radar.

[0066] During the flight of the UAV, the terminal device continuously monitors the UAV through various sensors configured on the UAV itself and various sensors of the ground base station. Thus, in the scenario where the UAV lands, the terminal device obtains, through the event camera of the ground base station, the event stream data obtained by the event camera for real-time asynchronous event collection of the UAV, and, through the millimeter-wave radar of the ground base station, the radar measurement data obtained by the millimeter-wave radar for real-time measurement of the UAV.

[0067] Step S102: Perform time-synchronized collaborative tracking processing on the UAV based on the event stream data and the radar measurement data to obtain the ground positioning measurement data of the UAV.

[0068] After the terminal device obtains the event stream data and the radar measurement data of the UAV, it filters the noise of the event stream data and the radar measurement data, and performs time-synchronized collaborative tracking on the UAV based on the filtered event stream data and radar measurement data, so as to obtain preliminary ground positioning measurement data based on the positioning detection of the ground base station in the UAV landing scenario.

[0069] In some embodiments, the terminal device may process the event stream data and the radar measurement data separately, and then align the results obtained by processing the event stream data and the radar measurement data respectively based on a measurement filter with time consistency, so as to obtain the ground positioning measurement data of the UAV in the landing scenario, that is, to realize the preliminary positioning of the UAV.

[0070] In the embodiments of the present application, the terminal device obtains the event stream data of the UAV through real-time asynchronous event collection of the UAV by the event camera of the ground base station, and obtains the radar measurement data of the UAV through real-time measurement of the UAV by the millimeter-wave radar of the ground base station. Then, based on the obtained event stream data and radar measurement data, time-synchronized collaborative tracking is performed on the UAV, and preliminary ground positioning measurement data is obtained based on the positioning detection of the ground base station in the UAV landing scenario.

[0071] Thus, compared with satellite-based or optical-based UAV ground positioning solutions, the embodiments of the present application utilize the millisecond-level sampling delay provided by the event camera, a bionic sensor, which highly matches the high sampling frequency of the millimeter-wave radar. Moreover, the two-dimensional imaging ability of the event camera can effectively complement the deficiency of the radar in spatial resolution. By combining in a manner similar to the working mode of a traditional frame camera and the working mode of the millimeter-wave radar in terms of temporal consistency and spatial complementarity, it effectively stimulates the possibility of using the event camera to fuse with the millimeter-wave radar to achieve more efficient ground positioning and tracking of UAVs.

[0072] That is to say, the embodiments of the present application obtain the event stream data of the UAV through the event camera and the radar measurement data of the UAV through the millimeter-wave radar. Then, based on the event stream data and the radar measurement data, accurate ground positioning of the UAV is performed. By fusing the ultra-high sampling frequency characteristics of these two sensors, namely the event camera and the millimeter-wave radar, the accuracy and latency bottlenecks of UAV ground positioning are overcome, thereby improving the performance of UAV ground positioning. It can achieve UAV ground positioning with higher accuracy and lower latency than traditional methods. Especially in the case of high-speed movement and complex environments of UAVs, it can stably provide accurate positioning information of UAVs.

[0073] In addition, through the innovative technology of fusing the event camera and the millimeter-wave radar for UAV ground positioning in the embodiments of the present application, the event camera and the millimeter-wave radar are complementary in the spatio-temporal dimension, and can also maximize the robustness of the system for UAV ground positioning. Especially, it can maintain good performance when the UAV is in a high-dynamic environment.

[0074] Please refer to Figure 2 , Figure 2 For Figure 1 the detailed step flow schematic diagram of step S102 in

[0075] As Figure 2 shown, in some embodiments, the above step S102: performing collaborative tracking processing for time synchronization of the UAV based on the event stream data and the radar measurement data to obtain the ground positioning measurement data of the UAV may include steps S201 to S203 as follows.

[0076] Step S201: Obtain the target event stream data in the event stream data and, obtain the target radar measurement data in the radar measurement data; wherein, the target event stream data is time-synchronized with the target radar measurement data.

[0077] After the terminal device obtains the event stream data and radar measurement data of the drone, when performing collaborative tracking processing for time synchronization of the drone based on the event stream data and radar measurement data, the terminal device first determines the target event stream data and target radar measurement data for time synchronization from the obtained event stream data and radar measurement data.

[0078] It should be noted that the time synchronization between the target event stream data and the target radar measurement data can be that the time when the terminal device acquires the target event stream data of the drone through the event camera is the same as the time when the terminal device measures the target radar measurement data of the drone through the millimeter-wave radar.

[0079] In some embodiments, the terminal device can extract the target event stream data and target radar measurement data for time synchronization by searching for the same timestamp information in the obtained event stream data and radar measurement data.

[0080] Step S202: Input the target event stream data into an event tracking model to obtain a first detection result; and input the target radar measurement data into a radar tracking model to obtain a second detection result.

[0081] After the terminal device extracts the target event stream data and target radar measurement data for time synchronization, it uses the target event stream data and target radar measurement data as inputs. Subsequently, it processes the asynchronous target event stream data through an event tracking model for event filtering, drone detection, and tracking to obtain a first detection result, and processes the target radar measurement data through a radar tracking model to obtain a second detection result.

[0082] It should be noted that the first detection result output by the terminal device based on the event tracking model for processing the target event stream data can be an estimated result of the preliminary three-dimensional position of the drone in the landing scenario. In addition, the second detection result output by the terminal device based on the radar tracking model for processing the target radar measurement data can be a sparse 3D point cloud, and the 3D point cloud generated by the radar tracking model can at least be used to characterize the distance and angle information between the drone and the millimeter-wave radar of the ground base station.

[0083] In some embodiments, the radar tracking model adopted by the terminal device can include a distance measurement model and an angle measurement model. Among them, the distance measurement model can be a model for calculating the distance through the frequency difference as Figure 3 shown, where the frequency difference between the transmitted (TX) signal and the received (RX) signal reflects the propagation time of the signal, and thus reveals the distance between the object (drone) and the radar (the millimeter-wave radar of the ground base station). Assume the distance at time i is D i, the transmitted (TX) signal and the received (RX) signal can be expressed as:

[0084]

[0085] where α represents the attenuation factor, f c is the initial frequency, K is the frequency modulation slope of the Frequency Modulated Continuous Wave (FMCW) signal, and c is the speed of light. After the TX and RX signals are mixed, the intermediate frequency signal (IF signal) s(t) is extracted through a low-pass filter (LPF). And the intermediate frequency value f IF extracted from it contains the distance information between the object and the radar. By performing Range-FFT (Range Fast Fourier Transform) processing on , f IF is extracted, and the distance D i between the object and the radar is calculated therefrom.

[0086] In addition, the angle measurement model can be a model for calculating the direction based on the phase difference as shown in Figure 4 . Among them, each linear array calculates the Angle of Arrival (AoA) based on the phase difference between adjacent antennas. For an antenna array with a spacing of d, the calculation formula for AoA θ is cosθ = Δφλ / 2πd. Where θ represents the Angle of Arrival, λ is the signal wavelength, and Δφ is the phase difference of the signals received by adjacent antennas. With the help of two orthogonal linear arrays, the radar (the millimeter-wave radar of the ground base station) can respectively obtain the Angle of Arrival θ x and θ y . At time i, the direction unit vector of the object (drone) is expressed as:

[0087]

[0088] In this embodiment, the terminal device calculates the direction by combining the angle information of the two orthogonal arrays through the angle measurement model, and can provide a high-precision three-dimensional space representation for the direction estimation of the drone.

[0089] In some embodiments, the event tracking model adopted by the terminal device can be as shown in Figure 5As shown in the figure. The terminal device estimates the initial three-dimensional position of the object (drone) based on the candidate bounding box and the projection function of the pinhole camera model. Among them, the projection function maps the three-dimensional points in the event camera coordinate system to two-dimensional pixel points on the image plane, and its calculation depends on the internal parameters of the event camera, including the focal length and the principal point coordinates. When estimating the object position, the center point of the bounding box is used as the initial estimate to calculate the position of the object in the event camera coordinate system. Specifically, the three-dimensional position of the object is jointly determined by the three-dimensional points in its object reference system, the translation vector between the reference systems, and the random noise of the center point of the bounding box.

[0090] In some embodiments, in order to improve the accuracy of estimating the position of the drone, the terminal device can perform preprocessing of undistorting the bounding box coordinates before extracting the center point from the candidate bounding box, so as to reduce the influence of the lens distortion of the event camera on the projection result. In this embodiment, the terminal device can significantly improve the reliability of the three-dimensional position estimation of the drone through the preprocessing step of undistorting the bounding box coordinates.

[0091] Step S203: Determine the ground positioning measurement data of the drone based on the first detection result and the second detection result.

[0092] After the terminal device takes the target event stream data and the target radar measurement data as inputs, and obtains the first detection result and the second detection result through the event tracking model and the radar tracking model respectively, it performs cross-modal fusion alignment on the first detection result (initial three-dimensional position) and the second detection result (3D point cloud) to achieve time-synchronized collaborative tracking of the drone, so as to obtain the initial ground positioning measurement data based on the positioning detection of the ground base station in the drone landing scenario.

[0093] Please refer to Figure 6 , Figure 6 For Figure 2 a schematic diagram of a refined step flow of step S203 in

[0094] As Figure 6 shown, in some embodiments, the above step S203: Determine the ground positioning measurement data of the drone based on the first detection result and the second detection result may include step S601 and step S602 as shown below.

[0095] Step S601: Obtain the cross-modal alignment result of the first detection result and the second detection result.

[0096] When the terminal device performs cross-modal fusion alignment on the first detection result and the second detection result to perform collaborative tracking for time synchronization of the drone, the terminal device first obtains the cross-modal alignment result obtained by performing cross-modal fusion alignment on the first detection result and the second detection result.

[0097] In some embodiments, step S601 above: obtaining the cross-modal alignment result of the first detection result and the second detection result may include the following steps:

[0098] Input the first detection result and the second detection result into the measurement filter;

[0099] Based on the time consistency and / or spatial relationship between the event camera and the millimeter-wave radar, the measurement filter performs cross-modal alignment on the first detection result and the second detection result to obtain the cross-modal alignment result of the first detection result and the second detection result.

[0100] When the terminal device performs cross-modal fusion alignment on the first detection result and the second detection result, it inputs the first detection result and the second detection result into the measurement filter based on time consistency, so as to utilize the time consistency between the two sensors, namely the event camera and the millimeter-wave radar, based on the measurement filter, and perform and / or spatial relationship on the first detection result and the second detection result, and then obtain the cross-modal alignment result between the first detection result and the second detection result.

[0101] Alternatively, the terminal device utilizes the spatial relationship between the drone and the two sensors, namely the event camera and the millimeter-wave radar, based on the measurement filter, and performs and / or spatial relationship on the first detection result and the second detection result, and then obtains the cross-modal alignment result between the first detection result and the second detection result.

[0102] Or, the terminal device utilizes the time consistency between the two sensors, namely the event camera and the millimeter-wave radar, and the spatial relationship between the drone and the two sensors, namely the event camera and the millimeter-wave radar, based on the measurement filter, and performs and / or spatial relationship on the first detection result and the second detection result, and then obtains the cross-modal alignment result between the first detection result and the second detection result.

[0103] Step S602: Based on the micro-motion cycle of the drone, extract the ground positioning measurement data of the drone from the cross-modal alignment result.

[0104] After the terminal device obtains the cross-modal alignment result between the first detection result and the second detection result, it further utilizes the periodic micro-motion characteristics of the drone to extract the measurement data related to the drone from the obtained cross-modal alignment result, and uses the extracted measurement data as the ground positioning measurement data of the drone.

[0105] Exemplarily, as Figure 7 shown, the terminal device can perform ground positioning of the drone in the landing scenario through a collaborative tracking method guided by consistency. Among them, for the original events (event stream data) generated by the drone monitored by the event camera of the ground base station, the terminal device first filters out the noise events triggered by the environment. Subsequently, the terminal device aligns the detection result (the first detection result) generated by the event camera with the detection result (the second detection result) of the millimeter-wave radar according to the sensor spatial relationship to obtain a cross-modal alignment result. Finally, the terminal device combines the periodic micro-motion characteristics of the drone to extract the detection results of the event camera and the millimeter-wave radar for the drone from the cross-modal alignment result, thereby realizing the preliminary ground positioning of the drone.

[0106] In this embodiment, by making full use of the spatio-temporal consistency of the event camera and the millimeter-wave radar in the data fusion process by the terminal device, the limitations of a single sensor can be overcome, thereby realizing more accurate trajectory tracking and ground positioning of the drone. In addition, by combining the event camera and the millimeter-wave radar by the terminal device, the anti-interference ability of the system for ground positioning of the drone can be greatly improved. Especially in complex weather conditions or low-light environments, it can effectively avoid the blur and mis-identification problems that may occur in traditional vision sensors.

[0107] Please refer to Figure 8 , Figure 8 For Figure 2 another detailed step flow diagram of step S203 in

[0108] As Figure 8 shown, in some embodiments, the above step S203: determining the ground positioning measurement data of the drone based on the first detection result and the second detection result may further include steps S801 to S803 as shown below.

[0109] Step S801: Construct a factor graph based on the first detection result and the second detection result.

[0110] The terminal device also performs fine positioning and trajectory optimization on the UAV through graph-driven dynamic joint optimization. That is, after the terminal device obtains the first detection result output by the event tracking model and the second detection result output by the radar tracking model, it further constructs a factor graph based on the first detection result and the second detection result. In some embodiments, the terminal device can construct a factor graph by using the first detection result output by the event tracking model and the second detection result output by the radar tracking model as measurement values.

[0111] Step S802: Obtain the position state estimate of the UAV based on dynamic joint optimization of the connected factor nodes in the factor graph.

[0112] Step S803: Determine the ground positioning measurement data of the UAV based on the position state estimate.

[0113] After the terminal device constructs the factor graph, it performs graph-driven dynamic joint optimization based on the factor graph, so as to obtain more accurate ground positioning measurement data of the UAV by performing fine positioning and trajectory optimization on the UAV. That is, the terminal device performs dynamic joint optimization on all connected factor nodes in the constructed factor graph, so as to obtain an accurate position state estimate of the UAV in the landing scenario. And the terminal device determines the accurate position state estimate of the UAV obtained through optimization as the ground positioning measurement data of the UAV in the landing scenario.

[0114] In some embodiments, the step of "performing dynamic joint optimization on the factor nodes in the factor graph" in step S802 above may include the following steps:

[0115] Optimize the prior probability represented by the connected factor nodes in the factor graph based on the maximum a posteriori estimation, and optimize the likelihood probability of each of the event tracking model and the radar tracking model represented by the connected factor nodes.

[0116] It should be noted that the prior probability is obtained by analyzing the flight characteristics of the UAV. The likelihood probability of the event tracking model includes the likelihood probability of the first detection result, and the likelihood probability of the radar tracking model includes the likelihood probability of the second detection result.

[0117] In some embodiments, the terminal device obtains the prior probability by analyzing the flight characteristics of the drone and uses the prior probability to describe the initial distribution of the variable nodes in the factor graph. In addition, the terminal device uses the likelihood probability of the pixel position measurement values (the first detection result) provided by the event tracking model to constrain the observation consistency of the event camera, and uses the distance vector in the measurement values of the radar tracking model (the second detection result) to correspondingly constrain the reliability of the millimeter-wave radar for distance measurement, and uses the direction vector in the measurement values of the radar tracking model to correspondingly constrain the accuracy of the direction information obtained by the millimeter-wave radar for direction measurement.

[0118] When the terminal device performs dynamic joint optimization on all connected factor nodes in the factor graph, it optimizes all connected factor nodes based on the maximum a posteriori estimation. That is, the terminal device accurately infers the state of the variable nodes by jointly optimizing the prior probability, the likelihood probability of the event tracking model, and the likelihood probability of the radar tracking model.

[0119] Exemplarily, as Figure 9 shown, when the terminal device performs fine positioning and trajectory optimization on the drone through graph-driven dynamic joint optimization, the factor graph consists of variable nodes and factor nodes, where the variable nodes represent the states to be optimized, such as the relative position between the drone and the event camera, and the factor nodes represent the probabilities of certain states given the measurement results. The measurement values are from the event tracking model and the radar tracking model, including the pixel position measurement values (the first detection result) provided by the event model and the distance and direction vectors (the second detection result) provided by the radar model. To estimate the state of the variable nodes given the measurement values, the terminal device optimizes all connected factor nodes in the factor graph based on the maximum a posteriori estimation. According to Bayes' theorem, this optimization process can be decomposed into the joint of the prior probability and the likelihood probability of the measurement model. Among them, the prior probability is obtained by analyzing the flight characteristics of the drone and is used to describe the initial distribution of the variable nodes; the likelihood probability of the pixel position measurement values provided by the event tracking model is used to constrain the observation consistency of the event camera; the likelihood probability of the measurement values of the radar model, including the distance and direction vectors, respectively corresponds to the reliability of the distance measurement and the accuracy of the direction information. Through joint optimization of these probability information, the state of the variable nodes can be accurately inferred, thereby improving the positioning and navigation capabilities of the drone.

[0120] In this embodiment, by adopting the dynamic joint optimization method based on the factor graph by the terminal device, the positioning trajectory of the drone can be adjusted in real time and the positioning accuracy can be adaptively optimized, so as to provide more accurate and stable ground positioning measurement data for the drone in a complex environment.

[0121] Next, a complete embodiment of the ground-based drone positioning scheme based on the high-frequency fusion of the event camera and the millimeter-wave radar proposed in the embodiments of the present application is presented.

[0122] As Figure 10 shown, the ground-based UAV positioning solution based on the high-frequency fusion of an event camera and a millimeter-wave radar proposed in the embodiments of the present application mainly includes two technical modules, namely, a collaborative tracking method guided by consistency, and a graph-driven dynamic joint optimization method. Among them, the collaborative tracking method guided by consistency is used for noise filtering, UAV detection, and preliminary positioning, and the graph-driven dynamic joint optimization method is used for the fine positioning and trajectory optimization of the UAV.

[0123] When the terminal device runs the method of this module, it uses the time-synchronized event stream data and radar measurement data as inputs. Subsequently, it processes the radar data based on the radar tracking model to generate sparse 3D point clouds, and at the same time processes the asynchronous event stream data based on the event tracking model for event filtering, UAV detection, and tracking. Finally, the terminal device aligns the outputs (the first detection result and the second detection result) of the two tracking models based on the measurement filter with time consistency, and uses the time consistency between the two sensors of the event camera and the millimeter-wave radar, as well as the periodic micro-motion characteristics of the UAV, to extract the ground positioning measurement data related to the UAV, thereby realizing the preliminary positioning of the UAV.

[0124] In addition, the graph-driven dynamic joint optimization method is a carefully designed factor graph-based optimization method adopted by the terminal device based on the working principles and noise distributions of the two sensors of the event camera and the millimeter-wave radar. When the terminal device runs the graph-driven dynamic joint optimization method, it utilizes the spatial complementarity of the two sensors to fully exert the potential of the event camera and the millimeter-wave radar in the ground positioning of the UAV. Specifically, the terminal device jointly fuses the preliminary positioning estimates of the event tracking model and the radar tracking model by running the graph-driven dynamic joint optimization method, and performs fine adjustment through adaptive optimization, so as to determine the precise position of the UAV with a processing time of milliseconds.

[0125] Please refer to Figure 11 , the embodiments of the present application also provide a ground-based UAV positioning device based on the high-frequency fusion of an event camera and a millimeter-wave radar, which can implement the above-mentioned ground-based UAV positioning solution based on the high-frequency fusion of an event camera and a millimeter-wave radar. The device includes:

[0126] An acquisition module, configured to acquire the event stream data of the UAV based on the event camera; and acquire the radar measurement data of the UAV based on the millimeter-wave radar;

[0127] A positioning module, configured to perform time-synchronized collaborative tracking processing on the UAV based on the event stream data and the radar measurement data to obtain the ground positioning measurement data of the UAV.

[0128] In some embodiments, the positioning module is further configured to obtain target event stream data from the event stream data, and obtain target radar measurement data from the radar measurement data; wherein, the target event stream data is time-synchronized with the target radar measurement data; input the target event stream data into an event tracking model to obtain a first detection result; and input the target radar measurement data into a radar tracking model to obtain a second detection result; and determine ground positioning measurement data of the UAV based on the first detection result and the second detection result.

[0129] In some embodiments, the positioning module is further configured to obtain a cross-modal alignment result of the first detection result and the second detection result; and extract the ground positioning measurement data of the UAV from the cross-modal alignment result based on the micro-motion period of the UAV.

[0130] In some embodiments, the positioning module is further configured to input the first detection result and the second detection result into a measurement filter; and perform cross-modal alignment on the first detection result and the second detection result through the measurement filter based on the time consistency and / or spatial relationship between the event camera and the millimeter-wave radar to obtain a cross-modal alignment result of the first detection result and the second detection result.

[0131] In some embodiments, the positioning module is further configured to construct a factor graph based on the first detection result and the second detection result; obtain a position state estimate of the UAV through dynamic joint optimization of the connected factor nodes in the factor graph; and determine the ground positioning measurement data of the UAV based on the position state estimate.

[0132] In some embodiments, the positioning module is further configured to optimize the prior probability represented by the connected factor nodes in the factor graph based on maximum a posteriori estimation, and optimize the likelihood probability of each of the event tracking model and the radar tracking model represented by the connected factor nodes;

[0133] wherein, the prior probability is obtained by analyzing the flight characteristics of the UAV, the likelihood probability of the event tracking model includes the likelihood probability of the first detection result, and the likelihood probability of the radar tracking model includes the likelihood probability of the second detection result.

[0134] It should be noted that the specific implementation manner of the ground-based UAV positioning device based on high-frequency fusion of an event camera and a millimeter-wave radar provided in the embodiments of the present application is basically the same as the specific embodiments of the above-mentioned ground-based UAV positioning solution based on high-frequency fusion of an event camera and a millimeter-wave radar, and will not be elaborated here.

[0135] The embodiments of the present application also provide a computer device, which includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the above-mentioned ground-based UAV positioning solution based on the high-frequency fusion of an event camera and a millimeter-wave radar. This computer device can be any intelligent terminal including a tablet computer, an in-vehicle computer, etc.

[0136] Please refer to Figure 12 , Figure 12 which schematically shows the hardware structure of a computer device according to another embodiment. The computer device includes:

[0137] A processor 1201, which can be implemented in ways such as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided by the embodiments of the present application;

[0138] A memory 1202, which can be implemented in forms such as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1202 can store an operating system and other application programs. When implementing the technical solutions provided by the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 1202 and are called by the processor 1201 to execute the ground-based UAV positioning solution based on the high-frequency fusion of an event camera and a millimeter-wave radar in the embodiments of the present application;

[0139] An input / output interface 1203, which is used to implement information input and output;

[0140] A communication interface 1204, which is used to implement communication interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.);

[0141] A bus 1205, which transmits information between various components of the device (such as the processor 1201, the memory 1202, the input / output interface 1203, and the communication interface 1204);

[0142] Among them, the processor 1201, the memory 1202, the input / output interface 1203, and the communication interface 1204 are communicatively connected to each other inside the device through the bus 1205.

[0143] The embodiment of the present application further provides a positioning system for an unmanned aerial vehicle (UAV). The positioning system includes the above-mentioned UAV and / or a ground base station, and can implement the above-mentioned ground-based UAV positioning solution based on the high-frequency fusion of an event camera and a millimeter-wave radar.

[0144] The embodiment of the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the above-mentioned ground-based UAV positioning solution based on the high-frequency fusion of an event camera and a millimeter-wave radar.

[0145] As a non-transitory computer-readable storage medium, a memory can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory can include high-speed random access memory, and can also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory may optionally include a memory remotely provided with respect to the processor, and these remote memories can be connected to the processor through a network. Examples of the above-mentioned network include but are not limited to the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0146] The embodiment of the present application further provides a computer program product. The computer program product stores a computer program, and when the computer program is executed by a processor, it implements the above-mentioned ground-based UAV positioning solution based on the high-frequency fusion of an event camera and a millimeter-wave radar.

[0147] The embodiments described in the embodiments of the present application are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present application are equally applicable to similar technical problems.

[0148] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown in the figures, or combine certain steps, or different steps.

[0149] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0150] Those of ordinary skill in the art can understand that all or some of the steps in the above-disclosed solutions / methods, and the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, and appropriate combinations thereof.

[0151] In the description of this application and the above-mentioned drawings, terms such as "first", "second", "third", "fourth", etc. (if any) are used to distinguish similar objects and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments of this application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0152] It should be understood that in this application, "at least one (item)" means one or more, and "a plurality" means two or more. "And / or" is used to describe the association relationship of associated objects and indicates that three relationships can exist. For example, "A and / or B" can mean: only A exists, only B exists, and both A and B exist at the same time. Among them, A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one (one) of the following" or its similar expression refers to any combination of these items, including any combination of single item (one) or plural items (ones). For example, at least one (one) of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0153] In several embodiments provided in this application, it should be understood that the disclosed devices and solutions can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the above-mentioned unit division is only a logical function division, and there can be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces, and the indirect coupling or communication connection of devices or units can be in electrical, mechanical or other forms.

[0154] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0155] In addition, in each embodiment of the present application, each functional unit can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0156] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the solutions of each embodiment of the present application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical discs that can store programs.

[0157] The preferred embodiments of the embodiments of the present application have been described above with reference to the accompanying drawings. This does not limit the scope of the rights of the embodiments of the present application. Any modifications, equivalent replacements, and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the rights of the embodiments of the present application.

Claims

1. A ground-based UAV positioning solution based on high-frequency fusion of event camera and millimeter-wave radar, characterized in that: include: Obtain the event stream data of the drone based on the event camera; and, obtaining radar measurement data of the UAV based on the millimeter wave radar; The UAV is subjected to time-synchronized collaborative tracking processing based on the event stream data and the radar measurement data to obtain ground positioning measurement data of the UAV.

2. The ground-based UAV positioning solution according to claim 1, characterized in that: The performing time-synchronous collaborative tracking processing on the UAV based on the event stream data and the radar measurement data to obtain ground positioning measurement data of the UAV includes: Acquire target event stream data in the event stream data, and acquire target radar measurement data in the radar measurement data; wherein the target event stream data is time synchronized with the target radar measurement data; Inputting the target event stream data into an event tracking model to obtain a first detection result; and inputting the target radar measurement data into a radar tracking model to obtain a second detection result; The ground positioning measurement data of the UAV is determined based on the first detection result and the second detection result.

3. The ground-based UAV positioning solution according to claim 2, characterized in that: The determining the ground positioning measurement data of the UAV based on the first detection result and the second detection result includes: Obtaining a cross-modal alignment result of the first detection result and the second detection result; Based on the micro-motion cycle of the UAV, ground positioning measurement data of the UAV is extracted from the cross-modal alignment result.

4. The ground-based UAV positioning solution according to claim 3, characterized in that: The obtaining a cross-modal alignment result of the first detection result and the second detection result includes: inputting the first detection result and the second detection result into a measurement filter; The first detection result and the second detection result are cross-modally aligned through the measurement filter based on the temporal consistency and / or spatial relationship between the event camera and the millimeter-wave radar to obtain a cross-modal alignment result of the first detection result and the second detection result.

5. The ground-based UAV positioning solution according to claim 2, characterized in that: The determining the ground positioning measurement data of the UAV based on the first detection result and the second detection result includes: constructing a factor graph based on the first detection result and the second detection result; Obtaining a position state estimate of the UAV based on dynamically joint optimization of factor nodes connected in the factor graph; Ground positioning measurements of the drone are determined based on the position state estimate.

6. The ground-based UAV positioning solution according to claim 5, characterized in that: The method of dynamically joint optimizing the factor nodes in the factor graph includes: Optimizing the prior probabilities of the connected factor nodes represented in the factor graph based on maximum a posteriori estimation, and optimizing the likelihood probabilities of the event tracking model and the radar tracking model represented by the connected factor nodes; The prior probability is obtained based on an analysis of the flight characteristics of the UAV, the likelihood probability of the event tracking model includes the likelihood probability of the first detection result, and the likelihood probability of the radar tracking model includes the likelihood probability of the second detection result.

7. A ground-based UAV positioning device based on high-frequency fusion of event camera and millimeter-wave radar, characterized in that: The device comprises: An acquisition module, configured to acquire event stream data of the drone based on an event camera; and acquire radar measurement data of the drone based on a millimeter wave radar; A positioning module is used to perform time-synchronized collaborative tracking processing on the UAV based on the event stream data and the radar measurement data to obtain ground positioning measurement data of the UAV.

8. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the ground-based UAV positioning solution based on high-frequency fusion of event camera and millimeter-wave radar as described in any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the ground-based UAV positioning solution based on high-frequency fusion of event camera and millimeter-wave radar according to any one of claims 1 to 6 is implemented.

10. A computer program product, wherein the computer program product stores a computer program, characterized in that: When the computer program is executed by a processor, the ground-based UAV positioning solution based on high-frequency fusion of event camera and millimeter-wave radar according to any one of claims 1 to 6 is implemented.

Images