Optical multi-sensor fixed-wing unmanned aerial vehicle detection and tracking guiding method and system

Abstract

The invention discloses an optical multi-sensor fixed-wing unmanned aerial vehicle detection and tracking guiding method and system. The method comprises the steps that a binocular camera, a laser radar and an optical pod are adopted to conduct information data collection on a cooperative unmanned aerial vehicle; integrating the collected information data by using a timestamp, and completing the joint calibration of the binocular camera and the optical pod and the joint calibration of the binocular camera and the laser radar; performing dual-light measurement on the target unmanned aerial vehicle to obtain spatial position coordinates and motion parameters of the target unmanned aerial vehicle; and the optical pod converts the spatial position coordinates and the motion parameters of the target unmanned aerial vehicle into a holder adjustment angle, and tracks and aims at the target unmanned aerial vehicle in real time to guide the target unmanned aerial vehicle. According to the method, the camera 2D image and the laser 3D point cloud data are registered and calibrated, so that the low-cost and high-resolution space information characteristics of the camera and the high-precision space information characteristics of the laser radar are fused, high-precision tracking guidance is realized, and real-time detection and landing process video monitoring of an unmanned aerial vehicle target are realized in combination with the three-light optical pod.

Description

Technical Field

[0001] The present invention relates to the field of unmanned aerial vehicle detection and guidance, and in particular to an optical multi-sensor fixed-wing unmanned aerial vehicle detection, tracking and guidance method and system. Background Art

[0002] With the rapid development of autonomous control technology and the rapid increase in the demand for carrying out diversified mission payloads, the demand for unmanned aerial vehicles (UAVs) to have good autonomous mission execution capabilities has become increasingly strong. During the execution of UAV missions, the safety risks during takeoff and landing are the greatest. Factors such as poor system navigation and positioning accuracy, large deviations in UAV flight parameters and attitude angles, etc. can cause the autonomous decision-making system or the guide of the UAV control and guidance system to make mistakes in the guidance decision-making process, which in turn causes frequent accidents during the landing of UAVs.

[0003] In terms of drone detection and landing guidance: At present, the domestic method of using traditional multi-sensor coordination, such as GPS, radar, and altimeter, is still the main method. This method not only has too high detection accuracy for drone targets, but also has too large deviations in positioning coordinate accuracy, flight parameters, and attitude angles. The system architecture is complex and electromagnetic interference is serious. In addition, the control and guidance system is highly dependent on the detection data of all sensors. If a sensor fails or works abnormally, it will often cause serious consequences, even aviation accidents in which drones fall and cause collateral damage. Affected by these unfavorable factors, the field of drone detection and landing guidance urgently needs a reliable, simple, non-contact, high-precision measurement and guidance system that does not interfere with drones and other systems. Summary of the invention

[0004] In view of the problems existing in the above-mentioned prior art, the present invention provides an optical multi-sensor fixed-wing UAV detection and tracking guidance method and system, the purpose of which is to design a ground dual-light optical detection system, by registering the camera 2D image and the laser 3D point cloud data, to integrate the low cost and high resolution of the camera and the high-precision spatial information characteristics of the laser radar, to perform visual and laser measurement of the UAV, and achieve high-precision tracking guidance. At the same time, it is equipped with a set of three-light optical pods containing visible light cameras, infrared cameras and laser ranging modules to achieve real-time detection of UAV targets and video monitoring of the landing process.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] In a first aspect, the present invention provides an optical multi-sensor fixed-wing UAV detection and tracking guidance method, comprising:

[0007] Use binocular cameras, laser radars, and optical pods to collect information and data from cooperative drones;

[0008] Use timestamps to integrate the collected information data and complete the joint calibration of the binocular camera and the optical pod, as well as the joint calibration of the binocular camera and the lidar;

[0009] The binocular camera and laser radar perform dual-light measurement on the target UAV to obtain the spatial position coordinates and motion parameters of the target UAV;

[0010] The optical pod converts the spatial position coordinates and motion parameters of the target UAV into gimbal adjustment angles, tracks the target UAV in real time, and guides the landing process of the target UAV in real time.

[0011] Preferably, using a binocular camera, a laser radar, and an optical pod to collect information data from the cooperative drones specifically includes:

[0012] The cooperative UAV collects the geographic coordinate matrix of the runway and the airspace above the runway in the landing area of ​​the target UAV to establish a runway coordinate system, and based on the runway coordinate system, sends the timestamp, the positioning information of the cooperative UAV, and the coordinate information of the calibration target carried by the cooperative UAV to the host computer;

[0013] The binocular camera collects the visible light images of the cooperative UAV; the laser radar collects the point cloud data of the cooperative UAV; the optical pod collects the visible light images, infrared images, and laser ranging information of the cooperative UAV to obtain the pitch angle and azimuth angle of the cooperative UAV;

[0014] The binocular camera and the laser radar respectively send the visible light image, point cloud data and the corresponding timestamp to the host computer, and the optical pod sends the pitch angle, azimuth angle and the corresponding timestamp and focal length information to the host computer.

[0015] Preferably, the joint calibration of the binocular camera and the optical pod includes:

[0016] Calibrate the binocular camera based on the coordinate information of the calibration target and the visible light image of the binocular camera to obtain the internal and external parameters of the binocular camera;

[0017] Based on the timestamp, the visible light images collected by the binocular camera and the optical pod at the same time are registered and matched to obtain the coordinate correspondence between the binocular camera and the optical pod;

[0018] The calibration parameters of the optical pod are obtained based on the coordinate correspondence between the optical pod and the binocular camera and the internal and external parameters of the binocular camera.

[0019] Preferably, the joint calibration of the binocular camera and the laser radar includes:

[0020] Converting the point cloud data into a point cloud intensity map;

[0021] Extracting first graph structure change features of each point in the point cloud intensity graphs of the previous and next frames, and obtaining a first change graph based on the first graph structure change feature graph structure;

[0022] Extracting second graph structural change features of each point in the visible light images of the front and rear frames of the binocular camera, and obtaining a second change graph based on the second graph structural change feature graph structure;

[0023] Inputting the first change map and the second change map into the LoFTR model to obtain the same-name point pairs of the two images;

[0024] Obtaining a coordinate transformation relationship between a binocular camera and a laser radar based on the same-name point pairs;

[0025] Calibrate the binocular camera based on the coordinate information of the calibration target and the visible light image of the binocular camera to obtain the internal and external parameters of the binocular camera;

[0026] The calibration parameters of the lidar are obtained based on the coordinate transformation relationship between the binocular camera and the lidar and the intrinsic and extrinsic parameters of the binocular camera.

[0027] Preferably, the step of extracting the first graph structure change features of each point in the point cloud intensity graphs of the previous and next frames, and obtaining the first change graph of the point cloud intensity graph based on the first graph structure change features comprises:

[0028] (1) Overlap and compare the point cloud intensity maps of the previous and next frames, extract the difference area to screen the change points;

[0029] (2) In the point cloud intensity map, points with similar pixel values ​​in the neighborhood of each point are grouped into multiple graph connected domains;

[0030] (3) calculating the weight of the connection edge between each change point and each point in the connected domain of the graph where it is located based on the distance between the two points and the pixel difference between the two points;

[0031] (4) weighting the plurality of connecting edge weights of each change point with the pixel value of the change point to obtain a first graph structural feature of each change point;

[0032] (5) calculating the difference between the first graph structural features of the same change point in the previous and next frames to obtain the first graph structural change feature of the change point;

[0033] (6) Mark the change points in the point cloud intensity map where the structural change characteristics of the first map are greater than a preset threshold, to obtain the first change map.

[0034] Preferably, the step of obtaining the second change graph is specifically:

[0035] The visible light images of the front and rear frames are used to replace the point cloud intensity images of the front and rear frames to obtain the second image structure change characteristics according to the steps (1)-(5), and the visible light images of the front and rear frames of the binocular camera are marked according to the same marking rules as in the step (6) to obtain a second change map.

[0036] Preferably, the calibration parameters of the laser radar obtained based on the coordinate transformation relationship between the binocular camera and the laser radar and the internal and external parameters of the binocular camera are specifically:

[0037]

[0038] Among them, K is the intrinsic parameter matrix of the binocular camera, R and T are the extrinsic parameter matrices of the binocular camera, M is the extrinsic parameter matrix of the lidar, and f is the coordinate transformation relationship between the binocular camera and the lidar.

[0039] Preferably, the optical pod converts the spatial position coordinates and motion parameters of the target UAV into a pan / tilt adjustment angle, including:

[0040] In the runway coordinate system, the coordinate positions of the runway origin and the optical pod measurement station in the world coordinate system are calibrated, and the world coordinate system is converted to the runway coordinate system by combining the positioning data of the runway origin and the coordinate conversion method;

[0041] An optical pod coordinate system is established, and the three-dimensional coordinate values ​​in the runway coordinate system are converted into the horizontal angle and pitch angle of the optical pod gimbal control in the optical pod coordinate system.

[0042] Preferably, the method further comprises:

[0043] Based on the spatial position coordinates and motion parameters of the target UAV and the gimbal rotation time, the gimbal control horizontal angle and pitch angle are corrected in advance.

[0044] In a second aspect, the present invention provides an optical multi-sensor fixed-wing UAV detection and tracking guidance system, comprising:

[0045] The data acquisition module is used to collect information data from the cooperative UAVs using binocular cameras, laser radars, and optical pods;

[0046] The joint calibration module is used to integrate the collected information data using timestamps to complete the joint calibration of the binocular camera and the optical pod, as well as the joint calibration of the binocular camera and the lidar;

[0047] The target measurement module is used for binocular camera and laser radar to perform dual-light measurement on the target UAV to obtain the spatial position coordinates and motion parameters of the target UAV;

[0048] The tracking, aiming and guiding module is used to convert the spatial position coordinates and motion parameters of the target UAV into the gimbal adjustment angle based on the optical pod, track the target UAV in real time, and guide the landing process of the target UAV in real time.

[0049] The optical multi-sensor fixed-wing UAV detection and tracking guidance method and system of the present invention has the following features:

[0050] Beneficial effects:

[0051] 1. The present invention realizes high-precision tracking and guidance by aligning and calibrating the camera's 2D image and the laser 3D point cloud data to integrate the low cost and high resolution of the camera and the high-precision spatial information characteristics of the laser radar, and combines the three-light optical pod to realize real-time detection of UAV targets and video monitoring of the landing process.

[0052] 2. The present invention converts point cloud data and visible light images into the same change feature map, in which consistent feature descriptions are provided. In the two images, the areas that change at the same time are the most likely areas to be the same. Therefore, by extracting points with significant change intensity to filter out a large number of irrelevant points, the problem of difficulty in extracting feature descriptors due to the unevenness and sparsity of the point cloud and the large difference between the feature descriptors and the feature descriptors of the visible light image is avoided.

[0053] 3. The optical multi-sensor fixed-wing UAV detection and tracking guidance system designed by the present invention only needs to be installed on one side of the airport runway. There is no need to install system equipment on the UAV platform. Only one set of system is needed at the landing site to complete the purpose of multi-type UAV detection and landing guidance. It does not occupy the payload resources of the UAV platform, and the system procurement cost and operating cost are low, with high cost performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a schematic diagram of the composition of the optical multi-sensor fixed-wing UAV detection and tracking guidance system provided by the present invention;

[0055] Figure 2 It is a schematic diagram of the design architecture of the optical multi-sensor fixed-wing UAV detection and tracking guidance system provided by the present invention;

[0056] Figure 3 This is a schematic diagram of system layout in an outdoor application according to an embodiment of the present invention;

[0057] Figure 4 It is a flow chart of a method for detecting, tracking and guiding an optical multi-sensor fixed-wing UAV according to the present invention;

[0058] Figure 5 is a schematic diagram of the conversion relationship between the runway coordinate system and the optical pod coordinate system in one embodiment of the present invention;

[0059] Figure 6 It is a structural block diagram of an optical multi-sensor fixed-wing UAV detection and tracking guidance system of the present invention. DETAILED DESCRIPTION

[0060] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] First, if Figure 1 As shown, the present invention provides an optical multi-sensor fixed-wing UAV detection and tracking guidance system architecture, including 2 high-resolution cameras, 2 laser radar groups, 1 three-light (visible light + infrared + laser) optical pod, 1 data processor, 3 small industrial computers, two portable power supplies, 1 portable workstation, 1 time system module and 1 terminal control host computer and data processing software.

[0062] Reference Figure 2 , in order to facilitate the system layout, disassembly and portability, the system measurement module is designed as three groups of highly integrated measurement modules. Specifically: the first high-resolution camera and the first laser radar form a group of visible light and laser dual-light optical measurement sensors, equipped with a small industrial computer as a data processing and control module, forming the first measurement module group, integrated in a sealed box, supported by a tripod, convenient for system layout and storage; the second high-resolution camera and the second laser radar form another group of visible light and laser dual-light optical measurement sensors, equipped with a small industrial computer as a data processing and control module, forming the second measurement module group, integrated in a sealed box, supported by a tripod, convenient for system layout and storage; the three-light pod is used as the three-light optical tracking and aiming system of the drone, which is laid out separately in the form of a complete machine, and equipped with a small industrial computer to form the third measurement module.

[0063] Two high-resolution cameras can be connected to the first industrial computer via a network cable or a serial cable, and the supporting data processing software system can provide camera triggering, synchronization, preview, high-speed acquisition, storage and data transmission through the GigE network or R422; two laser radars form a long-range small-field-of-view laser radar group, and the two laser radars can be connected to the second industrial computer via a network or a serial cable, and the supporting data processing software system can control the laser radar work through the GigE network or R422; the visible light camera, laser ranging module, infrared camera and pan / tilt form a three-light optical pod, which can be connected to the third industrial computer via a network cable or a serial cable, and the supporting data processing software system can control the laser radar work through the GigE network or R422. The system controls the triggering, synchronization, preview, high-speed acquisition, storage and data transmission of the visible light camera through the GigE network or R422. At the same time, the visible light camera provides a trigger signal to the infrared camera. The control and data transmission of the infrared camera can be completed through the GigE network or R422 interface. The control and data transmission of the laser ranging module can also be completed through the GigE network or R422 interface; the time system module provides the time information required by the system; the system calibration module completes the control and data transmission functions through the network, wireless transmission and other channels; the network industrial computer, time system module, portable workstation and flight control center are connected through the network to complete the command issuance and data transmission.

[0064] The above three measurement modules constitute a far-field integrated multi-sensor measurement system, which consists of a high-resolution camera group and a lidar group to form a visible light and laser dual-light detection system. The terminal controls the host computer and data processing software to solve the target space position coordinates and flight parameters in real time, and converts the target parameters into parameters such as the pitch angle, azimuth angle, visible light camera and infrared camera focal length of the small optical pod to automatically control the gimbal, so as to realize real-time aiming, tracking and monitoring of the target UAV by the three-light sensors in the three-light pod in the third measurement module.

[0065] In actual application, the location of the deployment site is preliminarily evaluated based on the target parameters (size, speed, and landing point range), and the first measurement module group, the second measurement module group, and the third measurement module group in the above system architecture are deployed on the same side, and the measurement fields of the above first measurement module group, the second measurement module group, and the third measurement module group are perpendicular to the target UAV's glide path direction, and the detection sensor fields of the above first measurement module group, the second measurement module group, and the third measurement module group cover the runway before the target UAV lands and the airspace above the runway. The system is powered on, the measurement modules inside the system and the communication link with the outside are established, and the system enters standby mode. The system architecture in this embodiment is deployed in the field as follows: Figure 3 shown.

[0066] After the system is deployed, an optical multi-sensor fixed-wing UAV detection and tracking guidance method disclosed in an embodiment of the present invention is introduced in detail below.

[0067] like Figure 1 As shown, the steps include:

[0068] Release the cooperative UAV, which is equipped with a calibration target for binocular camera calibration, to establish an RTK calibration link;

[0069] Use binocular cameras, laser radars, and optical pods to collect information and data from cooperative drones;

[0070] Use timestamps to integrate the collected information data and complete the joint calibration of the binocular camera and the optical pod, as well as the joint calibration of the binocular camera and the lidar;

[0071] The binocular camera and laser radar perform dual-light measurement on the target UAV to obtain the spatial position coordinates and motion parameters of the target UAV;

[0072] The optical pod converts the spatial position coordinates and motion parameters of the target UAV into gimbal adjustment angles, tracks the target UAV in real time, and guides the landing process of the target UAV in real time.

[0073] The present invention uses a camera and a laser radar to form a dual-light optical detection system, which has the function of fusing camera 2D images and laser 3D point clouds, and uses cooperative drones for visual and laser measurement calibration to achieve high-precision target space position analysis. At the same time, it is equipped with a set of three-light optical pods containing visible light cameras, infrared cameras and laser ranging modules to achieve real-time detection of drone targets and video monitoring of the landing process, guiding the auxiliary target drone to land safely.

[0074] Preferably, using a binocular camera, a laser radar, and an optical pod to collect information data from the cooperative drones specifically includes:

[0075] The cooperative UAV collects the geographic coordinate matrix of the runway and the airspace above the runway in the landing area of ​​the target UAV to establish a runway coordinate system, and based on the runway coordinate system, sends the timestamp, the positioning information of the cooperative UAV, and the coordinate information of the calibration target carried by the cooperative UAV to the host computer;

[0076] The timestamp records the precise time of data acquisition and is used to align data of different modalities acquired by different sensors based on time comparison.

[0077] The binocular camera collects visible light images of the cooperative UAV; the laser radar collects point cloud data of the cooperative UAV; as mentioned above, the optical pod integrates a visible light camera, an infrared camera and a laser ranging module to collect the visible light image, infrared image and laser ranging information of the cooperative UAV, and obtain the pitch angle and azimuth angle of the cooperative UAV;

[0078] The binocular camera and the laser radar respectively send the visible light image, point cloud data and the corresponding timestamp to the host computer, and the optical pod sends the pitch angle, azimuth angle and the corresponding timestamp and focal length information to the host computer.

[0079] The optical pod only performs three-light measurement without positioning, and is used to adjust the angle of the gimbal to track the target in real time. Target positioning is only completed by the binocular measurement system and laser radar. It should be understood that this is because the camera's external parameters will change with the shooting angle, so the optical pod is used to control the lens to follow the target drone in real time.

[0080] Preferably, the joint calibration of the binocular camera and the optical pod includes:

[0081] The binocular camera is calibrated based on the coordinate information of the calibration target and the visible light image of the binocular camera to obtain the intrinsic and extrinsic parameters of the binocular camera. In this embodiment, the intrinsic and extrinsic parameters of the camera are calibrated using the binocular vision measurement method in the prior art, which specifically includes:

[0082] The target point P is the coordinate point P in the world coordinate system NEU (X, Y, Z) and the camera imaging pixel coordinate point p(u, v) satisfy the following relationship:

[0083]

[0084] Among them, K, R and T are the camera's internal parameters, rotation matrix and translation matrix respectively. Using the principle of photogrammetry, this system can calculate the coordinate point p(u,v) of the RTK drone in the image coordinate system through the images taken in real time by camera 1 and camera 2; the positioning data sent back by the RTK drone positioning module can calculate the coordinate point P of the RTK drone in the world coordinate system NEU (X,Y,Z). According to the coordinate transformation relationship, is the projection matrix of the camera. Camera calibration is to determine the parameters of the matrix P, which contains 12 elements. 23 After normalization, only the remaining 11 unknowns need to be solved.

[0085] Through matrix operations, the pinhole imaging model is expanded into the following conversion equation:

[0086]

[0087] Therefore, at least 6 points are calibrated in the world coordinate system, and the other 11 unknowns can be solved to obtain the mapping matrix between the two coordinate systems.

[0088] Based on the timestamp, the visible light images collected by the binocular camera and the optical pod at the same time are registered and matched to obtain the coordinate correspondence between the binocular camera and the optical pod;

[0089] The calibration parameters of the optical pod are obtained based on the coordinate correspondence between the optical pod and the binocular camera and the internal and external parameters of the binocular camera.

[0090] Both the binocular camera and the optical pod collect visible light images, which belong to the same modality data. They can directly perform image registration and find the correspondence between the coordinates of the same points in two images at the same time.

[0091] The spatial coordinates and motion parameters of the subsequent target UAV are calculated using data collected by the binocular camera and lidar. The lidar is also calibrated based on the internal and external parameters of the binocular camera. Therefore, in order to be able to convert the spatial coordinates and motion parameters of the target UAV into the gimbal angle of the optical pod, the binocular camera and the optical pod need to be jointly calibrated.

[0092] The target recognition technology route based on monocular vision achieves satisfactory application performance at a low cost, but it cannot provide reliable three-dimensional parameters of spatial position. Although binocular cameras can provide spatial information, they have high computational costs and are limited by highly occluded and textureless environments. In addition, vision-based perception systems have low reliability in complex lighting environments, which limits their ability to work in all weather conditions. LiDAR is not affected by lighting conditions and can provide high-precision spatial information, but its limitations are low resolution and effective sampling rate, and high cost.

[0093] Camera-lidar fusion perception technology can improve performance and reliability while reducing system costs. The camera records information by projecting the physical world onto the camera plane, while the laser point cloud records and saves the spatial information of the surface of objects in the visible area. In terms of data structure and type, the spatial information reflected in the point cloud data is sparse and disordered, and the correlation between the spatial information and its overall attributes is weak, and the morphology changes dramatically, while the image data is regular, and the local information is more correlated with its overall, which leads to huge differences in image and point cloud processing algorithms. This system uses the methods of direct calibration of visible light cameras and indirect calibration of lidars to jointly calibrate the two types of sensors. Therefore, an embodiment of the present invention provides a method for aligning image data and point cloud data. Specifically, the joint calibration of the binocular camera and the lidar includes:

[0094] Converting the point cloud data into a point cloud intensity map;

[0095] In this embodiment, the original point cloud data is converted into a point cloud intensity map based on the rigid rotation and projection transformation of the Rodriguez equation, wherein the rigid rotation and projection transformation of the Rodriguez equation are prior arts and will not be described in detail here.

[0096] Extracting first graph structure change features of each point in the point cloud intensity graphs of the previous and next frames, and obtaining a first change graph based on the first graph structure change feature graph structure;

[0097] Extracting second graph structural change features of each point in the visible light images of the front and rear frames of the binocular camera, and obtaining a second change graph based on the second graph structural change feature graph structure;

[0098] Inputting the first change map and the second change map into the LoFTR model to obtain the same-name point pairs of the two images;

[0099] In one embodiment, the first change map and the second change map can be input into the LoFTR model to obtain the same-name point pairs of the two images and the confidence of the same-name point pairs. A confidence threshold is set, and the same-name point pairs with confidence greater than the threshold are retained to improve the accuracy of the analysis results of the same point between the two modal data.

[0100] The coordinate transformation relationship between the binocular camera and the laser radar is obtained based on the same-name point pair; the same-name point is the different coordinate representation of the same point in space in the two images.

[0101] Calibrate the binocular camera based on the coordinate information of the calibration target and the visible light image of the binocular camera to obtain the internal and external parameters of the binocular camera;

[0102] The calibration parameters of the lidar are obtained based on the coordinate transformation relationship between the binocular camera and the lidar and the intrinsic and extrinsic parameters of the binocular camera.

[0103] Radar can only detect the distance between objects, and cannot identify targets used for ordinary camera calibration. The prior art requires the design of special targets to calibrate or calibrate the relative position relationship between the lidar and the camera. However, during field applications, affected by factors such as system mechanical vibration, thermal radiation, wind and sand, the calibration parameters of the two types of sensors will no longer be appropriate as the system works over time. By applying the technical solution of the present invention, no specific targets are used, and no pre-calibration is required in the laboratory. The calibration coefficient can be adjusted in time, which has a wide range of application value.

[0104] Preferably, the step of extracting the first graph structure change features of each point in the point cloud intensity graphs of the previous and next frames, and obtaining the first change graph of the point cloud intensity graph based on the first graph structure change features comprises:

[0105] (1) Overlap and compare the point cloud intensity maps of the previous and next frames, and extract the difference areas to screen the change points.

[0106] It should be understood that point cloud data is the distance information between the radar and the surrounding objects. In point cloud data, intensity refers to the reflection intensity value returned by each point when the lidar sensor scans the environment. In the point cloud intensity map, the pixel represents the intensity information of the point.

[0107] The point cloud is non-uniform and sparse, and the point cloud intensity map contains a lot of useless information and noise, which affects the extraction of points with the same name in the next step of the algorithm.

[0108] Since the present invention is applied to the guided landing of target UAVs, the collected data are aerial scenes, most of which are sky backgrounds that do not change in a short time. Therefore, by overlapping and comparing the point cloud intensity maps of the previous and next frames, the areas with different changes are extracted.

[0109] In the difference area, the average change value of all pixels in the point cloud intensity maps of the previous and next frames can be calculated, and the change points can be screened based on the size of the average change value.

[0110] (2) In the point cloud intensity map, points with similar pixel values ​​in the neighborhood of each point are grouped into multiple graph connected domains. It should be noted that the neighborhood is an irregular shape, and all points that are connected and have similar pixel values ​​are classified into one category.

[0111] (3) Based on the distance between the two points and the pixel difference between the two points, the weight of the connection edge between each change point and each point in the connected domain of the graph is calculated. The connection edge weight represents the similarity between the two points in space. The farther the distance, the lower the weight; the larger the pixel difference, the lower the weight.

[0112] (4) The weights of the multiple connecting edges of each change point are weighted with the pixel value of the change point to obtain the first graph structure feature of each change point.

[0113] The existence of a connecting edge means that the pixel features of two points are similar, and the connecting edge weight measures the specific degree of similarity. Therefore, the first graph structural feature represents the relationship and mutual influence between points in the image, that is, whether the change of one point will cause the same or similar change in the other point.

[0114] (5) The difference between the first graph structural features of the same change point in the previous and next frames is calculated to obtain the first graph structural change feature of the change point. It should be noted that the front graph connected domain in the previous frame and the back graph connected domain in the back frame are fused. If the point is connected to point A in the front graph connected domain but not in the back graph connected domain, then in the back frame, it is still considered that there is a connection edge between the two points, but the weight is assigned to 0. In other words, a connection edge weight equal to 0 is equivalent to the fact that there is no connection edge.

[0115] (6) Mark the change points in the point cloud intensity map whose structural change characteristics of the first map are greater than a preset threshold, and obtain the first change map. In one embodiment, the marking rule can be: use a specific color to color the change points that meet the above conditions. After marking with the characteristic color, a new feature exists in the first change map, and this new feature gives the two images common feature information, which can then be compared.

[0116] The present invention evaluates the change intensity of points through the consistency of the graph structure in the previous and next frame images, and selects points with significant changes, that is, points with the first graph structure change characteristics greater than a preset threshold, as prominent features, which serve as the basis for subsequent comparison.

[0117] Preferably, the step of obtaining the second change graph is specifically:

[0118] The visible light images of the front and rear frames are used to replace the point cloud intensity images of the front and rear frames to obtain the second image structure change characteristics according to the steps (1)-(5), and the visible light images of the front and rear frames of the binocular camera are marked according to the same marking rules as in the step (6) to obtain a second change map.

[0119] Using the same marking rules allows the information in the two images to be interoperable. Not paying attention to other areas but only caring about the areas where changes occur can effectively reduce the number of points that need to be analyzed.

[0120] Since the information obtained by the laser radar is the distance information between the surrounding objects, which is different from the modality of the direct object image obtained by the camera, and since the two modalities lack appearance and geometric correlation, it is quite challenging to learn common feature descriptors to establish the corresponding relationship of registration. Based on the application scenario of pure background, the present invention converts the point cloud data into a point cloud intensity map, so that the two data can be compared in one mode at the same time. However, there is still a huge difference between the intensity information of the point cloud and the representation content of the visible light image. Therefore, the present invention extracts the respective change features in the two images. The change of the point cloud intensity means the change of the target surface, that is, the change of the incident angle direction, that is, the target movement produces a different incident angle. At this time, there must be pixel changes in the same area of ​​the target in the visible light image. By inputting this consistency change into the LoFTR model, the LoFTR model accepts the above-mentioned specific color marking information, analyzes each point of the specific color mark, first establishes pixel-by-pixel dense matching at a coarse granularity, and then improves the algorithm for fine-tuning the matching at a fine granularity. The feature descriptor is extracted based on the Transformer so that the image can produce dense matching even in weak texture areas.

[0121] Preferably, the calibration parameters of the laser radar obtained based on the coordinate transformation relationship between the binocular camera and the laser radar and the internal and external parameters of the binocular camera are specifically:

[0122]

[0123] Among them, K is the intrinsic parameter matrix of the binocular camera, R and T are the extrinsic parameter matrices of the binocular camera, M is the extrinsic parameter matrix of the lidar, and f is the coordinate transformation relationship between the binocular camera and the lidar.

[0124] In field applications, after the system is deployed, the visible light camera is calibrated by the RTK drone to establish the external parameters between the camera and the RTK drone mobile station. The coordinate transformation relationship between the visible light camera and the laser radar is solved, thereby indirectly obtaining the external parameters between the laser radar and the RTK drone mobile station.

[0125] On the basis of the above calibration, that is, under the constraints of the camera's internal parameters, the external parameters between the camera and the lidar, the constraint surface equations and other conditions, the three-dimensional coordinates, speed, attitude angle and other parameters of the fixed-wing UAV in the landing process relative to the runway coordinate system are measured through two optical measurement methods, photogrammetry and laser measurement, and the UAV target is tracked to obtain its motion trajectory.

[0126] Preferably, the optical pod converts the spatial position coordinates and motion parameters of the target UAV into the gimbal adjustment angle including:

[0127] In the runway coordinate system, the coordinate positions of the runway origin and the optical pod measurement station in the world coordinate system are calibrated, and the world coordinate system is converted to the runway coordinate system by combining the positioning data of the runway origin and the coordinate conversion method.

[0128] like Figure 5 As shown, the optical pod is arranged on one side of the runway, the value of the X axis relative to the runway origin is m, and the value of the Y axis is -n. Select point P in the runway coordinate system, and transform the coordinates of point P into the optical pod coordinate system according to the relative relationship between the runway origin and the optical pod measurement station.

[0129] The optical pod coordinate system is established, and the three-dimensional coordinate values ​​in the runway coordinate system are converted into the horizontal angle and pitch angle of the optical pod gimbal control in the optical pod coordinate system. In the figure, ω is the horizontal angle, ψ is the pitch angle, and R is the working distance of the optical pod camera. The focal length f of the lens can be calculated by combining the field of view and the camera target size.

[0130] In the actual application process of the present invention, when the target UAV enters the measurement field of view, the visible light binocular camera and the laser radar perform dual-light measurement on the target, and transmit the measured target space position coordinates and motion parameters in the runway coordinate system to the pan / tilt control conversion module of the optical pod measurement module industrial computer. The pan / tilt control conversion module converts the target space position coordinates in the runway coordinate system into the pan / tilt control horizontal angle ω and pitch angle ψ in the optical pod coordinate system in real time, and adjusts the lens to track, aim and shoot the target UAV in real time.

[0131] Taking into account the influence of the delay from the start of the gimbal rotation to the aiming target on the tracking and aiming effect, according to the measured target movement speed, acceleration, pitch angle, roll angle and heading parameters combined with the gimbal rotation time, this method also includes an advance correction of the gimbal control horizontal angle and pitch angle to achieve the control target real-time display at the center of the optical pod video image, thereby improving the target tracking and aiming quality.

[0132] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0133] Based on the same inventive concept, the embodiment of the present application also provides a system for realizing the above-mentioned optical multi-sensor fixed-wing UAV detection and tracking guidance. The implementation scheme for solving the problem provided by the system is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in the embodiment of the optical multi-sensor fixed-wing UAV detection and tracking guidance system provided below can be referred to the limitations of the method above, and will not be repeated here.

[0134] like Figure 6 As shown, the present invention also provides an optical multi-sensor fixed-wing UAV detection and tracking guidance system, comprising:

[0135] The data acquisition module is used to collect information data from the cooperative UAVs using binocular cameras, laser radars, and optical pods;

[0136] The joint calibration module is used to integrate the collected information data using timestamps to complete the joint calibration of the binocular camera and the optical pod, as well as the joint calibration of the binocular camera and the lidar;

[0137] The target measurement module is used for binocular camera and laser radar to perform dual-light measurement on the target UAV to obtain the spatial position coordinates and motion parameters of the target UAV;

[0138] The tracking, aiming and guiding module is used to convert the spatial position coordinates and motion parameters of the target UAV into the gimbal adjustment angle based on the optical pod, track the target UAV in real time, and guide the landing process of the target UAV in real time.

[0139] It should be noted that: the optical multi-sensor fixed-wing UAV detection and tracking guidance system provided in this embodiment only uses the division of the above-mentioned functional modules as an example when guiding the target UAV. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules. Each functional module can be composed of a single execution unit, or it can be integrated into a functional module by two or more execution units to realize all the functions of the functional module.

[0140] Those skilled in the art will appreciate that the above modules may be implemented in whole or in part by software, hardware, or a combination thereof. The above modules may be embedded in or independent of a processor in a computer device in the form of hardware, or may be stored in a memory in a computer device in the form of software, so that the processor may call and execute operations corresponding to the above modules.

[0141] The present invention is not limited to the above-mentioned specific implementation modes. Various changes made by ordinary technicians in this field based on the above-mentioned concepts without creative work are all within the protection scope of the present invention.

Claims

1. An optical multi-sensor fixed-wing UAV detection and tracking guidance method, characterized in that: The steps include: Use binocular cameras, laser radars, and optical pods to collect information and data from cooperative drones; Use timestamps to integrate the collected information data and complete the joint calibration of the binocular camera and the optical pod, as well as the joint calibration of the binocular camera and the lidar; The binocular camera and laser radar perform dual-light measurement on the target UAV to obtain the spatial position coordinates and motion parameters of the target UAV; The optical pod converts the spatial position coordinates and motion parameters of the target UAV into gimbal adjustment angles, tracks the target UAV in real time, and guides the landing process of the target UAV in real time.

2. The optical multi-sensor fixed-wing UAV detection and tracking guidance method according to claim 1 is characterized in that: The use of binocular cameras, laser radars, and optical pods to collect information data from cooperative drones specifically includes: The cooperative UAV collects the geographic coordinate matrix of the runway and the airspace above the runway in the landing area of ​​the target UAV to establish a runway coordinate system, and based on the runway coordinate system, sends the timestamp, the positioning information of the cooperative UAV, and the coordinate information of the calibration target carried by the cooperative UAV to the host computer; The binocular camera collects the visible light image of the cooperative UAV; the laser radar collects the point cloud data of the cooperative UAV; the optical pod collects the visible light image, infrared image, and laser ranging information of the cooperative UAV to obtain the pitch angle and azimuth angle of the cooperative UAV; The binocular camera and the laser radar respectively send the visible light image, point cloud data and the corresponding timestamp to the host computer, and the optical pod sends the pitch angle, azimuth angle and the corresponding timestamp and focal length information to the host computer.

3. The optical multi-sensor fixed-wing UAV detection and tracking guidance method according to claim 2 is characterized in that: The joint calibration of the binocular camera and the optical pod includes: Calibrate the binocular camera based on the coordinate information of the calibration target and the visible light image of the binocular camera to obtain the internal and external parameters of the binocular camera; Based on the timestamp, the visible light images collected by the binocular camera and the optical pod at the same time are registered and matched to obtain the coordinate correspondence between the binocular camera and the optical pod; The calibration parameters of the optical pod are obtained based on the coordinate correspondence between the optical pod and the binocular camera and the internal and external parameters of the binocular camera.

4. The optical multi-sensor fixed-wing UAV detection and tracking guidance method according to claim 2 is characterized in that: The joint calibration of the binocular camera and the laser radar includes: Converting the point cloud data into a point cloud intensity map; Extracting first graph structure change features of each point in the point cloud intensity graphs of the previous and next frames, and obtaining a first change graph based on the first graph structure change feature graph structure; Extracting second graph structural change features of each point in the visible light images of the front and rear frames of the binocular camera, and obtaining a second change graph based on the second graph structural change feature graph structure; Inputting the first change map and the second change map into the LoFTR model to obtain the same-name point pairs of the two images; Obtaining a coordinate transformation relationship between a binocular camera and a laser radar based on the same-name point pairs; Calibrate the binocular camera based on the coordinate information of the calibration target and the visible light image of the binocular camera to obtain the internal and external parameters of the binocular camera; The calibration parameters of the lidar are obtained based on the coordinate transformation relationship between the binocular camera and the lidar and the intrinsic and extrinsic parameters of the binocular camera.

5. The optical multi-sensor fixed-wing UAV detection and tracking guidance method according to claim 4 is characterized in that: The step of extracting first graph structure change features of each point in the point cloud intensity graphs of the previous and next frames, and obtaining a first change graph of the point cloud intensity graph based on the first graph structure change features comprises: (1) Overlap and compare the point cloud intensity maps of the previous and next frames, extract the difference area to screen the change points; (2) In the point cloud intensity map, points with similar pixel values ​​in the neighborhood of each point are grouped into multiple graph connected domains; (3) calculating the weight of the connection edge between each change point and each point in the connected domain of the graph where it is located based on the distance between the two points and the pixel difference between the two points; (4) weighting the plurality of connecting edge weights of each change point with the pixel value of the change point to obtain a first graph structural feature of each change point; (5) calculating the difference between the first graph structural features of the same change point in the previous and next frames to obtain the first graph structural change feature of the change point; (6) Mark the change points in the point cloud intensity map where the structural change characteristics of the first map are greater than a preset threshold, to obtain the first change map.

6. The optical multi-sensor fixed-wing UAV detection and tracking guidance method according to claim 5 is characterized in that: The step of obtaining the second change graph is specifically as follows: The visible light images of the front and rear frames are used to replace the point cloud intensity images of the front and rear frames to obtain the second image structure change characteristics according to the steps (1)-(5), and the visible light images of the front and rear frames of the binocular camera are marked according to the same marking rules as in the step (6) to obtain a second change map.

7. The optical multi-sensor fixed-wing UAV detection and tracking guidance method according to claim 6 is characterized in that: The calibration parameters of the laser radar obtained based on the coordinate transformation relationship between the binocular camera and the laser radar and the internal and external parameters of the binocular camera are specifically: Among them, K is the intrinsic parameter matrix of the binocular camera, R and T are the extrinsic parameter matrices of the binocular camera, M is the extrinsic parameter matrix of the lidar, and f is the coordinate transformation relationship between the binocular camera and the lidar.

8. The optical multi-sensor fixed-wing UAV detection and tracking guidance method according to claim 2, characterized in that: The optical pod converts the spatial position coordinates and motion parameters of the target UAV into the pan / tilt adjustment angle, including: In the runway coordinate system, the coordinate positions of the runway origin and the optical pod measurement station in the world coordinate system are calibrated, and the world coordinate system is converted to the runway coordinate system by combining the positioning data of the runway origin and the coordinate conversion method; An optical pod coordinate system is established, and the three-dimensional coordinate values ​​in the runway coordinate system are converted into the horizontal angle and pitch angle of the optical pod gimbal control in the optical pod coordinate system.

9. The optical multi-sensor fixed-wing UAV detection and tracking guidance method according to claim 8, characterized in that: The method further comprises: Based on the spatial position coordinates and motion parameters of the target UAV and the gimbal rotation time, the gimbal control horizontal angle and pitch angle are corrected in advance.

10. An optical multi-sensor fixed-wing UAV detection and tracking guidance system, characterized in that: include: The data acquisition module is used to collect information data from the cooperative UAVs using binocular cameras, laser radars, and optical pods; The joint calibration module is used to integrate the collected information data using timestamps to complete the joint calibration of the binocular camera and the optical pod, as well as the joint calibration of the binocular camera and the lidar; The target measurement module is used for binocular camera and laser radar to perform dual-light measurement on the target UAV to obtain the spatial position coordinates and motion parameters of the target UAV; The tracking, aiming and guiding module is used to convert the spatial position coordinates and motion parameters of the target UAV into the gimbal adjustment angle based on the optical pod, track the target UAV in real time, and guide the landing process of the target UAV in real time.

Images